Method, device and equipment for training strategy planning model and storage medium
By acquiring task data through long-term, multi-round interactions, and using a policy programming model for task decomposition and correction, training samples are generated, enabling continuous autonomous fine-tuning of a large language model. This solves the problems of high policy programming error rate and high fine-tuning cost in existing technologies, and improves the model's policy learning efficiency and consistency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-04
- Publication Date
- 2026-06-05
Smart Images
Figure CN122154909A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to the fields of big data analysis technology, intelligent agent technology, etc., and particularly to a training method, apparatus, device and storage medium for a strategy planning model. Background Technology
[0002] Large language models are widely used in long-term, multi-turn interaction scenarios, and are given the ability to call external tools. Such long-term interactions involve continuous tasks across rounds and sessions, requiring large language models to have memory, planning, and continuous learning capabilities.
[0003] Currently, the relevant technical solutions are long-range dialogue systems based on limited context windows and external vector retrieval, and continuous learning solutions that use batch data or online reinforcement learning to fine-tune the entire basic large model.
[0004] However, the relevant technical solutions still have the following problems: First, in long dialogues or complex tasks, key information is easily exceeded by the context or buried in massive amounts of text, making it difficult for the model to accurately extract and apply it. The model's knowledge and human persona are inconsistent, resulting in a high error rate in task decomposition and execution planning under multi-step complex instructions. Second, fine-tuning large models is costly and inefficient, making it difficult to optimize strategy decisions in a targeted manner. Simple planning lacks an automatic reflection and error correction mechanism for incorrect tool calls, resulting in low accuracy and slow performance improvement at the strategy level. Summary of the Invention
[0005] This application provides a training method, apparatus, device, and storage medium for a strategy planning model, which can solve the problem of poor planning ability of strategy planning models for long dialogues or complex tasks. The technical solution is as follows: Firstly, a method for training a strategy planning model is provided, the method comprising: Acquire task data from long-term user interactions; The task data is decomposed using a strategy planning model based on a large language model to obtain a sequence of subtasks. The subtask sequence is processed using the basic large model of the large language model to obtain failure execution results and success execution results; The failed execution result is corrected to determine the execution result after correction. Based on the successful execution result, and / or, correct the processed execution result, to obtain training sample data; Based on the training sample data, the policy planning model is updated and trained to obtain a trained policy planning model, and a fine-tuned large language model is obtained based on the trained policy planning model.
[0006] In one possible implementation, the strategy planning model utilizing a large language model performs task decomposition processing on the task data to obtain a sequence of subtasks, including: The task data is input into the strategy planning model to obtain historical cross-session reference information related to the task data in a preset cross-session database; Based on the historical cross-session reference information, the task data is decomposed to obtain the sub-task sequence; Based on the subtask sequence, the priority of each subtask and the corresponding tool call parameters are determined.
[0007] In one possible implementation, the step of performing execution processing on the subtask sequence to obtain failure execution results and success execution results further includes: The subtask sequence is processed using a preset executor. Based on the results of the execution process, we obtain the results of failed execution and successful execution.
[0008] In one possible implementation, the step of correcting the failed execution result to determine the corrected execution result includes: The failed execution results are corrected to obtain a corrected subtask sequence; The modified subtask sequence is input into the strategy planning model until the successful execution result corresponding to the modified subtask sequence is obtained; The successful execution result corresponding to the corrected subtask sequence is determined as the execution result after correction.
[0009] In one possible implementation, obtaining training sample data based on the successful execution result and / or correcting the processed execution result includes: Based on the successful execution result, and / or, the corrected execution result, the execution trajectory of the positive example task is determined; The training sample data is obtained based on the execution trajectory of the positive example task.
[0010] In one possible implementation, the method further includes: Obtain the current task data from the user input to be processed; The current task data is input into the fine-tuned large language model, so that the trained policy planning model can be used to obtain cross-session reference information related to the current task data from the preset cross-session database, and based on the cross-session reference information, the current task data is decomposed to obtain the sub-task sequence and the corresponding tool call parameters. Based on the subtask sequence and the corresponding tool call parameters, the response data output to the user is obtained using the basic large model of the fine-tuned large language model.
[0011] Secondly, a training device for a strategy planning model is provided, the device comprising: The acquisition unit is used to acquire task data of long-term interactions input by the user; The decomposition unit is used to perform task decomposition processing on the task data using the strategy planning model of the large language model to obtain a sequence of subtasks. An execution unit is used to perform execution processing on the subtask sequence using the basic large model of the large language model, so as to obtain failure execution results and success execution results; The correction unit is used to correct the failed execution result in order to determine the execution result after correction. The obtaining unit is used to obtain training sample data based on the successful execution result and / or, after correcting the processed execution result. The training unit is used to update and train the policy planning model based on the training sample data to obtain a trained policy planning model, and then to obtain a fine-tuned large language model based on the trained policy planning model.
[0012] Thirdly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the aspects and any possible implementations described above.
[0013] Fourthly, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.
[0014] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.
[0015] The beneficial effects of the technical solution provided in this application include at least the following: As can be seen from the above technical solution, the embodiments of this application can obtain task data of long-term interaction input by the user, and then use the strategy planning model of the large language model to decompose the task data to obtain a sequence of sub-tasks. Using the basic large model of the large language model, the sub-task sequence is executed to obtain failure execution results and success execution results. The failure execution results are corrected to determine the execution result after correction. Based on the success execution results and the execution result after correction, training sample data is obtained. Based on the training sample data, the strategy planning model is updated and trained to obtain a trained strategy planning model. Based on the trained strategy planning model, a fine-tuned large language model is obtained. Since the strategy planning model can be updated and trained using the success execution results and the execution result after correction generated during the long-term interaction, the continuous autonomous fine-tuning of the large language model is realized. This optimizes the efficiency of strategy learning and the robustness of hierarchical task decomposition in long-term multi-turn interaction, reduces the error rate of tool use, and improves the ability to deal with complex instructions. Thus, the consistency of the performance and inference of the large language model in long-term multi-turn interaction can be guaranteed.
[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart illustrating a training method for a strategy planning model provided in one embodiment of this application; Figure 2 This is a schematic diagram of a training method for a strategy planning model provided in another embodiment of this application; Figure 3 This is a structural block diagram of a training device for a strategy planning model provided in another embodiment of this application; Figure 4 This is a block diagram of an electronic device used to implement the training method of the strategy planning model in the embodiments of this application. Detailed Implementation
[0019] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0020] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.
[0021] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0022] Currently, large language models are applied to long-term, multi-turn interaction scenarios and have the ability to call external tools. Such long-term interactions involve continuous tasks across rounds and sessions, requiring the model to have memory, planning, and continuous learning capabilities.
[0023] Typically, related technical solutions include long-range dialogue systems based on limited context windows and external vector retrieval, such as RAG, and continuous learning solutions that use batch data or online reinforcement learning (RL) to fine-tune the entire underlying large model.
[0024] However, the solutions of related technologies still have problems such as poor consistency and insufficient ability to follow complex instructions, reliance on context windows or simple text retrieval to store historical information, high error rate in tool use, and low efficiency in strategy evolution.
[0025] Therefore, there is an urgent need for a training method for policy planning models that can enable the continuous autonomous evolution of large language models, optimize the efficiency of policy learning and the robustness of hierarchical task decomposition, thereby ensuring the continuous and significant improvement of the performance of large language models.
[0026] Please refer to Figure 1 This document illustrates a flowchart of a training method for a strategy planning model provided in one embodiment of this application. This training method for the strategy planning model, applied to long-range interaction scenarios of large language models, may specifically include: Step 101: Obtain task data for long-term user interactions.
[0027] Step 102: Using the strategy planning model of the large language model, the task data is decomposed to obtain a sequence of subtasks.
[0028] Step 103: Using the basic large model of the large language model, perform execution processing on the sub-task sequence to obtain failure execution results and success execution results.
[0029] Step 104: Correct the failed execution result to determine the corrected execution result.
[0030] Step 105: Based on the successful execution result, and / or, correct the processed execution result to obtain training sample data.
[0031] Step 106: Based on the training sample data, update the policy planning model to obtain a trained policy planning model, and then obtain a fine-tuned large language model based on the trained policy planning model.
[0032] It's important to note that long-term interaction task data can be user-inputted question data spanning multiple sessions at certain intervals. For example, user-inputted travel planning task data involves user travel preferences. Several days later, user-inputted attraction recommendation task data will be used, and the strategy planning model for processing the current task data will refer to the previous user travel preferences. Here, both the user-inputted travel planning task data and the attraction recommendation task data can be long-term interaction task data.
[0033] In this way, by acquiring task data from long-term user interactions, the strategy planning model of a large language model can be used to decompose the task data into sub-task sequences. The sub-task sequences are then executed using the basic large language model to obtain failure and success results. The failure results are corrected to determine the corrected execution result. Based on the success and corrected execution results, training sample data is obtained. The strategy planning model is then updated and trained based on this training sample data to obtain a trained strategy planning model. Finally, a fine-tuned large language model is obtained based on this trained strategy planning model. Because the strategy planning model can be updated and trained using the success and corrected execution results generated during long-term interactions, continuous autonomous fine-tuning of the large language model is achieved. This optimizes the efficiency of strategy learning and the robustness of hierarchical task decomposition in long-term multi-turn interactions, reduces the error rate of tool usage, and improves the ability to handle complex instructions. Therefore, the consistency of the large language model's performance and inference in long-term multi-turn interactions can be guaranteed.
[0034] Optionally, in one possible implementation of this embodiment, in step 102, firstly, the task data can be input into the strategy planning model to obtain historical cross-session reference information related to the task data in a preset cross-session database. Secondly, based on the historical cross-session reference information, the task data can be decomposed to obtain the sub-task sequence. Thirdly, based on the sub-task sequence, the priority of each sub-task and the corresponding tool invocation parameters can be determined.
[0035] In this implementation, historical cross-session reference information can include structured cross-session reference information. This historical cross-session reference information can include cross-session user preferences, task status, and other information.
[0036] In one specific implementation of this approach, the task data can be decomposed hierarchically based on the historical cross-session reference information to obtain a structured subtask sequence, and then the subtask sequence can be obtained based on the structured subtask sequence.
[0037] In this implementation, the tool invocation parameters can be the parameters of the tool invoked by the subtask. For example, tool invocation parameters may include invoking a search engine, querying the current time, executing program code, etc.
[0038] In this way, the task data can be decomposed into a sequence of subtasks by using historical cross-session reference information related to the task data obtained from a preset cross-session database. This allows the subsequent policy planning model to be updated and trained based on the execution results of the subtask sequence. By referring to relevant information in cross-sessions, task decomposition can be achieved, avoiding the loss or neglect of key information in long-term interactions and further ensuring the consistency of the large language model's inference before and after long-term interactions.
[0039] Optionally, in one possible implementation of this embodiment, in step 103, a preset executor can be used to perform execution processing on the subtask sequence, and then the failure execution result and the success execution result can be obtained based on the result of the execution processing.
[0040] In this implementation, the pre-defined executor can be an executor that can be called by a large language model.
[0041] In one specific implementation of this approach, the subtask sequence is input into the base model of the large language model to obtain the failure and success results of the subtask sequence.
[0042] In this implementation, failure results can include subtask sequence execution failures, inconsistent execution results across sessions, etc.
[0043] In another specific implementation of this approach, firstly, the subtask sequence is input into the base model of the large language model, or into a preset executor, to obtain the execution result. Secondly, the execution result is evaluated using the reflection and error correction module of the large language model to obtain the failure and success results of the subtask sequence.
[0044] In this way, by utilizing a pre-defined executor, the failure and success results of the subtask sequence can be obtained, thereby improving the accuracy of the execution result evaluation.
[0045] It should be noted that the specific implementation process provided in this embodiment can be combined with various specific implementation processes provided in the foregoing implementation methods to implement the training method of the policy planning model in this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.
[0046] Optionally, in one possible implementation of this embodiment, in step 104, firstly, the failed execution result can be corrected to obtain a corrected subtask sequence. Secondly, the corrected subtask sequence can be input into the strategy planning model until a successful execution result corresponding to the corrected subtask sequence is obtained. Thirdly, the successful execution result corresponding to the corrected subtask sequence is determined as the corrected execution result.
[0047] In a specific implementation of this approach, firstly, the reflection and error correction module of the large language model can be used to correct failed execution results, obtaining a corrected subtask sequence. Secondly, the corrected subtask sequence can be input into the strategy planning model to plan a new subtask sequence corresponding to the corrected subtask sequence. This new subtask sequence is then executed to obtain successful and failed execution results. Thirdly, for failed execution results, the aforementioned correction and planning steps can be repeated until a successful execution result corresponding to the corrected subtask sequence is obtained. Finally, the successful execution result corresponding to the corrected subtask sequence is determined as the corrected execution result.
[0048] In this way, by correcting the failed execution results, successful execution results can be identified, and more positive examples can be obtained as subsequent training sample data, thereby enhancing the training sample data and increasing its richness.
[0049] It should be noted that the specific implementation process provided in this embodiment can be combined with various specific implementation processes provided in the foregoing implementation methods to implement the training method of the policy planning model in this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.
[0050] Optionally, in one possible implementation of this embodiment, in step 105, firstly, the execution trajectory of the positive example task can be determined based on the successful execution result and / or the modified execution result. Secondly, the training sample data can be obtained based on the execution trajectory of the positive example task.
[0051] In this implementation, the execution trajectory of a positive example task can be the execution trajectory of a successfully executed task.
[0052] In this way, by using the execution trajectories of positive examples as training sample data for training the policy planning model, the reliability and performance of the trained policy planning model can be further improved.
[0053] It should be noted that the specific implementation process provided in this embodiment can be combined with various specific implementation processes provided in the foregoing implementation methods to implement the training method of the policy planning model in this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.
[0054] Optionally, in one possible implementation of this embodiment, after step 106, it may further be done as follows: First, the current task data input by the user to be processed is obtained. Then, the current task data can be input into the fine-tuned large language model to utilize the trained policy planning model to obtain cross-session reference information related to the current task data from a preset cross-session database. Based on the cross-session reference information, the current task data is decomposed to obtain the sub-task sequence and corresponding tool invocation parameters. Finally, based on the sub-task sequence and corresponding tool invocation parameters, the basic large model of the fine-tuned large language model can be used to obtain response data output to the user.
[0055] In this implementation, the current task data input by the user can include task data and question data of the current session in long-term interaction.
[0056] In one specific implementation of this approach, after inputting the current task data into the fine-tuned large language model, the trained policy planning model can first retrieve cross-session reference information related to the current task data from a pre-defined cross-session database. Based on this cross-session reference information, the trained policy planning model can perform task decomposition on the current task data to obtain the sub-task sequence and the corresponding tool invocation parameters. Then, based on the sub-task sequence and the corresponding tool invocation parameters, the base model of the fine-tuned large language model can be used to obtain the response data output to the user.
[0057] One specific implementation method is to obtain the response data output to the user by using the preset executor corresponding to the fine-tuned large language model, based on the sub-task sequence and the corresponding tool call parameters.
[0058] In this implementation, the response data can be the execution result of the task. The response data may include the solution data of the task data, the response data to the question data pair, etc.
[0059] In one specific implementation of this approach, the current task data, the subtask sequence of the current task data, and the response data can be structured, and the results of the structured processing can be stored in a preset cross-session database to update the cross-session memory data in the preset cross-session database.
[0060] In this way, a new policy planning model can be obtained by updating the training of a lightweight policy planning model, and then a fine-tuned large language model can be obtained based on the updated policy planning model. This eliminates the need to adjust and train the basic large model with a huge number of parameters, which can improve the training speed and stability of the large language model, reduce the cost of online incremental training, and improve the inference performance and reliability of the fine-tuned large language model.
[0061] It should be noted that the specific implementation process provided in this embodiment can be combined with various specific implementation processes provided in the foregoing implementation methods to implement the training method of the policy planning model in this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.
[0062] To better understand the methods of the embodiments of this application, the methods of the embodiments of this application will be described below in conjunction with the accompanying drawings and specific application scenarios.
[0063] Figure 2 This is a flowchart illustrating a training method for a strategy planning model provided in another embodiment of this application, as shown below. Figure 2 As shown.
[0064] Step 201: Obtain task data for long-term user interaction input.
[0065] Step 202: Input the long-range interaction task data into the strategy planning model of the large language model to obtain the sub-task sequence and corresponding tool call parameters after hierarchical decomposition.
[0066] In this embodiment, after receiving the input long-range interaction task data, the strategy planning model can retrieve relevant structured historical cross-session reference information from the preset cross-session database, and combine it with hierarchical task decomposition to decompose the task data to obtain a structured sub-task sequence, thereby determining the priority of the sub-tasks and the tool call parameters.
[0067] Step 203: Input the subtask sequence and corresponding tool call parameters into the basic large model of the large language model to obtain the failure execution result and / or the success execution result.
[0068] In this embodiment, the action executor corresponding to the large language model can also be used to execute the subtask sequence to obtain the execution result. The execution result may include a failed execution result and / or a successful execution result.
[0069] Step 204: Correct the failed execution result to obtain the corrected execution result.
[0070] In this embodiment, the failure execution result can include execution failure or execution inconsistency. The failure execution result is corrected to generate a corrected subtask sequence. The corrected subtask sequence can be sent to the policy planning model for retry until a successfully corrected subtask sequence is obtained. Based on the successfully corrected subtask sequence, the task execution trajectory is obtained, which is the corrected execution result.
[0071] Step 205: Based on the successful execution result, and / or, correct the processed execution result to obtain training sample data for training the policy planning model.
[0072] In this embodiment, based on the successful execution result and / or the corrected execution result, the execution trajectory of the positive example task is determined, and the execution trajectory of the positive example task is used as training sample data.
[0073] Understandably, the execution trajectories of tasks that ultimately succeed in historical reasoning within the large language model, and / or the execution trajectories of tasks that successfully correct errors, can be used for the evolution of the policy planning model.
[0074] Furthermore, the large language model can utilize the reflection and error correction module to store or update the structured new knowledge, updated state, and user information generated in each current session to a preset cross-session database for cross-session memory.
[0075] Step 206: Input the training sample data into the policy planning model to update and train the policy planning model, and obtain the trained policy planning model.
[0076] Step 207: Based on the trained policy planning model, obtain the fine-tuned large language model.
[0077] Step 208: Obtain the task data for the current session to be processed.
[0078] In this embodiment, the task data of the current session to be processed may include the current task data input by the user to be processed.
[0079] Step 209: Using the fine-tuned large language model, perform inference processing on the task data of the current session to obtain the response data output to the user.
[0080] In this embodiment, the task data from the previous session can be input into the fine-tuned large language model. The trained strategy planning model can then be used to obtain cross-session reference information related to the current task data from a preset cross-session database. Based on the cross-session reference information, the current task data can be decomposed to obtain a sequence of sub-tasks and corresponding tool call parameters. The sequence of sub-tasks and corresponding tool call parameters can then be input into the base model of the fine-tuned large language model to obtain response data output to the user.
[0081] Understandably, here, user-input task data enters the policy planning model. The policy planning model retrieves structured memory from the cross-session database and performs hierarchical task decomposition to generate a sequence of sub-tasks. After execution by the executor or the base model, the sub-task sequence generates execution result feedback, which is sent to the reflection and correction module. If the feedback is incorrect, the reflection and correction module generates a corrected sub-task sequence and sends it back to the policy planning model for retry, forming a correction loop. The successful task execution trajectory and the corrected task execution trajectory are used for continuous incremental training of the policy planning model. Simultaneously, the reflection and correction module can update key state changes to the cross-session memory.
[0082] For example, taking a user's multi-day travel planning task as an example, historical cross-session reference information includes the user profile indicating a fear of heights, the core task status being a draft Shandong itinerary, and tool calls. The user mentions "fear of heights" in the first session, and this is stored in the cross-session database. In the second session several days later, when the user asks for travel recommendations, the strategy planning model using a large language model retrieves the "fear of heights" tag and guides the planning model to exclude high-risk attractions involving climbing, thus maintaining consistency in user preferences across sessions. When querying the opening hours of a specific attraction in the itinerary, the tool call fails due to incorrect parameters. After automatically identifying the failure, the accurate parameters are extracted from the structured task status stored in the cross-session database, generating corrected call instructions and task sequences, and retrying in the strategy planning model until successful execution. This complete "failure-correction-success" task execution trajectory and the successfully executed task execution trajectory can be used for incremental training of the strategy planning model, enabling its planned strategies to handle tool call parameters more robustly in the future.
[0083] Thus, by adopting the scheme in this embodiment, a continuous autonomous evolution framework of "hierarchical planning - policy learning - continuous reflection" can be constructed. This framework focuses long-range interactive policy learning on a lightweight policy planning model, and enables the continuous autonomous evolution of this model through high-quality data generated by automatic error correction. Policy decision-making power is separated to a lightweight policy planning model, which performs incremental fine-tuning by learning high-quality success trajectories, improving the efficiency and targeting of policy evolution. Complex long-range tasks are decomposed into a sequence of executable sub-tasks, and high-dimensional semantic information is injected by retrieving structured long-range memories to ensure consistency in decision-making.
[0084] Furthermore, the scheme in this embodiment can automatically generate corrected sub-task sequences for retries when tool calls or task execution fail. Only successfully corrected trajectories are adopted for training the policy model, ensuring the quality of training data. Moreover, by introducing an automatic reflection and error correction closed loop, it ensures that failed calls can be internally repaired and retried. At the same time, the policy planning model uses only high-quality successful trajectories that have been verified through error correction for incremental training, accurately and efficiently optimizing the planning and tool call strategies, avoiding the interference of erroneous data on the strategy in traditional methods, and significantly improving the accuracy of tool usage and the completion rate of complex tasks.
[0085] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0086] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0087] Figure 3 This paper shows a structural block diagram of a training device for a strategy planning model provided in one embodiment of the present application, as follows: Figure 3 As shown. The training device 300 for the strategy planning model in this embodiment may include an acquisition unit 301, a decomposition unit 302, an execution unit 303, a correction unit 304, an acquisition unit 305, and a training unit 306. The acquisition unit 301 is used to acquire long-range interactive task data input by the user; the decomposition unit 302 is used to decompose the task data using the strategy planning model of the large language model to obtain a sequence of sub-tasks; the execution unit 303 is used to execute the sequence of sub-tasks using the basic large model of the large language model to obtain failure execution results and success execution results; the correction unit 304 is used to correct the failure execution results to determine the corrected execution results; the acquisition unit 305 is used to obtain training sample data based on the success execution results and / or the corrected execution results; the training unit 306 is used to update the strategy planning model based on the training sample data to obtain a trained strategy planning model, and then obtain a fine-tuned large language model based on the trained strategy planning model.
[0088] Optionally, in one possible implementation of this embodiment, the decomposition unit 302 is used to input the task data into the strategy planning model to obtain historical cross-session reference information related to the task data in a preset cross-session database; based on the historical cross-session reference information, the task data is decomposed to obtain the sub-task sequence; based on the sub-task sequence, the priority of each sub-task and the corresponding tool call parameters are determined.
[0089] Optionally, in one possible implementation of this embodiment, the execution unit 303 is used to perform execution processing on the subtask sequence using a preset executor; and obtain failure execution results and success execution results based on the execution processing results.
[0090] Optionally, in one possible implementation of this embodiment, the correction unit 304 is used to correct the failed execution result to obtain a corrected subtask sequence; input the corrected subtask sequence into the strategy planning model until a successful execution result corresponding to the corrected subtask sequence is obtained; and determine the successful execution result corresponding to the corrected subtask sequence as the corrected execution result.
[0091] Optionally, in one possible implementation of this embodiment, the obtaining unit 305 is used to determine the execution trajectory of the positive example task based on the successful execution result and / or the modified execution result; and to obtain the training sample data based on the execution trajectory of the positive example task.
[0092] Optionally, in one possible implementation of this embodiment, the training unit 306 is used to acquire the current task data of the user input to be processed; input the current task data into the fine-tuned large language model, so as to use the trained policy planning model to obtain cross-session reference information related to the current task data from a preset cross-session database, and perform task decomposition processing on the current task data based on the cross-session reference information to obtain the sub-task sequence and the corresponding tool call parameters; based on the sub-task sequence and the corresponding tool call parameters, use the base model of the fine-tuned large language model to obtain the response data output to the user.
[0093] In this embodiment, the acquisition unit acquires long-term interaction task data input by the user; the decomposition unit uses the strategy planning model of the large language model to decompose the task data into a sequence of sub-tasks; the execution unit uses the basic large model of the large language model to execute the sequence of sub-tasks to obtain failure and success results; the correction unit corrects the failure results to determine the corrected execution result; the acquisition unit obtains training sample data based on the success results and / or the corrected execution results; and the training unit obtains training sample data based on the training sample data. According to the method, the strategy planning model is updated and trained to obtain a trained strategy planning model. Based on the trained strategy planning model, a fine-tuned large language model is obtained. Since the strategy planning model can be updated and trained by utilizing the successful execution results and the corrected execution results generated during long-term interaction, the large language model can be continuously and autonomously fine-tuned. This optimizes the efficiency of strategy learning and the robustness of hierarchical task decomposition in long-term multi-turn interaction, reduces the error rate of tool use, and improves the ability to deal with complex instructions. Thus, the consistency of the performance and inference of the large language model in long-term multi-turn interaction can be guaranteed.
[0094] The technical solution of this application involves the collection, storage, use, processing, transmission, provision, and disclosure of user personal information, such as user image and attribute data, which comply with relevant laws and regulations and do not violate public order and good morals.
[0095] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.
[0096] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.
[0097] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0098] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0099] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the training method for a policy programming model. For example, in some embodiments, the training method for a policy programming model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the training method for the policy programming model described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the training method for a policy programming model by any other suitable means (e.g., by means of firmware).
[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0101] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0102] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0104] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0105] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0106] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.
[0107] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A training method for a strategy planning model, characterized in that, The method includes: Acquire task data from long-term user interactions; The task data is decomposed using a strategy planning model based on a large language model to obtain a sequence of subtasks. The subtask sequence is processed using the basic large model of the large language model to obtain failure execution results and success execution results; The failed execution result is corrected to determine the execution result after correction. Based on the successful execution result, and / or, correct the processed execution result, to obtain training sample data; Based on the training sample data, the policy planning model is updated and trained to obtain a trained policy planning model, and a fine-tuned large language model is obtained based on the trained policy planning model.
2. The method according to claim 1, characterized in that, The strategy planning model utilizing a large language model decomposes the task data to obtain a sequence of sub-tasks, including: The task data is input into the strategy planning model to obtain historical cross-session reference information related to the task data in a preset cross-session database; Based on the historical cross-session reference information, the task data is decomposed to obtain the sub-task sequence; Based on the subtask sequence, the priority of each subtask and the corresponding tool call parameters are determined.
3. The method according to claim 1, characterized in that, The step of performing execution processing on the subtask sequence to obtain failure execution results and success execution results also includes: The subtask sequence is processed using a preset executor. Based on the results of the execution process, the failure execution result and the success execution result are obtained.
4. The method according to claim 1, characterized in that, The step of correcting the failed execution result to determine the corrected execution result includes: The failed execution results are corrected to obtain a corrected subtask sequence; The modified subtask sequence is input into the strategy planning model until the successful execution result corresponding to the modified subtask sequence is obtained; The successful execution result corresponding to the corrected subtask sequence is determined as the execution result after correction.
5. The method according to claim 1, characterized in that, The step of obtaining training sample data based on the successful execution result and / or the corrected execution result includes: Based on the successful execution result, and / or, the corrected execution result, the execution trajectory of the positive example task is determined; The training sample data is obtained based on the execution trajectory of the positive example task.
6. The method according to any one of claims 1-5, characterized in that, The method further includes: Obtain the current task data from the user input to be processed; The current task data is input into the fine-tuned large language model, so that the trained policy planning model can be used to obtain cross-session reference information related to the current task data from the preset cross-session database, and based on the cross-session reference information, the current task data is decomposed to obtain the sub-task sequence and the corresponding tool call parameters. Based on the subtask sequence and the corresponding tool call parameters, the response data output to the user is obtained using the basic large model of the fine-tuned large language model.
7. A training device for a strategy planning model, characterized in that, The device includes: The acquisition unit is used to acquire task data for long-term user interactions. The decomposition unit is used to perform task decomposition processing on the task data using the strategy planning model of the large language model to obtain a sequence of subtasks. An execution unit is used to perform execution processing on the subtask sequence using the basic large model of the large language model, so as to obtain failure execution results and success execution results; The correction unit is used to correct the failed execution result in order to determine the execution result after correction. The obtaining unit is used to obtain training sample data based on the successful execution result and / or, after correcting the processed execution result. The training unit is used to update and train the policy planning model based on the training sample data to obtain a trained policy planning model, and then to obtain a fine-tuned large language model based on the trained policy planning model.
8. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.