A method, device and medium for optimizing instruction following capability of a large language model

Abstract

The present application relates to a method, device and medium for optimizing the instruction-following ability of a large language model, and belongs to the field of artificial intelligence technology. The present application adopts a self-game training framework, and uses a large language model to play the role of a generator model and a perfecter model to perform self-game; wherein, in each round of self-game training, the generator model generates a response to the given instruction data; the perfecter model judges and scores each response and finds the response that fails to execute; using the self-improvement strategy of tree search, the perfecter model corrects the response that failed to execute and obtains the corrected response; constructs generator training data to iteratively optimize the generator model; constructs perfecter training data to iteratively optimize the perfecter model. The self-game framework adopted in the present application uses tree search and self-improvement to generate effective preference data, which can eliminate interference factors and optimize the instruction-following ability of large-scale language models.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a method, device and medium for optimizing the instruction-following capability of a large language model. Background Art

[0002] In recent years, large-scale language models (LLMs) have made remarkable progress and have demonstrated strong capabilities in more and more tasks. As LLMs are applied in more and more real-world scenarios, their ability to follow instructions becomes crucial, especially the ability to follow complex instructions with multiple constraints. Failure to accurately follow user instructions may even bring major security risks in actual application scenarios.

[0003] In the instruction-following task, a prominent feature is that subtle differences can determine the success of the task, so preference optimization is a commonly used method to improve instruction-following ability. However, existing methods usually directly sample multiple independent responses from the model when constructing preference data. This method may introduce content that is irrelevant to instruction-following and inevitably introduce interference factors. Since these interference factors are irrelevant to the success of the instruction-following task, they will affect the model's learning of the key differences that truly determine the success of instruction-following. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a method, device and medium for optimizing the instruction-following capability of a large language model, aiming to generate effective preference data and eliminate interference factors.

[0005] In a first aspect of the present application, a method for optimizing instruction compliance capability of a large language model is provided, comprising:

[0006] A self-game training framework is used to perform self-game by using a large language model to act as a generator model and a perfecter model.

[0007] The following steps are performed in each round of self-game training:

[0008] The generator model generates responses to the given instruction data;

[0009] The Perfecter model judges and scores each response and finds responses that fail to execute;

[0010] By using the self-improvement strategy of tree search, the improver model corrects the reply of the failed execution to obtain a corrected reply;

[0011] Based on the corrected responses and the responses to the failed executions, generator training data is constructed to iteratively optimize the generator model; based on the responses to the failed executions and data in the self-improvement strategy of the tree search, perfecter training data is constructed to iteratively optimize the perfecter model.

[0012] Optionally, the self-improvement strategy using tree search, the improver model corrects the reply of the execution failure, and the corrected reply includes:

[0013] The failed instruction data is used as the root node of the search, and the tree nodes are expanded from the root node, and each expanded child node corresponds to a possible correction result;

[0014] The correctness of each correction result is judged and evaluated; the search process ends when the perfecter model corrects the correct response;

[0015] The corrected reply will be used as the corrected reply.

[0016] Optionally, the self-improvement strategy using tree search, the improver model corrects the reply of the execution failure, and the corrected reply includes:

[0017] Using a self-improvement strategy of a depth-first search tree search, the improver model corrects the failed response to obtain a corrected response, and the execution method is as follows:

[0018] Start from the root node and explore the unvisited child nodes downwards. After entering a new node, continue exploring downwards until the maximum exploration depth.

[0019] When a node is reached and the maximum search depth is reached, backtrack to the previous node and continue to explore other child nodes;

[0020] The search process ends until the perfecter model corrects the correct response.

[0021] Optionally, the self-improvement strategy using tree search, the improver model corrects the reply of the execution failure, and the corrected reply includes:

[0022] The self-improvement strategy of breadth-first search tree search is adopted. The improver model corrects the reply of the failed execution to obtain the corrected reply. The execution method is as follows:

[0023] Starting from the root node, create a queue to store the nodes at each level;

[0024] Put the root node into the queue, and then start the loop: dequeue the node, visit all the unvisited direct child nodes of the corresponding node, and put the child nodes into the queue;

[0025] The search process ends when the perfecter model corrects the correct response.

[0026] Optionally, the self-improvement strategy using tree search, the improver model corrects the reply of the execution failure, and the corrected reply includes:

[0027] During the tree search, the output of each node is checked and corrected by a self-improvement mechanism; the self-improvement mechanism analyzes the errors in the output and proposes a corrected response;

[0028] For each revised response, its quality and accuracy are re-evaluated and the evaluation results are fed back into the tree search to revise the response with the new evaluation results.

[0029] Optionally, constructing generator training data based on the corrected response and the failed response to iteratively optimize the generator model includes:

[0030] Pairing the corrected response with the failed response to generate training data pairs;

[0031] Define a loss function that directly reflects how well the model has learned the preferences;

[0032] Using the generated training data pairs and the loss function, training the generator model through preference learning;

[0033] After each round of training, the validation set is used to evaluate the performance of the model, and the model parameters or training strategy are adjusted based on the evaluation results. The optimized generator model is used for the next round.

[0034] Optionally, constructing perfecter training data based on the data in the self-improvement strategy of the failed execution and the tree search to iteratively optimize the perfecter model includes:

[0035] collecting data from the tree search strategy, including responses to failed executions and all potential fixes generated during the search; distinguishing all potential fixes into valid fixes and invalid fixes;

[0036] Pair failed responses with effective corrections to form perfect training data pairs;

[0037] Generate correction candidates; for each generated correction candidate, calculate the probability of being accepted; only when the correction candidate meets the set criteria, it is accepted as training data;

[0038] Train the perfecter model using the received training data;

[0039] After each round of training, the performance of the model is evaluated, and the model parameters or training strategy are adjusted as needed, and the optimized perfecter model is used for the next round.

[0040] Optionally, the instruction-following capability includes objective constraints and subjective constraints; the subjective constraints include emotional factors, scene adaptability factors, and contextual relevance factors.

[0041] According to a second aspect of the present application, there is provided a device for optimizing the instruction compliance capability of a large language model, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, a method for optimizing the instruction compliance capability of a large language model is implemented according to any one of the above-described methods.

[0042] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for optimizing the instruction compliance capability of a large language model according to any one of the above methods is implemented.

[0043] The method for optimizing the instruction-following ability of a large language model provided in the present application adopts a self-game training framework, and uses a large language model to play the role of a generator model and a perfecter model to perform self-game; wherein, in each round of self-game training, the generator model generates a response to the given instruction data; the perfecter model judges and scores each response and finds the response that fails to execute; using the self-perfection strategy of tree search, the perfecter model corrects the response that fails to execute and obtains the corrected response; based on the corrected response and the response that fails to execute, the generator training data is constructed to iteratively optimize the generator model; based on the response that fails to execute and the data in the self-perfection strategy of tree search, the perfecter training data is constructed to iteratively optimize the perfecter model. The self-game framework adopted in the present application uses tree search and self-perfection to generate effective preference data, which can eliminate interference factors, greatly exceeds the effect of preference data constructed by independent sampling, and optimizes the instruction-following ability of large-scale language models.

[0044] In addition, the present application also provides devices and media that optimize the ability to follow large language model instructions with the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0046] Figure 1A flowchart of a specific implementation of the method for optimizing the instruction compliance capability of a large language model provided in the present application;

[0047] Figure 2 A schematic diagram of a specific implementation of the method for optimizing the instruction compliance capability of a large language model provided in the present application;

[0048] Figure 3 Schematic diagram of the process of iterative optimization of the generator model;

[0049] Figure 4 A schematic diagram of the process of iterative optimization of the Perfector model;

[0050] Figure 5 A structural block diagram of the device for optimizing the large language model instruction compliance capability provided by the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0052] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0053] The instruction-following ability of a large-scale language model refers to the ability of the model to perform a specific task according to the instructions given by the user. This ability involves understanding the user's instructions, identifying the requirements and context of the task, and generating an answer or output that meets the expectations. Many studies have evaluated this ability from different perspectives. As LLMs are more widely used, the tasks they need to deal with are becoming more and more complex, and these instructions often contain multiple constraints. Therefore, some works have also proposed evaluation benchmarks for these complex instruction-following abilities. At the same time, some works have also explored how to further improve the instruction-following ability of the model. In the prior art, attempts have been made to construct more complex supervised fine-tuning datasets, and further DPO optimization has been performed. Or the dataset is augmented and preference data is constructed by perturbing the original instructions. However, these methods do not take into account the interference factors in the preference data, and interference factors that are not related to the success of instruction-following will greatly affect the performance of the model.

[0054] In view of this, the present application provides a method for optimizing the instruction following ability of a large language model. The method adopts a self-play training framework (SPAR) to perform self-play by having a large language model act as a generator model and a perfecter model.

[0055] Among them, the generator model is responsible for executing the given complex instructions. In this role, the generator model attempts to generate a preliminary output or solution based on its existing knowledge and understanding. The goal of the generator model is to understand and execute the given instructions as accurately as possible. The improver model is responsible for judging the responses of the generator model and further improving the responses that failed to execute. In the framework of self-game, these two roles are played by the same model at different stages. This structure allows the model to learn from its own behavior and continuously improve its capabilities through internal criticism and self-improvement mechanisms. Through this improvement method, preference data that excludes interference content and highlights key factors can be obtained to achieve effective improvement in instruction compliance.

[0056] Figure 1 A flowchart showing a specific implementation of the method for optimizing the instruction compliance capability of a large language model provided by the present application, combined with Figure 1 , in each round of self-game training, the method includes the following specific steps:

[0057] S101: The generator model generates a response to the given instruction data.

[0058] The generator model is responsible for generating a preliminary response based on the given input instruction data. This can be any form of instruction, such as text commands, user queries, etc. The generator model needs to produce corresponding outputs based on these instructions.

[0059] S102: The Perfector Model judges and scores each response and finds responses that fail to execute.

[0060] The Perfecter model evaluates and scores each response generated by the Generator model. This step is key to identifying responses that failed to execute or did not meet expectations. The Perfecter model determines whether each response successfully executed the instruction using evaluation criteria such as accuracy, relevance, sentiment, etc.

[0061] S103: Using the self-improvement strategy of tree search, the improver model corrects the reply of the failed execution to obtain a corrected reply.

[0062] For the responses that are identified as failed, the improver model uses a tree search strategy to improve itself. Through systematic exploration, it finds the best way to correct the failed response and finally obtains the corrected response.

[0063] S104: Based on the corrected responses and the responses to the failed executions, generator training data is constructed to iteratively optimize the generator model; based on the responses to the failed executions and data in the self-improvement strategy of the tree search, perfecter training data is constructed to iteratively optimize the perfecter model.

[0064] This step converts the previous process into a training opportunity to further optimize the two models:

[0065] Iterative optimization of the generator model: Based on the corrected responses and the original failed responses, a dataset is constructed for training the generator model. This helps the generator model learn how to improve from its mistakes to produce higher quality preliminary responses.

[0066] Iterative optimization of the perfecter model: At the same time, the perfecter model is trained and optimized using data generated from failed replies and the tree search self-improvement strategy. This helps the perfecter model better identify failed replies and improve the effectiveness of the correction strategy.

[0067] The above process forms a closed loop. Based on self-game and tree search, the performance of the two models is continuously improved through generation, evaluation, correction and retraining to improve the instruction-following ability of the large language model. This self-improvement is achieved by constantly comparing the results of different strategies and selecting the best path. The model uses the above technology to generate preference data, that is, data points that can show better instruction-following ability. Compared with traditional random or independent sampling, the method provided in this application can more accurately capture the key differences related to instruction-following and reduce the interference of irrelevant variables.

[0068] Figure 2 A schematic diagram showing a specific implementation of the method for optimizing the instruction compliance capability of a large language model provided by the present application, combined with Figure 2,The SPAR framework provided in this application supports the iterative optimization of ,models, starting from a powerful LLM and optimizing its ability to act as a ,generator and perfecter at the same time.

[0069] In each round of training, for each instruction, multiple responses are first sampled from the generator model. Then, for each response, the perfecter model is used to judge and score it to find the response that failed to execute. During the scoring process, a voting mechanism is used to improve the accuracy of the scoring and create RFT data for subsequent optimization of the perfecter model.

[0070] Next, using the tree search improvement strategy, the improver model can effectively correct the failed replies collected in the previous step, while the self-improvement mechanism avoids additional interference factors. Using the replies corrected by the tree search and the original failed replies, preference data without interference factors can be constructed to effectively improve the generator model. At the same time, using the data from the failed replies and the self-improvement strategy of the tree search, the improver's RFT data can be constructed to simultaneously improve the improver model.

[0071] It can be understood that the instruction-following capability in the present application includes objective constraints and subjective constraints; the subjective constraints include emotional factors, scene adaptability factors, and contextual relevance factors, which makes the scope of tasks involved in the present application wider.

[0072] The existing technology directly samples and generates preference data. The responses are obtained by independent sampling, have no relationship with each other, and have large differences in content. The method provided by this application takes into account the comparability of preference data for the first time. Based on the collected responses of failed executions, the self-improvement mechanism is used to correct only the responses of failed executions, which greatly reduces interference factors. Furthermore, using preference data without interference factors can help the model better learn key differences.

[0073] Through extensive experiments, the SPAR framework has been proven to be effective in surpassing traditional self-improvement techniques, especially in eliminating interference factors and optimizing data quality. These experiments not only verified the theoretical effect of SPAR, but also demonstrated its advantages in practical applications. According to the verification results, SPAR enables the generator to continuously generate high-quality responses, while improving the adaptability of the perfecter to the generator's capabilities, and further guiding and optimizing the generator's output. The method used in this application is significantly better than other self-improvement techniques that do not eliminate interference factors, and significantly improves the instruction-following ability of large language models.

[0074] It is understandable that in order to obtain preference data without interference factors, this application uses the self-improvement mechanism of LLM. At the same time, this application finds that the success rate of direct self-improvement is low, that is, the success rate of correcting failed instructions is low. Therefore, this application proposes a method combining tree search and self-improvement mechanism to effectively correct the response to failed instruction compliance and construct preference data without interference factors.

[0075] Specifically, using the self-improvement strategy of tree search, the perfecter model corrects the reply of the failed execution, and the corrected reply includes: taking the instruction data of the failed execution as the root node of the search, expanding the tree nodes from the root node, and each expanded child node corresponds to a possible correction result; judging the correctness of each correction result and evaluating it; the search process ends until the perfecter model corrects the correct reply; and the corrected reply is used as the corrected reply.

[0076] The self-improvement strategy using tree search may specifically adopt a self-improvement strategy based on a depth-first search or a self-improvement strategy based on a breadth-first search.

[0077] Among them, a self-improvement strategy of depth-first search tree search is adopted, and the improver model corrects the reply of the failed execution to obtain a corrected reply. The execution method is as follows:

[0078] Starting from the root node, explore the unvisited child nodes downward. After entering a new node, continue to explore downward until the maximum exploration depth. When a node is reached and the maximum exploration depth is reached, backtrack to the previous node and continue to explore other child nodes. The search process ends until the perfecter model corrects the correct response.

[0079] Among them, a self-improvement strategy of breadth-first search tree search is adopted, and the improver model corrects the reply of the failed execution to obtain a corrected reply. The execution method is as follows:

[0080] Starting from the root node, create a queue to store the nodes of each layer; put the root node into the queue, and then start the loop: dequeue the node, visit all the unvisited direct child nodes of the corresponding node, and put the child nodes into the queue; execute the loop until the perfecter model corrects the correct response, and the search process ends.

[0081] Using the self-improvement strategy of tree search, the improver model corrects the failed response to obtain the corrected response, including: during the tree search process, the output of each node is checked and corrected by the self-improvement mechanism; the self-improvement mechanism analyzes the errors in the output and proposes a corrected response; for each corrected response, its quality and accuracy are re-evaluated, and the evaluation results are fed back to the tree search, and the response is corrected with the new evaluation results.

[0082] The tree search strategy starts at the root node, which represents the output of the model when it initially tries to execute the instruction. The goal of the tree search is to explore all possible execution paths from the root node, each path representing a series of decisions or actions. At each node in the tree, the model generates a series of possible next actions. These actions may be different interpretations or implementations of the current instruction. Each action generates a new node and connects to the current node. An evaluation function is used to evaluate the effect of each node. During the tree search process, the output of each node is checked and corrected by the self-improvement mechanism. The self-improvement mechanism analyzes errors or deficiencies in the output and proposes improvements. For each corrected output, the model re-evaluates its quality and accuracy, ensuring that each correction moves towards correctly executing the instruction. The correction results can be fed back into the tree search to help the model make better decisions in future searches. After multiple corrections and evaluations, the model selects the final, optimal output as the solution to the task. This process enhances the model's understanding of the instructions and improves its ability to follow complex instructions.

[0083] This implementation effectively combines the advantages of tree search and self-improvement mechanisms to explore multiple correction paths in a structured manner while minimizing irrelevant interference to produce effective preference learning data.

[0084] like Figure 3 As shown in the schematic diagram of the process of iteratively optimizing the generator model, based on the corrected response and the failed response, constructing the generator training data to iteratively optimize the generator model includes the following steps:

[0085] S301: Pairing the corrected response with the failed response to generate training data pairs.

[0086] Collect the corrected responses and failed responses, which provide direct positive examples (corrected responses) and negative examples (failed responses) for training. Pair the corrected responses with the failed responses to generate training data pairs.

[0087] S302: Define a loss function, which directly reflects the degree to which the model learns the preferences.

[0088] S303: Using the generated training data pairs and the loss function, train the generator model through preference learning.

[0089] During training, the model learns to distinguish between good responses (corrected responses) and bad responses (responses that failed to execute).

[0090] S304: After each round of training, the validation set is used to evaluate the performance of the model, and the model parameters or training strategy are adjusted according to the evaluation results. The optimized generator model is used for the next round.

[0091] Based on the direct preference optimization method, the generator model is able to continuously learn and improve, enhancing its ability to handle complex instructions in practical applications.

[0092] like Figure 4 As shown in the schematic diagram of the process of iteratively optimizing the perfecter model, based on the data in the failed replies and the self-improvement strategy of the tree search, constructing the perfecter training data to iteratively optimize the perfecter model includes the following steps:

[0093] S401: Collect data from the tree search strategy, including replies to failed executions and all potential corrections generated during the search process; and distinguish all potential corrections into valid corrections and invalid corrections.

[0094] S402: Pair the failed replies with the effective corrections to form a complete training data pair.

[0095] S403: Generate correction candidates; for each generated correction candidate, calculate the probability of being accepted; and accept the correction candidate as training data only when it meets the set standard.

[0096] S404: Train the perfecter model using the received training data.

[0097] S405: After each round of training, the performance of the model is evaluated, and the model parameters or training strategy are adjusted as needed. The optimized perfecter model is used for the next round.

[0098] In this way, rejection sampling tuning helps the perfecter model learn to recover from errors effectively and generate more accurate and reliable correction responses.

[0099] In addition, the present application also provides a device for optimizing the instruction following capability of a large language model, such as Figure 5As shown in the structural block diagram of the device for optimizing the instruction compliance capability of a large language model provided in the present application, the device specifically includes a memory 51 and a processor 52, and the memory 51 stores a computer program. When the computer program is executed by the processor 52, the method for optimizing the instruction compliance capability of a large language model according to any one of the above-mentioned methods is implemented.

[0100] It can be understood that the device for optimizing the instruction compliance capability of a large language model provided in the present application corresponds to the method for optimizing the instruction compliance capability of a large language model mentioned above, and their specific implementation contents can be referenced to each other and will not be repeated here.

[0101] In addition, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned methods for optimizing the instruction compliance capability of a large language model.

[0102] Computer readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0103] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0104] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0105] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for optimizing the instruction following capability of a large language model, characterized in that: include: A self-game training framework is used to perform self-game by using a large language model to act as a generator model and a perfecter model. The following steps are performed in each round of self-game training: The generator model generates responses to the given instruction data; The Perfecter model judges and scores each response and finds responses that fail to execute; By using the self-improvement strategy of tree search, the improver model corrects the reply of the failed execution to obtain a corrected reply; Based on the corrected responses and the responses to the failed executions, generator training data is constructed to iteratively optimize the generator model; based on the responses to the failed executions and data in the self-improvement strategy of the tree search, perfecter training data is constructed to iteratively optimize the perfecter model; The self-improvement strategy using tree search, the improver model corrects the reply of the failed execution, and the corrected reply includes: The failed instruction data is used as the root node of the search, and the tree nodes are expanded from the root node, and each expanded child node corresponds to a possible correction result; The correctness of each correction result is judged and evaluated; the search process ends when the perfecter model corrects the correct response; The corrected reply will be used as the corrected reply.

2. The method for optimizing the instruction compliance capability of a large language model according to claim 1, characterized in that: The self-improvement strategy using tree search, the improver model corrects the reply of the failed execution, and the corrected reply includes: Using a self-improvement strategy of a depth-first search tree search, the improver model corrects the failed response to obtain a corrected response, and the execution method is as follows: Start from the root node and explore the unvisited child nodes downwards. After entering a new node, continue exploring downwards until the maximum exploration depth. When a node is reached and the maximum search depth is reached, backtrack to the previous node and continue to explore other child nodes; The search process ends until the perfecter model corrects the correct response.

3. The method for optimizing the instruction compliance capability of a large language model according to claim 1, characterized in that: The self-improvement strategy using tree search, the improver model corrects the reply of the failed execution, and the corrected reply includes: The self-improvement strategy of breadth-first search tree search is adopted. The improver model corrects the reply of the failed execution to obtain the corrected reply. The execution method is as follows: Starting from the root node, create a queue to store the nodes at each level; Put the root node into the queue, and then start the loop: dequeue the node, visit all the unvisited direct child nodes of the corresponding node, and put the child nodes into the queue; The search process ends when the perfecter model corrects the correct response.

4. The method for optimizing the instruction compliance capability of a large language model according to any one of claims 1 to 3, characterized in that: The self-improvement strategy using tree search, the improver model corrects the reply of the failed execution, and the corrected reply includes: During the tree search, the output of each node is checked and corrected by a self-improvement mechanism; the self-improvement mechanism analyzes the errors in the output and proposes a corrected response; For each revised response, its quality and accuracy are re-evaluated and the evaluation results are fed back into the tree search to revise the response with the new evaluation results.

5. The method for optimizing the instruction compliance capability of a large language model according to claim 4, characterized in that: The constructing generator training data based on the corrected response and the failed response to iteratively optimize the generator model includes: Pairing the corrected response with the failed response to generate training data pairs; Define a loss function that directly reflects how well the model has learned the preferences; Using the generated training data pairs and the loss function, training the generator model through preference learning; After each round of training, the validation set is used to evaluate the performance of the model, and the model parameters or training strategy are adjusted based on the evaluation results. The optimized generator model is used for the next round.

6. The method for optimizing the instruction compliance capability of a large language model according to claim 4, characterized in that: The step of constructing perfecter training data based on the data in the self-improvement strategy of the failed execution and the tree search to iteratively optimize the perfecter model includes: collecting data from the tree search strategy, including responses to failed executions and all potential fixes generated during the search; distinguishing all potential fixes into valid fixes and invalid fixes; Pair failed responses with effective corrections to form perfect training data pairs; Generate correction candidates; for each generated correction candidate, calculate the probability of being accepted; only when the correction candidate meets the set criteria, it is accepted as training data; Train the perfecter model using the received training data; After each round of training, the performance of the model is evaluated, and the model parameters or training strategy are adjusted as needed, and the optimized perfecter model is used for the next round.

7. The method for optimizing the instruction compliance capability of a large language model according to claim 1, characterized in that: The instruction-following capability includes objective constraints and subjective constraints; the subjective constraints include emotional factors, scene adaptability factors, and contextual relevance factors.

8. A device for optimizing the instruction following capability of a large language model, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method for optimizing the instruction compliance capability of a large language model according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for optimizing the instruction compliance capability of a large language model according to any one of claims 1 to 7 is implemented.

Images