Large language model iteration generation process optimization control method and system
By optimizing the iterative generation process of large language models using the POMDP model, the problem of unreliable iterative feedback in complex tasks is solved, and high-quality content generation is achieved.
Patent Information
- Application Number
- CN202511003574.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2025-10-31
AI Technical Summary
Large language models struggle to meet multidimensional objectives when handling complex tasks, and iterative feedback methods are unreliable, potentially leading to a decline in quality or getting stuck in local optima after iteration.
An iterative optimization control method based on partially observable Markov decision processes (POMDP) is adopted. Through multi-dimensional state management, iterative action decision-making, and multi-source feedback integration, the iterative path is optimized. Combined with belief state estimation and optimal action selection, the generation quality is ensured to meet user requirements.
Significantly reduce invalid iterations, improve generation quality, avoid blind modifications, ensure that generated content meets user needs, and improve feedback accuracy and generation efficiency.
Smart Images
Figure CN120874771A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and system for iterative generation optimization control of large language models in complex generation tasks based on partially observable Markov decision processes (POMDP). Background Technology
[0002] With the rapid development of Large Language Models (LLMs), they have demonstrated unprecedented performance in numerous applications, making them a crucial pillar of artificial intelligence. However, LLMs often fall short when handling complex tasks such as creative writing. Complex tasks typically have multi-dimensional objectives, requiring high levels of logical reasoning, originality, and other aspects. Evaluation criteria are often subjective and uncertain, making it difficult to guide models to generate high-quality results through explicit rules. Complex tasks often have long and complex contexts, causing models to struggle to understand the details of the task or the expected output. Some complex tasks may involve data domains that the model has not trained on or require logical reasoning, which are not LLMs' strengths, often resulting in unsatisfactory quality in single-shot outputs.
[0003] Iterative methods are gaining increasing attention, with more and more approaches leveraging the self-feedback and self-optimization of LLMs for multi-round interactions. This allows the model to progressively improve output quality during task execution, meeting the demands of complex tasks. Take creative writing tasks as an example. These tasks emphasize open-ended generation, typically requiring creators to possess strong language skills and creativity. An excellent article not only needs semantic coherence and rich content but also logical coherence to attract readers' interest. These requirements make generating a satisfactory article very difficult, especially when it needs to satisfy specific themes, styles, or reader preferences. In this context, iterative optimization is an effective improvement strategy. However, the iterative process itself presents many challenges. For example, ensuring that each iteration's modifications move in the right direction is crucial. Failure to properly control the iteration path may lead to a decline in output quality, wasting significant time and effort. Summary of the Invention
[0004] The one-time output of LLMs is insufficient to meet the multidimensional objectives of complex tasks, while existing iterative feedback methods (such as SELF-REFINE) are unreliable, potentially leading to quality degradation or getting trapped in local optima after iteration. This invention provides an iterative optimization control method and system for large language models based on partially observable Markov decision processes, suitable for high-quality content generation in complex tasks (such as creative writing and code generation).
[0005] In a first aspect, the present invention provides a method for optimizing and controlling the iterative generation process of a large language model, comprising the following steps:
[0006] Step 1. Based on the task requirements of the complex task, use the large language model to generate an initial version and initialize the belief state and worker capability parameters of the iterative control model;
[0007] Step 2. Based on the belief state of the iterative control model, simulate actions and calculate rewards, and select the optimal next action;
[0008] If an improvement action is selected, the current higher-quality version will be improved based on feedback, and the improved version will replace the lower-quality version.
[0009] If the evaluation action is selected, a quality assessment and feedback collection will be performed on the current version to update the belief state;
[0010] If you choose to submit, the iteration process ends.
[0011] Step 3. Repeat step 2 until you select the submission action or meet the preset conditions, and finally select the version with the highest quality as the final generated result.
[0012] A second aspect of the present invention provides a large language model iterative generation process optimization and control system, the system comprising:
[0013] The multidimensional state management module is used to build and maintain a multidimensional state space for character quality. It supports a custom multidimensional quality assessment system, maintains a multidimensional joint distribution for each version, and updates the quality state distribution based on feedback results after each iteration.
[0014] The iterative action decision module is used to determine the optimal action for the next step. Based on the reward function of different actions, it simulates and calculates the expected reward of each action. It adopts a greedy strategy K-step prediction algorithm to select the action sequence that maximizes the reward and executes the action until the termination condition of the iteration is reached.
[0015] The multi-source feedback integration module is used to collect and process feedback, integrate feedback information from different workers, optimize the iteration direction, and adaptively adjust model parameters based on the feedback obtained to improve decision-making accuracy.
[0016] The beneficial effects of this invention are as follows: This invention uses the POMDP model to dynamically plan the iterative path, combined with multi-dimensional feedback, to significantly reduce ineffective iterations. By estimating belief states and selecting optimal actions, it avoids blind modifications. Combined with worker capability modeling, it improves the accuracy of feedback and reduces erroneous decisions caused by unreliable feedback. At the same time, it ensures that the generated content meets user needs and further optimizes the generation quality. Attached Figure Description
[0017] Figure 1 This is a flowchart of the method of the present invention.
[0018] Figure 2 This is a system block diagram of the present invention.
[0019] Figure 3 A comparison chart showing the win rate of articles generated by the method of this invention compared with those generated by other methods on a creative writing dataset, using human evaluation.
[0020] Figure 4 This is a graph comparing the win rates of the method of this invention with other methods in generating articles on a conditionally constrained dataset. Detailed Implementation
[0021] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0022] This application provides an optimization and control method for the iterative generation process of a large language model, such as... Figure 1 As shown, the method includes the following steps:
[0023] (1) Based on the task requirements of the complex task, the initial version is generated using LLM to initialize the belief state and worker capability parameters of the iterative control model.
[0024] (2) Calculate rewards by simulating actions based on the belief state of the iterative control model, select the optimal next action, simulate K steps (K≥1) of greedy prediction, and calculate the expected cumulative reward for each action. Based on current historical feedback and text version Select an action sequence This maximizes the total expected return. If improvement is chosen, the current higher-quality version will be selected and improved based on feedback. The improved version will replace the lower-quality version. Evaluation will be conducted based on the two current versions to assess quality and collect feedback for belief state updates and improvements.
[0025] (3) Repeat (2) until you select to submit or the preset conditions are met. If you submit, the current model will select the version with the highest quality as the final content generation result.
[0026] Specifically, step (1) in this embodiment includes the following steps:
[0027] Based on the task requirements of the complex task, an initial version is generated using LLM to initialize the belief state and worker capability parameters of the iterative control model.
[0028] A state space is defined for quality control and observation, describing the quality status of two versions of a specific task: before and after improvement. The quality of each version can be measured through multiple dimensions, the specific dimensions of which need to be manually set according to different task requirements. For example, in a writing scenario, the design quality includes innovativeness. grammatical expression Content and plot Therefore, each version corresponds to a real but unobservable mass vector state, denoted as ,in Let represent the score of the i-th quality dimension in version t, and k be the set number of quality dimensions. The quality states of the two versions constitute the state space S, denoted as: ,in and These represent the quality ratings for version one and version two, respectively, as the actual quality of each version varies. It cannot be directly observed; the model needs to estimate it.
[0029] Furthermore, for each dimension, a distribution of belief states regarding the quality of the corresponding version is maintained. This is used to represent the probability of the quality state in each dimension under the current information. The overall quality belief state for each version is represented by a multidimensional joint distribution. This indicates that the belief state is dynamically updated based on past actions (e.g., behavior modification) and evaluation feedback. The distribution of belief states for each dimension of quality in the first version will use the average performance of performing this type of task through LLM as prior knowledge to define the prior distribution. In the creative writing task, the average quality of the articles directly generated by performing the writing task through LLM combined with a certain standard deviation will be used as the prior distribution.
[0030] Furthermore, worker capability parameters The initial setting is based on prior data and is set to a medium level. After each iteration, the setting is adjusted based on worker evaluation feedback. If the worker's feedback is correct, then If the worker reports an error, then Where α is the learning rate and d is the task difficulty, the ability parameters are adaptively adjusted according to the worker's performance. The ability of workers who make correct assessments continuously increases, and the reliability of assessment feedback is improved; while the ability of workers who make frequent mistakes decreases, and the reliability of feedback decreases.
[0031] Specifically, step (2) in this embodiment includes the following steps:
[0032] The reward is calculated based on the belief state of the iterative control model, and the next action is selected. If improvement is selected, the current higher-quality version will be selected and improved in combination with feedback. The improved version will replace the lower-quality version. The evaluation will be carried out based on the quality assessment of the two current versions and feedback will be collected for belief state updates and improvements.
[0033] Based on the requirements of the iterative task, the actions are defined to mainly include three actions:
[0034] Improvement: Select version A with the highest quality score from the two current versions, generate a new version B by combining worker feedback, retain version A and the new version B together to form the next round of candidate versions, and maintain their belief state distribution respectively.
[0035] Assessment: Conduct worker assessments or combine automated tools to score the quality of the two current versions, and obtain quality feedback for each version for status estimation and improvement of subsequent versions.
[0036] Submit: Based on the current quality estimate, select the version with the higher quality estimate to submit and end the iteration process.
[0037] First, note that under the current problem model, the potential state of POMDP only changes when the agent requests an improvement action. When the agent requests an evaluation action, the quality of each version does not change; the evaluation only affects the agent's belief state.
[0038] Furthermore, suppose there are currently two versions. Their masses are unknown and are denoted as . The current state is .
[0039] (2-1) For the improvement action, if the agent currently selects based on Improvements were made, and LLM incorporated worker feedback to generate a new version. Its mass is denoted as , Depends on the quality of the previous version It also depends on workers The ability to estimate workers' capabilities through observation models. The greater the worker's ability, the more accurate the feedback provided during the evaluation phase, and the greater the potential for improvement. Therefore, to determine the next stage, it is necessary to calculate the worker's feedback on quality. The version provides effective feedback generation quality of [missing information]. Conditional distribution Current state will be based on probability Transition to a new state .thus The distribution of belief states in each dimension will be updated as follows: Then calculate the multidimensional joint distribution. Since the agent does not know in advance which specific workers will submit feedback, the agent will assume that the workers’ abilities are at an average level when conducting predictive analysis for subsequent strategy formulation.
[0040] (2-2) For the evaluation action, observations are only obtained during the iteration process by performing the evaluation action. Evaluation and feedback are crucial and can be obtained through automated tools (such as pre-trained scoring models and evaluation metrics for the corresponding task) or humans. Humans have always been an effective source of feedback. Different observations will be defined for different tasks to obtain feedback that is conducive to completing the task requirements. However, a vote evaluation on which of the two versions is better will be obtained based on each dimension of the task. This evaluation will be used to estimate the parameters of the observation model and the subsequent quality state. For the writing task, observations about the current version state will be obtained through the evaluation action. An article evaluation questionnaire will be issued to allow workers to evaluate each article from multiple dimensions (such as emotional expression, structural integrity, etc.) and provide optimization suggestions to help the model improve the quality of the articles in the next iteration.
[0041] Specifically, voting questions will be set for each dimension. For example, in the dimension of innovation, the version... Is its innovation greater than the version? Even better, workers will vote to choose yes or no. The vote result will be expressed as... The multi-dimensional voting results are represented as At the same time, workers are required to provide improvement suggestions for this dimension as prompts for LLM when making improvement actions. These will all be used as observations for surrogate calculation of the posterior belief state distribution.
[0042] Based on the observations obtained in the current state, the posterior probabilities of quality calculated based on voting are as follows: and Posterior probability has two important functions. First, more accurate beliefs increase the probability of retaining a better version in later stages. Second, more accurate beliefs also help determine when to stop the iteration process and submit the final result. To achieve this, it is necessary to model the accuracy of the workers. The accuracy of each worker's feedback will vary with the problem difficulty, thus defining the inherent difficulty of the problem. The inherent difficulty depends on how close the quality of the two versions is; the closer they are, the more difficult it becomes to determine which is better. , Let k be the difficulty constant and k be the task dimension, which defines the worker's accuracy. As the problem becomes more difficult, the accuracy will approach random behavior, that is, when When the accuracy is close to 0.5, while when At that time, the accuracy is close to 1. Therefore, if... ,but Next, the new voting results are calculated based on the existing posterior state distribution of quality. The posterior state distribution , ,in The other version of the posterior distribution undergoes a similar distribution update.
[0043] (2-3) The reward function is used to guide the model to make quality optimization decisions and calculate rewards based on the current version of the belief state. The higher the value, the higher the reward, as defined. For quality The reward that can be obtained at any time, and the next action decision is made based on the reward value of the current state.
[0044] The reward for submitting the action is defined as follows: it represents selecting the version currently considered to be of the highest quality.
[0045]
[0046] in The reward that can be obtained under the current belief state.
[0047] The reward that can be obtained by taking an evaluation action is calculated based on the probability of the feedback result, and the reward is calculated based on the new posterior distribution.
[0048]
[0049] in ,
[0050] The result obtained by taking evaluation actions is The probability of P is calculated, and then all possibilities are calculated to estimate the possible reward. E This indicates the cost of taking an evaluation action.
[0051] The reward for taking an improvement action is calculated based on the reward for submitting the action in the above formula. This allows us to determine which version currently has the highest quality. The simulation is based on the higher-quality version for improvement. The quality of the newly generated version is used to obtain the prior distribution of the belief state through the state transition described above. The posterior state distribution of the improved version is then transformed into a new round of prior state distribution, and then the reward is calculated.
[0052]
[0053] Where P I This indicates the cost of implementing the improvement measures.
[0054] (2-4) A K-step prediction greedy algorithm is used for action selection. First, actions are performed based on the belief state in the current state space. For the three optional actions of improvement, evaluation, and submission, corresponding virtual posterior belief states are constructed (simulating the state distribution after action execution based on the current belief state). Based on each virtual posterior belief state, the model's state transition rules and reward function are used to recursively simulate all possible action sequences to calculate the cumulative expected reward for the next K steps. Finally, the cumulative expected rewards of each action are sorted, and the action with the largest reward is selected as the optimal action. Specifically, step (3) in this embodiment includes the following steps:
[0055] Repeat (2) until submission is selected or the preset conditions are met. If submission is made, the current model selects the version with the highest quality as the final content generation result. The preset conditions are: reaching the preset maximum number of iterations, or the cumulative cost (such as manual evaluation expenses, computational resource consumption, etc.) reaching the preset threshold, or the change in the multidimensional quality distribution of the belief state is small and lower than the preset threshold in N (N>=2) consecutive iterations (i.e., the belief state tends to be stable). The iteration terminates when the preset conditions are met or submission is selected.
[0056] This application provides a large language model iterative optimization control system based on POMDP, such as... Figure 2 As shown, the system includes:
[0057] The multi-dimensional state management module is used to construct and maintain a multi-dimensional state space for character quality. It supports custom multi-dimensional quality assessment systems (such as innovation, logic, and emotional expression in creative writing), maintains a multi-dimensional joint distribution for each version, and updates the quality state distribution based on feedback results after each iteration.
[0058] The iterative action decision module is used to determine the optimal action for the next step. Based on the reward function of different actions, it simulates and calculates the expected reward of each action. It adopts a greedy strategy K-step prediction algorithm to select the action sequence that maximizes the reward and executes the action until the termination condition of the iteration is reached.
[0059] The multi-source feedback integration module is used to collect and process feedback, integrate feedback information from different workers, optimize the iteration direction, and adaptively adjust model parameters based on the feedback obtained to improve decision-making accuracy.
[0060] Verification example:
[0061] First, this embodiment was compared with other large model iterative generation methods, and the results are as follows: Figure 3 , Figure 4As shown. Experiments were conducted using the GPT-4o model on creative writing datasets and conditional generation datasets, with evaluation metrics covering dimensions such as creativity, rationality, and overall performance. Specifically, a double-blind human evaluation method was used to compare the articles generated by each method pairwise, record the winners, and calculate the average win rate. The method described in this application exhibits a higher average win rate and better overall performance. The method described in this application constructs a POMDP model to dynamically maintain the belief state of candidate versions and, based on multiple rounds of human feedback, achieves belief-driven optimal iterative path planning. This effectively solves the problem of quality fluctuations caused by blind rewriting in traditional methods. Driven by human feedback, and combined with belief updating and strategy planning mechanisms, it improves the actual performance of large language models in complex generation tasks.
[0062] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for optimizing and controlling the iterative generation process of a large language model, characterized in that, Includes the following steps: Step 1. Based on the task requirements of the complex task, use the large language model to generate an initial version and initialize the belief state and worker capability parameters of the iterative control model; Step 2. Based on the belief state of the iterative control model, simulate actions and calculate rewards, and select the optimal next action; If an improvement action is selected, the current higher-quality version will be improved based on feedback, and the improved version will replace the lower-quality version. If the evaluation action is selected, a quality assessment and feedback collection will be performed on the current version to update the belief state; If you choose to submit, the iteration process ends. Step 3. Repeat step 2 until you select the submission action or meet the preset conditions, and finally select the version with the highest quality as the final generated result.
2. The method for optimizing and controlling the iterative generation process of a large language model according to claim 1, characterized in that, The belief states include the belief distribution for each generated version of the multidimensional quality assessment system, which supports customization and covers innovativeness, logic and emotional expression.
3. The method for optimizing and controlling the iterative generation process of a large language model according to claim 1 or 2, characterized in that, The worker competency parameters are initialized based on prior data and dynamically adjusted according to worker evaluation feedback to reflect the accuracy and reliability of worker feedback.
4. The method for optimizing and controlling the iterative generation process of a large language model according to claim 1, characterized in that, The improvements include: Choose the version with the highest current quality as the base version; New, improved versions were generated based on worker feedback; Replace the lower-quality version with the improved version and update the belief state.
5. The method for optimizing and controlling the iterative generation process of a large language model according to claim 1, characterized in that, The evaluation actions include: Conduct a quality assessment of the two current versions and obtain quality feedback for each version; Update the belief state distribution based on the assessment results; Collect worker feedback to optimize the iteration direction.
6. The method for optimizing and controlling the iterative generation process of a large language model according to claim 1, characterized in that, The submission action includes: Select the highest quality version based on your current belief state; End the iteration process and use this version as the final result.
7. The method for optimizing and controlling the iterative generation process of a large language model according to any one of claims 4 to 6, characterized in that, The preset conditions include at least one of the following: The preset maximum number of iterations has been reached; The accumulated cost has reached a preset threshold; The change in the multidimensional quality distribution of belief states during multiple consecutive iterations is lower than a preset threshold.
8. The method for optimizing and controlling the iterative generation process of a large language model according to claim 7, characterized in that, The selection of the optimal next action adopts a K-step predictive greedy algorithm, which calculates the expected cumulative reward of each action through simulation and selects the action sequence that maximizes the reward.
9. The method for optimizing and controlling the iterative generation process of a large language model according to claim 2, characterized in that, The multidimensional quality assessment system collects feedback through manual evaluation and / or automated tools to optimize the iteration direction and belief state updates.
10. A large language model iterative generation process optimization control system, characterized in that, The system includes: The multidimensional state management module is used to build and maintain a multidimensional state space for character quality. It supports a custom multidimensional quality assessment system, maintains a multidimensional joint distribution for each version, and updates the quality state distribution based on feedback results after each iteration. The iterative action decision module is used to determine the optimal action for the next step. Based on the reward function of different actions, it simulates and calculates the expected reward of each action. It adopts a greedy strategy K-step prediction algorithm to select the action sequence that maximizes the reward and executes the action until the termination condition of the iteration is reached. The multi-source feedback integration module is used to collect and process feedback, integrate feedback information from different workers, optimize the iteration direction, and adaptively adjust model parameters based on the feedback obtained to improve decision-making accuracy.