Long text generation model optimization method and device based on adaptive constraint reward

Abstract

This application relates to the field of natural language processing technology, and provides a method and apparatus for optimizing a long text generation model based on adaptive constraint rewards. The method involves selecting sample data including long text generation tasks from human-computer interaction data, using a pre-trained agent to determine an adaptive constraint criterion list based on the sample data, generating N sample responses using a policy model based on long text generation instructions using a reinforcement learning algorithm, scoring the sample responses using a reward model based on the aforementioned adaptive constraint criterion list, determining the relative advantage of each sample response, and then updating the model parameters of the policy model based on this relative advantage. This achieves an optimized reinforcement learning algorithm for generating long text without requiring large-scale, high-quality manually labeled data, and the generated long text can be quantitatively evaluated. It can adapt to the specific instruction requirements of different task scenarios, improving the optimization efficiency of the reinforcement learning algorithm and enhancing the user experience.

Claims

1. An optimization method for a long text generation model based on adaptive constraint reward, characterized in that, include: Obtain sample data; the sample data includes human-computer interaction data, and the human-computer interaction data includes at least a long text generation task; Invoke the pre-trained agent to determine an adaptive constraint standard list based on the sample data; Obtain the long text generation instruction, input the long text generation instruction into the policy model of the reinforcement learning algorithm, and obtain N sampled responses; N is a positive integer greater than 1; The reward model of the reinforcement learning algorithm is used to score the N sampled responses respectively, and the relative advantage of each sampled response is determined based on the score of each sampled response; wherein, the reward model includes at least a quantization model, and the quantization conditions of the quantization model include at least the adaptive constraint criterion list; The model parameters of the strategy model are updated based on the relative advantage. The adaptive constraint criteria list determined based on the sample data includes: Obtain the target user instructions from the sample data, wherein the target user instructions are long text generation related instructions; The pre-trained agent is used to identify explicit requirements in the target user's instructions; the explicit requirements include at least key information, format requirements, and constraints in the target user's instructions. The pre-trained agent is used to identify implicit expectations in the target user instructions; the implicit expectations include at least the context, user preferences, and task type of each instruction; the implicit expectations are obtained by the pre-trained agent through contextual reasoning of the target user instructions and by comprehensively analyzing the context, user preferences, and task nature of the target user instructions. An initial list of adaptive constraint criteria is generated based on the explicit requirements and the implicit expectations; The pre-trained evaluation model is invoked to evaluate the initial adaptive constraint criteria list; In response to determining that the initial adaptive constraint standard list does not meet at least one of the preset conditions, the initial adaptive constraint standard list is updated until the updated adaptive constraint standard list meets all preset conditions. The updated adaptive constraint criteria list is determined to be the adaptive constraint criteria list; The preset conditions include: Each requirement in the list is a question that can be answered using a three-level rating system, which includes fully satisfied, partially satisfied, and not satisfied; and The list covers all key elements of the instruction, and the number of items in the list is less than a preset threshold; and The terminology in the list is clear and accurate; and The correlation between items in the list is less than the preset correlation threshold.

2. The method according to claim 1, characterized in that, The target sampled response is scored using a reward model, including: The adaptive constraint criteria list and the target sampled response are input into the quantization model of the reward model; The quantization model determines the verification result for each item in the adaptive constraint criteria list corresponding to the target sampled response; the verification result includes at least one of fully satisfied, partially satisfied, and not satisfied; Based on the verification results, the scores for each item corresponding to the target sampled response are determined; wherein, the verification results that fully satisfy the first score correspond to the first score, those that partially satisfy the second score correspond to the second score, and those that do not satisfy the third score correspond to the third score, and the first score is greater than the second score, and the second score is greater than the third score; The average score of each item in the target sampled response is determined as the score of the target sampled response; The target sampled response is any one of the N sampled responses.

3. The method according to claim 2, characterized in that, The verification results also include the verification basis; After determining the verification result, the method further includes: The credibility of the verification results shall be determined at least based on the verification criteria. In response to determining that the credibility of the verification result is less than a preset credibility threshold, the quantization model is updated, and the updated quantization model is used to determine the verification result of the target sample response again; In response to determining that the credibility of the verification result is greater than or equal to the preset credibility threshold, the score of each item corresponding to the target sampled response is determined based on the verification result.

4. The method according to claim 2, characterized in that, The reward model also includes a length reward function; Using a reward model to score the target sampled response also includes: The length reward function is invoked to generate the target length based on the long text generation instruction; In response to determining that the absolute value of the difference between the text length of the target sampled response and the target length is less than or equal to a preset deviation threshold, the length reward value of the target sampled response is determined to be a first value; the first value is the maximum value of the length reward. In response to determining that the absolute value of the difference between the text length of the target sampled response and the target length is greater than the preset deviation threshold, the length reward value of the target sampled response is determined to be a second value; the difference between the first value and the second value is positively correlated with the absolute value of the difference; The score of the target sampled response is determined based on the average score of each item corresponding to the target sampled response and the length reward value of the target sampled response.

5. The method according to any one of claims 1 to 4, characterized in that, The pre-trained agent is a pre-trained large language model, and the reinforcement learning algorithm is the Group Relative Policy Optimization (GRPO) algorithm. The step of updating the model parameters of the strategy model based on the relative advantage is to update the model parameters of the strategy model to increase the generation probability of sampled responses with positive relative advantage and decrease the generation probability of sampled responses with negative relative advantage.

6. An optimization device for a long text generation model based on adaptive constraint reward, characterized in that, include: The acquisition module is configured to acquire sample data; the sample data includes human-computer interaction data, and the human-computer interaction data includes at least a long text generation task; The determination module is configured to invoke a pre-trained agent to determine an adaptive constraint criteria list based on the sample data; The sampling module is configured to acquire long text generation instructions, input the long text generation instructions into the policy model of the reinforcement learning algorithm, and obtain N sampled responses; N is a positive integer greater than 1; The scoring module is configured to score the N sampled responses using the reward model of the reinforcement learning algorithm, and determine the relative advantage of each sampled response based on the score of each sampled response; wherein the reward model includes at least a quantization model, and the quantization conditions of the quantization model include at least the adaptive constraint criteria list; The optimization module is configured to update the model parameters of the strategy model based on the relative advantage; The adaptive constraint criteria list determined based on the sample data includes: Obtain the target user instructions from the sample data, wherein the target user instructions are long text generation related instructions; The pre-trained agent is used to identify explicit requirements in the target user's instructions; the explicit requirements include at least key information, format requirements, and constraints in the target user's instructions. The pre-trained agent is used to identify implicit expectations in the target user instructions; the implicit expectations include at least the context, user preferences, and task type of each instruction; the implicit expectations are obtained by the pre-trained agent through contextual reasoning of the target user instructions and by comprehensively analyzing the context, user preferences, and task nature of the target user instructions. An initial list of adaptive constraint criteria is generated based on the explicit requirements and the implicit expectations; The pre-trained evaluation model is invoked to evaluate the initial adaptive constraint criteria list; In response to determining that the initial adaptive constraint standard list does not meet at least one of the preset conditions, the initial adaptive constraint standard list is updated until the updated adaptive constraint standard list meets all preset conditions. The updated adaptive constraint criteria list is determined to be the adaptive constraint criteria list; The preset conditions include: Each requirement in the list is a question that can be answered using a three-level rating system, which includes fully satisfied, partially satisfied, and not satisfied; and The list covers all key elements of the instruction, and the number of items in the list is less than a preset threshold; and The terminology in the list is clear and accurate; and The correlation between items in the list is less than the preset correlation threshold.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images