Long text generation model optimization method and device based on adaptive constraint reward

Abstract

The invention relates to the technical field of natural language processing, and provides a long text generation model optimization method and device based on adaptive constraint rewards. The method comprises the following steps: screening sample data comprising a long text generation task from human-computer interaction data, and determining an adaptive constraint standard list based on the sample data by utilizing a pre-training agent; generating N sampling replies based on the long text generation instruction by using a strategy model of a reinforcement learning algorithm, scoring the sampling replies based on the self-adaptive constraint standard list by using a reward model, determining relative advantages of the sampling replies, and updating model parameters of the strategy model according to the relative advantages, therefore, the optimized reinforcement learning algorithm for generating the long text can be obtained without large-scale high-quality manual data annotation, the generated long text can be quantitatively evaluated, specific instruction requirements in different task scenes can be met, the optimization efficiency of the reinforcement learning algorithm is improved, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and in particular to a long text generation model optimization method and device based on adaptive constraint rewards. BACKGROUND

[0002] Recent research progress in large language models has significantly improved their long context understanding capabilities. This breakthrough naturally pushes the research frontier to more challenging areas: high-quality long text generation (such as story creation, report writing, or legal document drafting). However, for large language models with long context processing capabilities, generating thousands of words of text with coherence, structural rigor, and content appeal remains an important academic challenge that has not yet been solved.

[0003] Although many researchers focus on SFT (Supervised Fine-Tuning) methods, such methods are generally limited by two constraints: they rely on high-cost synthetic datasets generated by specialized large language models, and they lack systematic optimization of response quality attributes. In contrast, RL (Reinforcement Learning) directly optimizes target characteristics through a reward mechanism, providing a more flexible and robust training paradigm for solving this problem, especially when training long text generation models for specific human preferences. However, existing reinforcement learning-based methods mainly rely on pairwise preference reward mechanisms with coarse-grained evaluation dimensions. While these methods can ensure basic quality indicators such as relevance, coherence, and practicality, they struggle to meet the specific needs of diverse tasks with fine-grained, instruction-adaptive requirements.

[0004] Therefore, how to design a training method that does not rely on pairwise preference data and can perform fine-grained, verifiable evaluation of generation quality is a technical problem that needs to be solved in the field of long text generation for large language models. SUMMARY

[0005] Therefore, the embodiments of the present application provide a long text generation model optimization method and device based on adaptive constraint rewards to solve the problem that long text generation in the prior art relies on pairwise preference data and the generated long text is difficult to evaluate in fine-grained and verifiable manner.

[0006] The first aspect of the embodiments of the present application provides a long text generation model optimization method based on adaptive constraint rewards, comprising:

[0007] Obtaining sample data; the sample data includes human-computer interaction data, and the human-computer interaction data at least includes a long text generation task;

[0008] Calling a pre-trained agent to determine an adaptive constraint standard list based on the sample data;

[0009] obtain the long text generation instruction, input the long text generation instruction into a policy model of a reinforcement learning algorithm, and obtain N sampled replies; N is a positive integer greater than 1;

[0010] score the N sampled replies respectively using a reward model of the reinforcement learning algorithm, and determine a relative advantage of each sampled reply based on the scores of the sampled replies; wherein the reward model at least includes a quantification model, and a quantification condition of the quantification model at least includes the adaptive constraint standard list;

[0011] update model parameters of the policy model based on the relative advantage.

[0012] In a second aspect, the embodiment of the present application provides a long text generation model optimization device based on adaptive constraint rewards, which comprises:

[0013] an obtaining module configured to obtain sample data; the sample data comprises human-computer interaction data, and the human-computer interaction data at least includes a long text generation task;

[0014] a determining module configured to call a pre-trained agent, and determine an adaptive constraint standard list based on the sample data;

[0015] a sampling module configured to obtain a long text generation instruction, input the long text generation instruction into a policy model of a reinforcement learning algorithm, and obtain N sampled replies; N is a positive integer greater than 1;

[0016] a scoring module configured to score the N sampled replies respectively using a reward model of the reinforcement learning algorithm, and determine a relative advantage of each sampled reply based on the scores of the sampled replies; wherein the reward model at least includes a quantification model, and a quantification condition of the quantification model at least includes the adaptive constraint standard list;

[0017] an optimization module configured to update model parameters of the policy model based on the relative advantage.

[0018] In a third aspect, the embodiment of the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0019] In a fourth aspect, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0020] The beneficial effects of the embodiments of the present application compared with the prior art are: the embodiments of the present application filter out sample data including long text generation tasks from massive human-computer interaction data, determine an adaptive constraint standard list based on the sample data by using a pre-trained agent; generate N sample replies based on a long text generation instruction by using a policy model of a reinforcement learning algorithm, score the N sample replies based on the adaptive constraint standard list by using a reward model, determine the relative advantages of each sample reply, and then update the model parameters of the policy model according to the relative advantages, complete the optimization of the reinforcement learning algorithm, so as to realize the optimized reinforcement learning algorithm for generating long text without large-scale high-quality artificial annotation data, and the generated long text can be quantitatively evaluated, can adapt to specific instruction requirements in different task scenarios, improves the reinforcement learning algorithm optimization efficiency, and improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 is a flowchart of a long text generation model optimization method based on adaptive constraint rewards provided by the embodiments of the present application.

[0023] Figure 2 is a flowchart of a method for determining an adaptive constraint standard list based on sample data provided by the embodiments of the present application.

[0024] Figure 3 is a flowchart of a method for determining an adaptive constraint standard list based on explicit requirements and implicit expectations provided by the embodiments of the present application.

[0025] Figure 4 is a flowchart of a method for scoring a target sample reply by using a reward model provided by the embodiments of the present application.

[0026] Figure 5 is a flowchart of another method for scoring a target sample reply by using a reward model provided by the embodiments of the present application.

[0027] Figure 6 is a flowchart of still another method for scoring a target sample reply by using a reward model provided by the embodiments of the present application.

[0028] Figure 7is a system structure diagram for implementing a long text generation model optimization method based on adaptive constraint rewards provided by embodiments of the present application.

[0029] Figure 8 is a schematic diagram of a long text generation model optimization device based on adaptive constraint rewards provided by embodiments of the present application.

[0030] Figure 9 is a schematic diagram of an electronic device provided by embodiments of the present application. DETAILED DESCRIPTION

[0031] In the following description, specific details are set forth such as particular system structures, techniques, etc. in order to provide a thorough understanding of embodiments of the present application. However, persons skilled in the art will understand that other embodiments of the present application can be practiced without these specific details. In other instances, well-known structures, devices, circuits, and methods have not been described in detail in order to avoid obscuring the description of the present application.

[0032] A long text generation model optimization method and device based on adaptive constraint rewards according to embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0033] As mentioned above, the SFT method is generally limited by double constraints: it relies on high-cost synthetic data sets generated by proprietary large language models, and it lacks systematic optimization of response quality attributes. The RL method can ensure basic quality indicators such as relevance, coherence, and practicality, but it is difficult to meet the specific needs of fine-grained and instruction-adaptive tasks in diversified tasks.

[0034] Taking a story creation instruction as an example, if the long text generation instruction is "write a 3000-word novel, describe a pair of young couples struggling for survival, and use an O. Henry-style ending", the core is to require an unexpected and satirical turning style - this quality is not a universal standard for "high-quality" writing, but a highly specific stylistic requirement. More importantly, relying on the collection mechanism of additional preference response pairs will bring significant data burden, resulting in high cost and difficulty in scaling the training process.

[0035] Therefore, how to design a training method that does not rely on paired preference data and can perform fine-grained and verifiable evaluation of generation quality is a technical problem that needs to be solved in the field of long text generation of large language models.

[0036] In view of this, the embodiment of the present application provides a long text generation model optimization method based on adaptive constraint reward. The method filters sample data including a long text generation task from a large amount of human-computer interaction data, and determines an adaptive constraint standard list based on the sample data by using a pre-trained agent. A strategy model of a reinforcement learning algorithm is used to generate N sample replies based on a long text generation instruction. A reward model is used to score the N sample replies based on the adaptive constraint standard list, and the relative advantages of each sample reply are determined. Then, the model parameters of the strategy model are updated according to the relative advantages, and the optimization of the reinforcement learning algorithm is completed, so that the optimized reinforcement learning algorithm for generating long text can be obtained without large-scale high-quality artificial annotation data, and the generated long text can be quantitatively evaluated, which can adapt to the specific instruction requirements in different task scenarios, improves the reinforcement learning algorithm optimization efficiency, and improves the user experience.

[0037] Figure 1 is a flowchart of a long text generation model optimization method based on adaptive constraint reward provided by the embodiment of the present application. As shown in Figure 1 the long text generation model optimization method based on adaptive constraint reward includes the following steps:

[0038] In step S101, sample data is obtained.

[0039] The sample data includes human-computer interaction data, and the human-computer interaction data at least includes a long text generation task.

[0040] In step S102, a pre-trained agent is called to determine an adaptive constraint standard list based on the sample data.

[0041] In step S103, a long text generation instruction is obtained, and the long text generation instruction is input into a strategy model of a reinforcement learning algorithm to obtain N sample replies.

[0042] Wherein, N is a positive integer greater than 1.

[0043] In step S104, a reward model of a reinforcement learning algorithm is used to score the N sample replies respectively, and the relative advantages of each sample reply are determined based on the scores of the sample replies.

[0044] The reward model at least includes a quantification model, and the quantification condition of the quantification model at least includes the adaptive constraint standard list.

[0045] In step S105, the model parameters of the strategy model are updated based on the relative advantages.

[0046] In some embodiments of the present application, the method can be executed by a server or a terminal device with certain processing capability.

[0047] In some embodiments of the present application, sample data can be acquired first, which can include human-computer interaction data, and the human-computer interaction data at least includes long text generation tasks.

[0048] That is, long text generation tasks can be automatically mined from a large amount of user-model interaction data, and the model can be a long text automatic generation model constructed based on a reinforcement learning algorithm. In an example, a real-world human-computer interaction data set can be quality filtered, and instructions related to long text generation can be screened from it.

[0049] In some embodiments of the present application, a pre-trained agent can be called to determine an adaptive constraint standard list based on sample data.

[0050] In an example, a large language model can be used as an instruction analyzer, a Prompt template can be designed to identify the requirements of each user instruction, and then the identified requirements can be analyzed to decompose the long text generation task related instructions in the sample data into a series of clear, unique and verifiable constraint standard list.

[0051] In some embodiments of the present application, a long text generation instruction can also be acquired, which is a user input instruction. The long text generation instruction can be input into a policy model of a reinforcement learning algorithm to obtain N sampled replies.

[0052] After obtaining N sampled replies, a reward model in the reinforcement learning algorithm can be used to score each of the N sampled replies, and the relative advantages of each sampled reply can be determined based on the scores of each sampled reply. The reward model at least includes a quantification model, which determines the score of each sampled reply based on the matching degree of each sampled reply and the quantification condition. The quantification condition of the quantification model at least includes the adaptive constraint standard list described above.

[0053] In some embodiments of the present application, the model parameters of the policy model in the reinforcement learning algorithm can be updated based on the determined relative advantages to complete the optimization of the reinforcement learning algorithm.

[0054] According to the technical scheme provided in the embodiments of the present application, the sample data including the long text generation task is filtered out from the massive human-computer interaction data, and the pre-trained agent is used to determine the adaptive constraint standard list based on the sample data; the strategy model of the reinforcement learning algorithm is used to generate N sample replies based on the long text generation instruction, the reward model is used to score the N sample replies based on the adaptive constraint standard list, and the relative advantages of the sample replies are determined, and then the model parameters of the strategy model are updated according to the relative advantages, the optimization of the reinforcement learning algorithm is completed, so that the optimized reinforcement learning algorithm for generating long text is obtained without large-scale high-quality artificial annotation data, and the generated long text can be quantitatively evaluated, which can adapt to the specific instruction requirements in different task scenarios, improves the optimization efficiency of the reinforcement learning algorithm, and improves the user experience.

[0055] Figure 2 is a flowchart of a method for determining an adaptive constraint standard list based on sample data provided by the embodiments of the present application. As shown in Figure 2 , the method comprises the following steps:

[0056] In step S201, the target user instruction in the sample data is obtained.

[0057] Among them, the target user instruction is a long text generation related instruction.

[0058] In step S202, the pre-trained agent is used to identify the explicit requirements in the target user instruction.

[0059] Among them, the explicit requirements at least include the key information, format requirements and restriction conditions in the target user instruction.

[0060] In step S203, the pre-trained agent is used to identify the implicit expectations in the target user instruction.

[0061] Among them, the implicit expectations at least include the context, user preference and task type of each instruction.

[0062] In step S204, the adaptive constraint standard list is determined based on the explicit requirements and the implicit expectations.

[0063] In some embodiments of the present application, when determining the adaptive constraint standard list based on the sample data, the target user instruction in the sample data can be obtained first, and the target user instruction is a long text generation related instruction. The long text generation related instruction may, for example, include a long text generation instruction, an instruction for adjusting the generated long text, an instruction for evaluating the generated long text, and other instructions related to long text generation.

[0064] In some embodiments, a pre-trained agent can be used to identify explicit requirements in the target user instructions, where the explicit requirements at least include key information, format requirements and restrictions in the target user instructions.

[0065] In other embodiments, a pre-trained agent can also be used to identify implicit expectations in the target user instructions, where the implicit expectations at least include the context of each instruction, user preferences and task types.

[0066] After determining the explicit requirements and implicit expectations, an adaptive constraint standard list can be determined based on the explicit requirements and implicit expectations.

[0067] That is, a large language model can be used as an instruction analyzer to identify requirements for each user instruction to extract explicit requirements and identify implicit expectations. For the explicit requirements of the user, a large language model can be used to automatically identify and extract all key information, format requirements and restrictions in the instructions, etc., to ensure that each explicit user requirement or intention is captured completely and no information is missed. For the implicit expectations of the user, a large language model can be used for in-depth context reasoning to comprehensively analyze the context of the entire instruction, the user's preferences and the nature of the task, and to infer the user's "unspoken intentions" and infer these implicit expectations.

[0068] Further, based on the above analysis of the explicit requirements and implicit expectations of the user, the original instruction can be deconstructed into a series of clear, unique and verifiable constraint standard lists.

[0069] Figure 3 is a flowchart of a method for determining an adaptive constraint standard list based on explicit requirements and implicit expectations provided by an embodiment of the present application. As shown in Figure 3 The method comprises the following steps:

[0070] In step S301, an initial adaptive constraint standard list is generated based on explicit requirements and implicit expectations.

[0071] In step S302, a pre-trained evaluation model is called to evaluate the initial adaptive constraint standard list.

[0072] In step S303, in response to determining that the initial adaptive constraint standard list does not meet at least one of the preset conditions, the initial adaptive constraint standard list is updated until the updated adaptive constraint standard list meets all the preset conditions.

[0073] In step S304, the updated adaptive constraint standard list is determined as the adaptive constraint standard list.

[0074] In some embodiments of the present application, when determining the adaptive constraint standard list based on explicit requirements and implicit expectations, an initial adaptive constraint standard list can be first generated by the pre-trained agent based on the explicit requirements and the implicit expectations, and then the pre-trained evaluation model can be called to evaluate the initial adaptive constraint standard list. The pre-trained evaluation model can be built-in the pre-trained agent or can be an independent evaluation model, which is not limited here.

[0075] If the evaluation result of the pre-trained evaluation model indicates that the initial adaptive constraint standard list does not meet at least one of the preset conditions, the initial adaptive constraint standard list is updated, and then the pre-trained evaluation model is used to evaluate the updated initial adaptive constraint standard list until the updated adaptive constraint standard list meets all the preset conditions.

[0076] The preset conditions include that each requirement in the list is a question that can be answered with a three-level score, the three-level score includes complete satisfaction, partial satisfaction and non-satisfaction; the list covers all key elements of the instruction, and the number of items in the list is less than a preset number threshold; the terms in the list are clear and accurate; and the correlation between the items in the list is less than a preset correlation threshold. The all key elements of the instruction can include all constraint information related to long text generation in the target user instruction. The specific values of the preset number threshold and the preset correlation threshold are set according to actual needs, which are not limited here.

[0077] That is, the adaptive constraint standard list constructed should meet the following requirements: 1) each requirement in the list is a question that can be answered with a three-level score "completely satisfied / partially satisfied / not satisfied"; 2) the list ensures to cover all key aspects of the instruction, and avoids redundant or overly detailed questions; 3) each requirement in the list is described in objective and accurate language, avoiding the use of ambiguous terms; 4) each requirement in the list should be as independent as possible, and each requirement preferably corresponds to a specific element in the evaluation answer.

[0078] Further, the updated adaptive constraint standard list can be determined as the adaptive constraint standard list.

[0079] Figure 4 is a flowchart of a method for scoring a target sampling reply using a reward model provided by an embodiment of the present application. As shown in Figure 4 , the method includes the following steps:

[0080] In step S401, the adaptive constraint standard list and the target sampling reply are input into the quantization model of the reward model.

[0081] In step S402, the quantization model determines the verification result of the target sampling reply for each item in the adaptive constraint standard list.

[0082] The verification result includes at least one of complete satisfaction, partial satisfaction, and non-satisfaction.

[0083] In step S403, the score of each item corresponding to the target sampling reply is determined based on the verification result.

[0084] The verification result corresponds to a first score for complete satisfaction, a second score for partial satisfaction, and a third score for non-satisfaction, and the first score is greater than the second score, and the second score is greater than the third score.

[0085] In step S404, the average of the scores of each item corresponding to the target sampling reply is determined as the score of the target sampling reply.

[0086] The target sampling reply is any one of the N sampling replies.

[0087] In some embodiments of the present application, when the reward model is used to score the target sampling reply, the adaptive constraint standard list and the target sampling reply can be first input into the quantization model of the reward model, and the verification result of the target sampling reply corresponding to each item in the adaptive constraint standard list is determined by the quantization model. The verification result includes at least one of complete satisfaction, partial satisfaction, and non-satisfaction.

[0088] The score of each item corresponding to the target sampling reply can be determined based on the verification result. For example, if the verification result of the target sampling reply is complete satisfaction, the score of the target sampling reply can be determined as the first score; if the verification result of the target sampling reply is partial satisfaction, the score of the target sampling reply can be determined as the second score; and if the verification result of the target sampling reply is non-satisfaction, the score of the target sampling reply can be determined as the third score. The first score is greater than the second score, and the second score is greater than the third score.

[0089] In an example, the first score can be set to 1.0, the second score can be set to 0.5, and the third score can be set to 0. Alternatively, the first score, the second score, and the third score can be set to other numerical values according to actual needs, which are not limited herein.

[0090] Further, the average of the scores of each item corresponding to the target sampling reply can be determined as the score of the target sampling reply.

[0091] That is, the embodiments of the present application propose a novel reward model. When the reward model is used to evaluate each sampling reply, on the one hand, the quality of each sampling reply can be evaluated by quantifying the degree of satisfaction of the adaptive constraint standard list. This converts the traditional coarse-grained and subjective quality evaluation into a series of fine-grained and objective constraint verification tasks.

[0092] In some embodiments, the complex long text reply evaluation problem is redefined and simplified into a verification task based on a constraint list, which makes a small-scale policy model sufficient to perform the function of a reward model without introducing a larger reward model. Therefore, a Prompt template can be designed to guide the reward model to verify the satisfaction of the given reply to the constraint list. The specific verification method is as follows: given the original user instruction, the constraint criteria, and the reply content, the reward model needs to analyze the reply content in detail against the constraint criteria, for example, citing the relevant original text in the reply as evidence. Finally, the satisfaction of the reply to the constraint is given: “fully satisfied”, “partially satisfied”, or “not satisfied”.

[0093] For each sample reply generated by the policy model, the above verification method is used to verify whether it satisfies each item in the constraint list. The verification result is divided into three grades: “fully satisfied”, “partially satisfied”, and “not satisfied”, which correspond to a numerical score (such as 1.0, 0.5, 0) respectively. The final constraint reward is the average of all item scores.

[0094] In some embodiments of the present application, the verification result can also include a verification basis. In an example, the long text generation instruction, the adaptive constraint criteria list, and the target sample reply are input into the quantization model of the reward model, and the quantization model can verify the target sample reply based on the content in the long text generation instruction to obtain the verification result. That is, the verification basis can be the content in the long text generation instruction, or inferred from the content in the long text instruction. In addition, the verification basis can also be common sense.

[0095] Figure 5 is a flowchart of another method for scoring a target sample reply using a reward model provided by an embodiment of the present application. Wherein, Figure 5 The steps S501 to S502 in the embodiment shown are basically the same as the steps S401 to S402 in the embodiment shown, Figure 4 The steps S501 to S502 in the embodiment shown are basically the same as the steps S401 to S402 in the embodiment shown, Figure 5 The step S506 in the embodiment shown is basically the same as the step S404 in the embodiment shown, and will not be described here again. As Figure 4 The step S506 in the embodiment shown is basically the same as the step S404 in the embodiment shown, and will not be described here again. As Figure 5 The method further includes the following steps:

[0096] In step S503, the credibility of the verification result is determined based on at least the verification basis.

[0097] In step S504, in response to determining that the credibility of the verification result is less than a preset credibility threshold, the quantization model is updated, and the verification result of the target sample reply is determined again using the updated quantization model.

[0098] In step S505, in response to determining that the credibility of the verification result is greater than or equal to the preset credibility threshold, scores of each item corresponding to the target sampling reply are determined based on the verification result.

[0099] In some embodiments of the present application, after determining the verification result, the credibility of the verification result can also be determined based on at least the verification basis. In an example, an additional intelligent agent can be called to determine the credibility of the verification result. In another example, manual judgment can also be performed. In yet another example, the credibility of the verification result can also be determined by combining the judgment of the additional intelligent agent and the manual judgment.

[0100] If the credibility of the determined verification result is less than the preset credibility threshold, the quantization model is updated, and the verification result of the target sampling reply is determined again using the updated quantization model. Otherwise, if the credibility of the determined verification result is greater than or equal to the preset credibility threshold, scores of each item corresponding to the target sampling reply are determined based on the verification result. The specific value of the preset credibility threshold is set according to actual needs, which is not limited here.

[0101] In some embodiments of the present application, the reward model can also include a length reward function.

[0102] Figure 6 is a flowchart of another method for scoring the target sampling reply using the reward model provided by the embodiments of the present application. Wherein, Figure 6 The steps S601 to S603 in the embodiment shown are basically the same as the steps S401 to S403 in the embodiment shown Figure 4 The steps S401 to S403 in the embodiment shown are basically the same as the steps S401 to S403 in the embodiment shown, which will not be repeated here. As Figure 6 The method further includes the following steps:

[0103] In step S604, the length reward function is called, and the target length is generated based on the long text generation instruction.

[0104] In step S605, in response to determining that the absolute value of the difference between the text length of the target sampling reply and the target length is less than or equal to the preset deviation threshold, the length reward value of the target sampling reply is determined as a first value.

[0105] The first value is the maximum value of the length reward.

[0106] In step S606, in response to determining that the absolute value of the difference between the text length of the target sampling reply and the target length is greater than the preset deviation threshold, the length reward value of the target sampling reply is determined as a second value.

[0107] The difference between the first value and the second value is positively correlated with the absolute value of the difference.

[0108] In step S607, the score of the target sampling reply is determined based on the mean of the scores of each item corresponding to the target sampling reply and the length reward value of the target sampling reply.

[0109] In some embodiments of the present application, when scoring the target sampling reply using the reward model, a length reward function can also be called to generate a target length based on the long text generation instruction.

[0110] If it is determined that the absolute value of the difference between the text length of the target sampling reply and the target length is less than or equal to a preset deviation threshold, it is determined that the length reward value of the target sampling reply is a first value. The first value is the maximum value of the length reward.

[0111] On the contrary, if it is determined that the absolute value of the difference between the text length of the target sampling reply and the target length is greater than the preset deviation threshold, it is determined that the length reward value of the target sampling reply is a second value. The difference between the first value and the second value is positively correlated with the absolute value of the difference.

[0112] Further, the score of the target sampling reply can be determined based on the mean of the scores of each item corresponding to the target sampling reply and the length reward value of the target sampling reply. In an example, the weighted sum of the mean of the scores of each item corresponding to the target sampling reply and the length reward value of the target sampling reply can be determined as the score of the target sampling reply. Alternatively, the mean of the scores of each item corresponding to the target sampling reply and the length reward value of the target sampling reply can be combined in other ways to obtain the score of the target sampling reply, which is not limited here.

[0113] That is, the embodiments of the present application can also set a reasonable length constraint target for the generated content by data augmentation on the original instruction. The reward signal aims to encourage the model to generate content of moderate length, avoiding excessively long or short output.

[0114] For example, the existing large language model can be used to analyze the original instruction and the list of constraints to be met, predict a reasonable target generation length, and integrate the length constraint back into the original instruction. For each sampling reply generated by the policy model, a length reward is calculated based on the target generation length, and a relative deviation threshold is set. If the length of the sampling reply is within the threshold range, a full score reward is obtained; if it is out of range, the reward decreases with the increase of the deviation.

[0115] In some embodiments of the present application, the pre-trained agent can be a pre-trained large language model, such as large language model Qwen3-235B-A22B. On the other hand, the reinforcement learning algorithm can be a GRPO (Group Relative Policy Optimization) algorithm.

[0116] Further, the step of updating the model parameters of the policy model based on the relative advantage can be updating the model parameters of the policy model to increase the generation probability of the sampled reply with positive relative advantage and decrease the generation probability of the sampled reply with negative relative advantage.

[0117] After the reward model scores the N sampled replies respectively, the above reward signals can be integrated to optimize and guide the language model through reinforcement learning, so that the language model can generate high-quality long content that meets the specific instruction requirements.

[0118] In an example, the policy model and the reward model can be deployed first, and the GRPO reinforcement learning algorithm only using the reward model is used to train the instruction data, i.e., the long text generation instruction and the corresponding adaptive constraint standard list.

[0119] In each training batch, a group of sampled replies is obtained by sampling the policy model, and the relative advantage of each generated sample is estimated by intra-group comparison based on the reward value, and the parameter update of the policy model is guided. The reward value is calculated by the reward model.

[0120] The following takes the long text generation task of generating a disk topology design and dataset layout document as an example to illustrate the working process of the long text generation model optimization method based on adaptive constraint reward provided by the embodiments of the present application.

[0121] The long text generation instruction is:

[0122] Generate a NAS (Network Attached Storage) disk topology design and dataset layout document based on TrueNAS Scale construction, and the available disks are as follows:

[0123] -2 18TB disks;

[0124] -5 14TB disks;

[0125] -3 12TB disks;

[0126] -4 8TB disks;

[0127] -2 120GB disks;

[0128] -2 8TB slow disks;

[0129] There are 17 hard drive bays. The two smallest disks will be used for a mirror pool, serving as boot devices. The two slow hard drives are SMR (Shingled Magnetic Recording) disks, which will be used in their own pool, providing Time Machine targets for certain Macs. The topology can be freely designed to optimize for redundancy, space, and performance. The stored data includes video files, music files, disk images, archived software, photos, and some text files. While most data can be rebuilt or downloaded, some data cannot be replaced. Bays can be left for hot spares or for future expansion. RAIDZ (Redundant Array of Independent Disks) is not recommended as mirror arrays rebuild faster.

[0130] The main considerations for the above requirements are redundancy to reduce the risk of data loss, and space efficiency and cost-effectiveness. Performance is not a primary consideration.

[0131] After obtaining the above long text generation instructions, the existing large language model Qwen3-235B-A22B can be used as a command analyzer to design a Prompt template to extract explicit requirements and identify implicit expectations for each user instruction. The analysis process is as follows:

[0132] The instruction requires providing a detailed disk topology and data layout for a NAS using TrueNAS Scale, and assigning specific disks to different roles (boot, Time Machine, and general storage). The user emphasizes redundancy, space efficiency, and cost-effectiveness, while avoiding the use of RAID-Z arrays and preferring mirror arrays. The implicit expectations include proper use of disk types, optimal redundancy, and logical data set organization. The response should also leave room for expansion and clearly explain the design, avoiding unnecessary complexity.

[0133] Next, based on the above analysis of user explicit requirements and implicit expectations, the original instruction can be deconstructed into a series of clear, unique, and verifiable adaptive constraint standards list. The resulting adaptive constraint standards list may be, for example:

[0134] <Checklist>

[0135] Question 1: Does the design use two smallest capacity disks (120GB) to form a mirror pool as boot devices?

[0136] Question 2: Are the two low-speed 8TB SMR disks specifically used for a Time Machine backup pool?

[0137] Question 3: Does the design use mirrored vdevs instead of RAIDZ for all data pools as required?

[0138] Question 4: Is the data set layout logically organized by data type (e.g., videos, music, photos, text files)?

[0139] Question 5: Does the design provide redundancy for all data pools to reduce the risk of data loss?

[0140] Question 6: Does the topology efficiently utilize available space, considering future expansion or hot spare disks?

[0141] Question 7: Are low-speed or small-capacity disks reasonably isolated to minimize their impact on main storage performance and reliability?

[0142] Question 8: Do the answers clearly state the design decisions and demonstrate how they prioritize redundancy, space efficiency, and cost-effectiveness?

[0143] < / Checklist>

[0144] Then input the long text generation instruction into the policy model of the reinforcement learning algorithm to get N sampled answers. Call the reward model for each sampled answer based on the adaptive constraint standard list to verify it. The verification result may be, for example:

[0145] Question 1: Does the design use the two smallest disks (120GB) in the mirror pool as the boot device?

[0146] [The answer clearly states under "Boot Pool (2 disk slots)" that "2 x 120GB disks in the mirror (as specified)", which meets the requirements of the instructions.]

[0147] <Answer>

[0148] Fully satisfied

[0149] < / Answer>

[0150] On the other hand, we can also set a reasonable length constraint target for the generated content by data augmentation on the original instruction. The reward signal aims to encourage the model to generate content of appropriate length, avoiding excessively long or short outputs. The target length data augmentation is as follows:

[0151] The instructions require a detailed storage topology and dataset layout for a TrueNAS Scale NAS that uses a set of specific drives with varying size and performance characteristics. The evaluation checklist confirms that the response must include multiple distinct sections: proper allocation of startup and Time Machine pools (Issue 1), use of RAIDZ mirrors (Issue 2), logical dataset separation (Issue 3), efficient use of drive bays (Issue 4), strategic use of large drives for critical data (Issue 5), and a trade-off justification (Issue 6). Each item on the checklist requires explanation, justification, and structural layout, so a comprehensive yet not overly lengthy response is required. Explaining design decisions, justifying the reasonableness of redundant choices, and mapping dataset requirements add to the information burden. Given the complexity of the issues and the checklist requirements, a detailed yet concise technical explanation is appropriate.

[0152] <length>

[0153] [800, 2000]

[0154] < / length>

[0155] Finally, the above reward signals are integrated to optimize and guide the language model through reinforcement learning, enabling it to generate high-quality long-form content that meets specific instruction requirements.

[0156] Figure 7 is a system structure diagram for implementing the long text generation model optimization method based on adaptive constraint rewards provided by the embodiments of the present application. As Figure 7 shown, the system can include an adaptive constraint standard checklist generation module and a reinforcement learning algorithm module.

[0157] In the adaptive constraint standard checklist generation module, the target user instructions obtained by filtering the self-sample data can be parsed to generate an adaptive constraint standard checklist including n questions. n is a positive integer.

[0158] In the reinforcement learning algorithm module, the policy model can receive user input long text generation instructions and generate N sample replies, O1, O2, …, ON. For any sample reply Oi, on the one hand, the reward model can determine its score based on the sample reply Oi, the long text generation instruction, and the adaptive constraint standard checklist. The score includes the score for each question in the adaptive constraint standard checklist, a total of n scores. The average of the n scores can obtain the score reward of the sample reply Oi. On the other hand, the length reward can also be calculated. Wherein, i is a positive integer greater than 1 and less than or equal to n.

[0159] Finally, the relative advantage of the sampled reply Oi can be determined based on the length reward and the score reward, and the relative advantage is fed back to the policy model to update the model parameters of the policy model, so as to realize the optimization of the reinforcement learning algorithm.

[0160] By adopting the technical solutions provided in the embodiments of the present application, the abstract user instruction is concretized into a fine-grained and verifiable constraint standard list, and an accurate reward signal is constructed based on the constraint standard list to guide the reinforcement learning, so that the specific instruction requirements in different task scenarios can be adapted, the optimization efficiency of the reinforcement learning algorithm is improved, and the user experience is improved.

[0161] All the optional technical solutions described above can be combined to form optional embodiments of the present application, and will not be described one by one here.

[0162] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0163] Figure 8 is a schematic diagram of a long text generation model optimization apparatus based on adaptive constraint rewards provided by an embodiment of the present application. As shown in Figure 8 , the apparatus comprises:

[0164] The acquisition module 801 is configured to acquire sample data; the sample data comprises human-computer interaction data, and the human-computer interaction data at least comprises a long text generation task.

[0165] The determination module 802 is configured to call a pre-trained agent and determine an adaptive constraint standard list based on the sample data.

[0166] The sampling module 803 is configured to acquire a long text generation instruction, input the long text generation instruction into a policy model of a reinforcement learning algorithm, and obtain N sampled replies; N is a positive integer greater than 1.

[0167] The scoring module 804 is configured to score the N sampled replies respectively using a reward model of the reinforcement learning algorithm, and determine the relative advantage of each sampled reply based on the scores of the sampled replies; wherein the reward model at least comprises a quantification model, and the quantification condition of the quantification model at least comprises the adaptive constraint standard list.

[0168] The optimization module 805 is configured to update the model parameters of the policy model based on the relative advantage.

[0169] According to the technical scheme provided in the embodiments of the present application, the sample data including the long text generation task is filtered out from the massive human-computer interaction data, and the pre-trained agent is used to determine the adaptive constraint standard list based on the sample data; the strategy model of the reinforcement learning algorithm is used to generate N sample replies based on the long text generation instruction, the reward model is used to score the N sample replies based on the adaptive constraint standard list, and the relative advantages of the sample replies are determined, and then the model parameters of the strategy model are updated according to the relative advantages, and the optimization of the reinforcement learning algorithm is completed, so that the optimized reinforcement learning algorithm for generating long text is obtained without large-scale high-quality artificial annotation data, and the generated long text can be quantitatively evaluated, which can adapt to the specific instruction requirements in different task scenarios, improves the optimization efficiency of the reinforcement learning algorithm, and improves the user experience.

[0170] In some embodiments, determining the adaptive constraint standard list based on the sample data comprises: obtaining target user instructions in the sample data, the target user instructions being long text generation related instructions; using the pre-trained agent to identify explicit requirements in the target user instructions; the explicit requirements at least including key information, format requirements and limitation conditions in the target user instructions; using the pre-trained agent to identify implicit expectations in the target user instructions; the implicit expectations at least including contexts, user preferences and task types of the instructions; and determining the adaptive constraint standard list based on the explicit requirements and the implicit expectations.

[0171] In some embodiments, determining the adaptive constraint standard list based on the explicit requirements and the implicit expectations comprises: generating an initial adaptive constraint standard list based on the explicit requirements and the implicit expectations; calling a pre-trained evaluation model to evaluate the initial adaptive constraint standard list; in response to determining that the initial adaptive constraint standard list does not meet at least one of the preset conditions, updating the initial adaptive constraint standard list until the updated adaptive constraint standard list meets all the preset conditions; determining the updated adaptive constraint standard list as the adaptive constraint standard list; wherein the preset conditions include: each requirement in the list is a question that can be answered with a three-level score, the three-level score including complete satisfaction, partial satisfaction and non-satisfaction; and the list covers all key elements of the instructions, and the number of items in the list is less than a preset number threshold; and the terms in the list are clear and accurate; and the correlation between the items in the list is less than a preset correlation threshold.

[0172] In some embodiments, the scoring the target sampled reply using the reward model comprises: inputting the adaptive constraint standard list and the target sampled reply into a quantization model of the reward model; the quantization model determining a verification result of the target sampled reply corresponding to each item in the adaptive constraint standard list; the verification result comprising at least one of complete satisfaction, partial satisfaction and non-satisfaction; determining a score of the target sampled reply corresponding to each item based on the verification result; wherein the verification result of complete satisfaction corresponds to a first score, the verification result of partial satisfaction corresponds to a second score, and the verification result of non-satisfaction corresponds to a third score, and the first score is greater than the second score, and the second score is greater than the third score; determining a mean value of the scores of the target sampled reply corresponding to each item as the score of the target sampled reply; wherein the target sampled reply is any one of the N sampled replies.

[0173] In some embodiments, the verification result further comprises a verification basis; after determining the verification result, the method further comprises: determining a credibility of the verification result based at least on the verification basis; in response to determining that the credibility of the verification result is less than a preset credibility threshold, updating the quantization model, and determining the verification result of the target sampled reply again using the updated quantization model; in response to determining that the credibility of the verification result is greater than or equal to the preset credibility threshold, determining the score of the target sampled reply corresponding to each item based on the verification result.

[0174] In some embodiments, the reward model further comprises a length reward function; the scoring the target sampled reply using the reward model further comprises: calling the length reward function to generate a target length based on the long text generation instruction; in response to determining that an absolute value of a difference between the text length of the target sampled reply and the target length is less than or equal to a preset deviation threshold, determining that the length reward value of the target sampled reply is a first value; the first value is a maximum value of the length reward; in response to determining that the absolute value of the difference between the text length of the target sampled reply and the target length is greater than the preset deviation threshold, determining that the length reward value of the target sampled reply is a second value; the difference between the first value and the second value is positively correlated with the absolute value of the difference; determining the score of the target sampled reply based on the mean value of the scores of the target sampled reply corresponding to each item and the length reward value of the target sampled reply.

[0175] In some embodiments, the pre-trained agent is a pre-trained large language model, and the reinforcement learning algorithm is a GRPO algorithm; the step of updating the model parameters of the policy model based on the relative advantage comprises: updating the model parameters of the policy model to increase the generation probability of the sampled reply with a positive relative advantage and to reduce the generation probability of the sampled reply with a negative relative advantage.

[0176] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0177] Figure 9 is a schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 9 the electronic device 9 of this embodiment includes a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. The processor 901 implements the steps in each of the above method embodiments when executing the computer program 903. Alternatively, the processor 901 implements the functions of each module / unit in each of the above apparatus embodiments when executing the computer program 903.

[0178] The electronic device 9 can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The electronic device 9 can include but is not limited to the processor 901 and the memory 902. Those skilled in the art can understand that Figure 9 the electronic device 9 is merely an example and does not constitute a limitation on the electronic device 9, and can include more or fewer components or different components than those shown.

[0179] The processor 901 can be a central processing unit (CPU), or other general purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc.

[0180] The memory 902 can be an internal storage unit of the electronic device 9, for example, a hard disk or a memory of the electronic device 9. The memory 902 can also be an external storage device of the electronic device 9, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 9. The memory 902 can also include both the internal storage unit and the external storage device of the electronic device 9. The memory 902 is used to store computer programs and other programs and data required by the electronic device.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0182] If the integrated module / unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can be executed by a processor to implement the steps of each method embodiment. The computer program can include computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0183] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An optimization method for a long text generation model based on adaptive constraint reward, characterized in that, include: Obtain sample data; the sample data includes human-computer interaction data, and the human-computer interaction data includes at least a long text generation task; Invoke the pre-trained agent to determine an adaptive constraint standard list based on the sample data; Obtain the long text generation instruction, input the long text generation instruction into the policy model of the reinforcement learning algorithm, and obtain N sampled responses; N is a positive integer greater than 1; The reward model of the reinforcement learning algorithm is used to score the N sampled responses respectively, and the relative advantage of each sampled response is determined based on the score of each sampled response; wherein, the reward model includes at least a quantization model, and the quantization conditions of the quantization model include at least the adaptive constraint criterion list; The model parameters of the strategy model are updated based on the relative advantage.

2. The method according to claim 1, characterized in that, Based on the sample data, a list of adaptive constraint criteria is determined, including: Obtain the target user instructions from the sample data, wherein the target user instructions are long text generation related instructions; The pre-trained agent is used to identify explicit requirements in the target user's instructions; the explicit requirements include at least key information, format requirements, and constraints in the target user's instructions. The pre-trained agent is used to identify implicit expectations in the target user's instructions; the implicit expectations include at least the context of each instruction, user preferences, and task type. The adaptive constraint criteria list is determined based on the explicit requirements and the implicit expectations.

3. The method according to claim 2, characterized in that, The adaptive constraint criteria list is determined based on the explicit requirements and the implicit expectations, including: An initial list of adaptive constraint criteria is generated based on the explicit requirements and the implicit expectations; The pre-trained evaluation model is invoked to evaluate the initial adaptive constraint criteria list; In response to determining that the initial adaptive constraint standard list does not meet at least one of the preset conditions, the initial adaptive constraint standard list is updated until the updated adaptive constraint standard list meets all preset conditions. The updated adaptive constraint criteria list is determined to be the adaptive constraint criteria list; The preset conditions include: Each requirement in the list is a question that can be answered using a three-level rating system, which includes fully satisfied, partially satisfied, and not satisfied; and The list covers all key elements of the instruction, and the number of items in the list is less than a preset threshold; and The terminology in the list is clear and accurate; and The correlation between items in the list is less than the preset correlation threshold.

4. The method according to claim 1, characterized in that, The target sampled response is scored using a reward model, including: The adaptive constraint criteria list and the target sampled response are input into the quantization model of the reward model; The quantization model determines the verification result for each item in the adaptive constraint criteria list corresponding to the target sampled response; the verification result includes at least one of fully satisfied, partially satisfied, and not satisfied; Based on the verification results, the scores for each item corresponding to the target sampled response are determined; wherein, the verification results that fully satisfy the first score correspond to the first score, those that partially satisfy the second score correspond to the second score, and those that do not satisfy the third score correspond to the third score, and the first score is greater than the second score, and the second score is greater than the third score; The average score of each item in the target sampled response is determined as the score of the target sampled response; The target sampled response is any one of the N sampled responses.

5. The method according to claim 4, characterized in that, The verification results also include the verification basis; After determining the verification result, the method further includes: The credibility of the verification results shall be determined at least based on the verification criteria. In response to determining that the credibility of the verification result is less than a preset credibility threshold, the quantization model is updated, and the updated quantization model is used to determine the verification result of the target sample response again; In response to determining that the credibility of the verification result is greater than or equal to the preset credibility threshold, the score of each item corresponding to the target sampled response is determined based on the verification result.

6. The method according to claim 4, characterized in that, The reward model also includes a length reward function; Using a reward model to score the target sampled response also includes: The length reward function is invoked to generate the target length based on the long text generation instruction; In response to determining that the absolute value of the difference between the text length of the target sampled response and the target length is less than or equal to a preset deviation threshold, the length reward value of the target sampled response is determined to be a first value; the first value is the maximum value of the length reward. In response to determining that the absolute value of the difference between the text length of the target sampled response and the target length is greater than the preset deviation threshold, the length reward value of the target sampled response is determined to be a second value; the difference between the first value and the second value is positively correlated with the absolute value of the difference; The score of the target sampled response is determined based on the average score of each item corresponding to the target sampled response and the length reward value of the target sampled response.

7. The method according to any one of claims 1 to 6, characterized in that, The pre-trained agent is a pre-trained large language model, and the reinforcement learning algorithm is the Group Relative Policy Optimization (GRPO) algorithm. The step of updating the model parameters of the strategy model based on the relative advantage is to update the model parameters of the strategy model to increase the generation probability of sampled responses with positive relative advantage and decrease the generation probability of sampled responses with negative relative advantage.

8. An optimization device for a long text generation model based on adaptive constraint reward, characterized in that, include: The acquisition module is configured to acquire sample data; the sample data includes human-computer interaction data, and the human-computer interaction data includes at least a long text generation task; The determination module is configured to invoke a pre-trained agent to determine an adaptive constraint criteria list based on the sample data; The sampling module is configured to acquire long text generation instructions, input the long text generation instructions into the policy model of the reinforcement learning algorithm, and obtain N sampled responses; N is a positive integer greater than 1; The scoring module is configured to score the N sampled responses using the reward model of the reinforcement learning algorithm, and determine the relative advantage of each sampled response based on the score of each sampled response; wherein the reward model includes at least a quantization model, and the quantization conditions of the quantization model include at least the adaptive constraint criteria list; The optimization module is configured to update the model parameters of the strategy model based on the relative advantage.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images