Strategy model training method and device, medium and equipment

Abstract

This application discloses a strategy model training method, apparatus, medium, and device. The method involves acquiring prompt words corresponding to input text and inputting these prompt words into an initial strategy model to obtain a first preset number of initial responses. The prompt words and each initial response are then input into a preset evaluation model to obtain an evaluation result for each initial response. Based on the evaluation results, a second preset number of responses to be modified are selected from the initial responses. Each response to be modified is input into the initial strategy model, and a second preset number of optimized responses are output based on modification requirements. A response group for the prompt words is constructed using the first preset number of initial responses and the second preset number of optimized responses. The initial strategy model is then trained based on the in-group reward value of each target response in the response group to obtain a trained strategy model. This method can shorten the exploration path during model self-exploration, reduce computer resource consumption, improve processing power, and enhance the inference accuracy of the strategy model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a strategy model training method, apparatus, medium, and device. Background Technology

[0002] Currently, with the rapid development of artificial intelligence technology, large language models are widely used in various business scenarios such as content generation, intelligent dialogue, and industry consulting. In order to improve the alignment, output quality and task adaptability of the model in specific tasks, reinforcement learning technology has become one of the core technologies in the post-training stage of large language models. Its training efficiency and effect directly affect the business implementation pace and application value of large language models.

[0003] In related technologies, online reinforcement learning algorithms primarily rely on reward signals to drive the policy model to explore and optimize its generation capabilities during actual training. However, this purely self-exploration training mode may suffer from several drawbacks due to a lack of effective external guidance. These drawbacks include: insufficient exploration due to deviations in the exploration direction (i.e., inability to generate high-quality outputs that meet the task objectives); inefficient exploration processes leading to significantly prolonged training cycles; or policy models obtaining high rewards by circumventing core task requirements, resulting in outputs that fail to meet actual business needs, such as generating redundant content or deviating from the topic. Consequently, the policy model trained through reinforcement learning exhibits poor inference accuracy during the inference phase, and the blind self-exploration during training consumes significant computer processing resources, resulting in high resource consumption and training costs. Summary of the Invention

[0004] This application provides a strategy model training method, apparatus, medium, and device. By modifying the initial answer output by the initial strategy model during self-exploration, it can help the model avoid ineffective exploration and anchor the correct direction, shorten the exploration path during self-exploration, thereby reducing the consumption of computer processing resources, reducing computer resource occupation, improving processing power, and avoiding the problem of the strategy model obtaining high rewards by circumventing the core requirements of the task, thus improving the reasoning accuracy of the strategy model.

[0005] To address the aforementioned technical problems, this application provides the following technical solutions:

[0006] A policy model training method, comprising:

[0007] Obtain the prompt words corresponding to the input text and input the prompt words into the initial strategy model to obtain the first preset number of initial answers output by the initial strategy model;

[0008] The prompt words and each initial answer are input into a preset evaluation model to obtain an evaluation result for each initial answer;

[0009] Based on the evaluation results, a second preset number of answers to be modified are selected from the initial answers;

[0010] Each of the answers to be modified is input into the initial strategy model, and the optimized answer after modification of each answer to be modified is output based on the modification requirements, so as to obtain a second preset number of optimized answers;

[0011] The initial response group of the prompt word is constructed using the first preset number of initial responses and the second preset number of optimized responses. The initial strategy model is trained based on the in-group reward value of each target response in the response group to obtain the trained strategy model.

[0012] A policy model training device, comprising:

[0013] The first input unit is used to obtain the prompt words corresponding to the input text and input the prompt words into the initial strategy model to obtain a first preset number of initial answers output by the initial strategy model.

[0014] The second input unit is used to input the prompt words and each initial answer into a preset evaluation model to obtain the evaluation result of each initial answer;

[0015] A filtering unit is used to filter out a second preset number of answers to be modified from the initial answers based on the evaluation results;

[0016] The third input unit is used to input each of the answers to be modified into the initial strategy model, and output the optimized answer after modification of each of the answers to be modified based on the modification requirements, so as to obtain a second preset number of optimized answers.

[0017] The construction unit is used to construct a group of responses to the prompt words using the first preset number of initial responses and the second preset number of optimized responses, and to train the initial strategy model based on the in-group reward value of each target response in the group of responses to obtain the trained strategy model.

[0018] In some implementations, the evaluation result includes at least an evaluation score, and the screening unit includes:

[0019] The sorting subunit is used to sort each of the initial answers in ascending order of evaluation scores to obtain a sorted sequence;

[0020] The first determining subunit is used to determine the initial answers that are in the first second preset number in the sorting sequence as answers to be modified, thereby obtaining the second preset number of answers to be modified.

[0021] In some implementations, the evaluation result further includes evaluation opinions, and the third input unit is used for:

[0022] Obtain the target evaluation opinion for each of the aforementioned answers to be modified;

[0023] Based on the prompt words, each of the answers to be modified, and the target evaluation opinions for each of the answers to be modified, a first answer modification template is constructed;

[0024] The modification requirement for the first answer modification template is to make the initial strategy model modify the corresponding answer to be modified by referring to the target evaluation opinion of each answer to be modified.

[0025] The first answer modification template is input into the initial strategy model, and the optimized answer after modification for each answer to be modified is output to obtain a second preset number of optimized answers.

[0026] In some implementations, the evaluation result further includes evaluation opinions, and the third input unit is further used for:

[0027] Obtain the target evaluation opinion for each of the aforementioned answers to be modified;

[0028] Based on the prompt words, each of the answers to be modified, the target evaluation opinion for each of the answers to be modified, and the reference answers for the prompt words, a second answer modification template is constructed;

[0029] The modification requirement for the second answer modification template is to make the initial strategy model modify the corresponding answer to be modified by referring to the reference answer and the target evaluation opinion of each answer to be modified.

[0030] The second answer modification template is input into the initial strategy model, and the optimized answer after modification for each answer to be modified is output to obtain a second preset number of optimized answers.

[0031] In some embodiments, the third input unit is further configured to:

[0032] Based on the prompt words, each of the answers to be modified, and the reference answers for the prompt words, a third answer modification template is constructed;

[0033] The modification requirement for the third answer modification template is to make the initial strategy model modify each of the answers to be modified by referring to the reference answer;

[0034] The third answer modification template is input into the initial strategy model, and the optimized answer after modification for each answer to be modified is output to obtain a second preset number of optimized answers.

[0035] In some embodiments, the third input unit is further configured to:

[0036] The prompt words are input into the teacher model to obtain the reference answer output by the teacher model.

[0037] In some implementations, the building unit is used for:

[0038] Calculate the in-group reward value for each target answer in the answer group;

[0039] The relative advantage value of each target answer in the answer group is determined by the in-group reward value of each target answer;

[0040] The objective function of the reinforcement learning algorithm is calculated based on the relative advantage value and KL divergence of each target answer;

[0041] The gradient ascent algorithm that maximizes the objective function is run to update the parameters of the initial policy model, resulting in the trained policy model.

[0042] A computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the above-described strategy model training method.

[0043] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described strategy model training method.

[0044] A computer program product or computer program includes computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions to implement the aforementioned policy model training method.

[0045] This application embodiment obtains prompt words corresponding to input text and inputs the prompt words into an initial strategy model to obtain a first preset number of initial answers output by the initial strategy model; inputs the prompt words and each initial answer into a preset evaluation model to obtain an evaluation result for each initial answer; based on the evaluation results, selects a second preset number of answers to be modified from the initial answers; inputs each answer to be modified into the initial strategy model, and outputs an optimized answer after modification for each answer to be modified based on modification requirements to obtain a second preset number of optimized answers; constructs an answer group for the prompt words using the first preset number of initial answers and the second preset number of optimized answers, and trains the initial strategy model based on the in-group reward value of each target answer in the answer group to obtain a trained strategy model.

[0046] Therefore, after the initial strategy model outputs a first preset number of initial answers based on the prompt words, the prompt words and each initial answer are input into a preset evaluation model. The preset evaluation model evaluates each initial answer and, based on the evaluation results, selects a second preset number of answers to be modified from the initial answers. Each answer to be modified is then input into the initial strategy model, which outputs an optimized answer after modification based on the modification requirements, resulting in high-quality optimized answers. This leads to an answer group constructed from the initial answers and optimized answers. The initial strategy model is then trained based on the in-group reward value of each target answer in the answer group. Compared to related technologies where the self-exploration method during the enhanced training phase leads to insufficient exploration, deviation from the correct direction, lower model prediction accuracy, and higher computer resource consumption, this application helps the model avoid ineffective exploration and anchor to the correct direction, shortening the exploration path during self-exploration. This reduces the consumption of computer processing resources, improves processing power, and avoids the problem of the strategy model obtaining high rewards by circumventing the core requirements of the task, thus improving the inference accuracy of the strategy model.

[0047] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of a strategy model training system provided in an embodiment of this application.

[0050] Figure 2 This is a flowchart illustrating the strategy model training method provided in an embodiment of this application.

[0051] Figure 3 This is a schematic diagram of the framework for training the strategy model provided in the embodiments of this application.

[0052] Figure 4 This is a schematic diagram of the structure of the strategy model training device provided in the embodiments of this application.

[0053] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0054] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] It is understood that in the specific implementation of this application, data such as input text and prompt words are involved. When the embodiments of this application are applied to specific products or technologies, permission or consent from the subject is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0056] Furthermore, when the embodiments of this application require separate permission or separate consent to obtain input text, prompt words, and other related data, the necessary input text, prompt words, and other related data for enabling the embodiments of this application to operate normally shall be obtained only after obtaining separate permission or separate consent to obtain input text, prompt words, and other related data.

[0057] It should be noted that while some processes described in the specification, claims, and accompanying drawings contain multiple steps that appear in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not represent any particular order of execution. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0059] Reinforcement learning (RL) is a learning paradigm in artificial intelligence. Its core goal is to enable an agent to learn decision-making strategies that maximize long-term rewards through continuous interaction and trial and error with the environment. In other words, it enables artificial intelligence (AI) to grow from experience like a human, knowing what actions will yield good results and what actions will yield bad results.

[0060] Reinforcement learning algorithms are broadly classified into online and offline categories. Offline reinforcement learning algorithms include Direct Preference Optimization (DPO), Kahneman-Tversky Optimization (KTO), Offline Reinforcement Learning with Preference Optimization (ORPO), and Simple Preference Optimization (SimPO), among others. Online reinforcement learning algorithms include Proximal Policy Optimization (PPO), among others. The difference lies in whether the large language model (also known as the policy model) to be fine-tuned interacts with the environment and generates empirical data for training during the reinforcement learning training process. Offline reinforcement learning does not involve interaction, while online reinforcement learning does, thus gaining wider application.

[0061] Please see Figure 1 , Figure 1 This is a schematic diagram of a strategy model training system provided in an embodiment of this application. It includes a server 140, an internet connection 130, a gateway 120, and a client 110, etc.

[0062] Client 110 includes, but is not limited to, mobile phones, tablets, pre-configured desktop computers, laptops, tablets, and other electronic devices. Client 110 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0063] Server 140 refers to a computer system that provides storage services to client 110, and its internal components include a database for storing related operations of client 110. Compared to a regular client 110, server 140 has higher requirements in terms of stability, security, and performance. Server 140 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines), etc., and the server stores pre-defined timing logic.

[0064] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Input text sent from client 110 to server 140 is routed through gateway 120 to the appropriate server 140.

[0065] The strategy model training method of this application embodiment can be implemented on server 140.

[0066] The server obtains the prompt words corresponding to the input text and inputs the prompt words into an initial strategy model to obtain a first preset number of initial answers output by the initial strategy model; the prompt words and each initial answer are input into a preset evaluation model to obtain an evaluation result for each initial answer; based on the evaluation results, a second preset number of answers to be modified are selected from the initial answers; each answer to be modified is input into the initial strategy model, and optimized answers after modification are output based on the modification requirements to obtain a second preset number of optimized answers; an answer group for the prompt words is constructed using the first preset number of initial answers and the second preset number of optimized answers, and the initial strategy model is trained based on the in-group reward value of each target answer in the answer group to obtain a trained strategy model.

[0067] It should be noted that, Figure 2 The schematic diagram of the strategy model training system shown is merely an example. The strategy model training system and scenario described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of strategy model training and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0068] In this embodiment, the description will be from the perspective of a strategy model training device. Specifically, the strategy model training device can be integrated into a computer device that has a storage unit and a microprocessor installed, thus having computing power. The computer device can be a server, and in this embodiment, the computer device will be described as a server.

[0069] Please see Figure 2 , Figure 2 This is a flowchart illustrating the strategy model training method provided in an embodiment of this application. The strategy model training method is applied to a server and includes:

[0070] In step 201, the prompt words corresponding to the input text are obtained and the prompt words are input into the initial strategy model to obtain the first preset number of initial answers output by the initial strategy model.

[0071] Understandably, the policy model plays a central decision-making role in reinforcement learning training. The input text is relevant to the reasoning task; for example, in a text generation model, the input text is the topic of the text generation task, such as "Are sweet or savory rice dumplings tastier?". The prompt is an instruction assembled from the input text and a prompt template corresponding to the task, used to guide the initial policy model to output an answer.

[0072] The prompt corresponding to the input text is shown in the following example:

[0073] "As a self-media creator, please write a short video script based on a given topic. Your script must meet the standards I provide."

[0074] Topic selection {input text};

[0075] Are sweet or savory rice dumplings tastier?

[0076] The standard requires {requirements};

[0077] The output should contain 100 characters.

[0078] Output {initial answer};

[0079] Please first output your thought process, then... <think>< / think> Within the tags, output a copy that meets the requirements. Output the copy itself, without a title, and do not output any other content besides the copy.

[0080] The initial strategy model, also known as the large language model to be trained (such as a topic selection copywriting generation model), already possesses basic generation capabilities. It can be a pre-trained model, a supervised learning model, or a simple baseline model, and requires further optimization through reinforcement learning. The initial response is the first preset number of responses generated by the initial strategy model to the prompt words, denoted as... That is, to obtain a set The initial responses varied in quality, causing the model to deviate in its training direction and fail to output results that meet actual business needs.

[0081] In step 202, the prompt words and each initial answer are input into the preset evaluation model to obtain the evaluation result of each initial answer.

[0082] Specifically, prompts and each initial response can be used to construct an evaluation template, which clarifies the evaluation scenario (such as short video scripts) and standards (such as conversational style and relevance to the topic), thereby guiding the evaluation model to score. A pre-set evaluation model (such as GPT-4o, a model with comprehension and scoring capabilities) has been trained on evaluation task data and can evaluate the quality of responses (relevance, fluency, and scenario suitability) to output evaluation results.

[0083] Taking the initial strategy model as the copy generation model as an example, the following content can be used to provide the first preset number of initial answers corresponding to the prompt words to the preset evaluation model for evaluation:

[0084] "As an impartial copywriting evaluation expert, please carefully analyze whether the copywriting generated from the selected topic meets the standard based on the application scenario of the copywriting and each dimension of the copywriting evaluation criteria, and explain the reasons."

[0085] ## Copywriting Application Scenarios

[0086] Short video scripts that develop content around a specific topic.

[0087] ##Evaluation Criteria {requirements}

[0088] ## Topic Selection {Enter Text}

[0089] ## Copywriting {Initial Q&A}

[0090] ## Output Format

[0091] Output the final evaluation results in JSON list format.

[0092] By replacing the environmental rewards of traditional reinforcement learning with an external evaluation model, the problems of difficulty in designing reward functions and susceptibility to model hacks (such as padding the word count to get a high score) are solved; the structured design of the evaluation template (binding task scenarios and standards) ensures the consistency of evaluation and provides a basis for subsequent precise modifications.

[0093] In step 203, based on the evaluation results, a second preset number of answers to be modified are selected from the initial answers.

[0094] After obtaining the evaluation results for each initial response, a second preset number of responses to be modified can be selected from the initial responses based on the evaluation results, for example... indivual, Thus, this The initial responses are designated as "responses to be modified" for subsequent revisions. This optimizes the responses to be modified, thereby improving the average quality of the entire group with minimal computational cost.

[0095] In some implementations, the evaluation result includes at least an evaluation score, and the step of selecting a second preset number of answers to be modified from the initial answers based on the evaluation result includes:

[0096] (1) Sort each of the initial answers in ascending order of the evaluation scores to obtain a sorted sequence;

[0097] (2) The initial answers in the first second preset number in the sorting sequence are determined as answers to be modified, and the second preset number of answers to be modified are obtained.

[0098] The evaluation result is the evaluation score within the first preset number of initial response groups, denoted as . ,Right now this The evaluation score for each of the initial responses. Because the quality of the initial responses varies, in order to... The second preset number is selected from the initial responses. Each initial answer is considered a candidate for modification, and therefore, the selection process is based on the evaluation score corresponding to each initial answer.

[0099] Specifically, the method for filtering answers to be modified is to sort each initial answer according to its evaluation score from smallest to largest, resulting in a sorted sequence, and then select the second-to-last preset number of answers from the sorted sequence. The initial responses with low evaluation scores are identified as responses to be modified, resulting in a second preset number of responses to be modified.

[0100] For example, the first preset number (the initial total number of answers). The answer is 8, meaning the initial answer is... Second preset quantity (number of answers to be modified) 2 ( Evaluation results (scores range from 1 to 10, with lower scores indicating lower quality): The sorted sequence obtained by ranking the evaluation scores from smallest to largest is as follows: Then it will be in the first two ( The initial answer (i.e.) as well as This answer is confirmed as needing revision.

[0101] Therefore, by sorting the scores from smallest to largest and selecting the top... The initial responses are used as those awaiting revision. This system uses quantitative scores to clearly differentiate the quality of the initial responses, ensuring an objective and traceable screening process and avoiding subjective judgment bias. It also allows for precise targeting of the lowest-quality responses. Each answer makes subsequent modifications more targeted, allowing resources to be concentrated on optimizing the weak output links of the model and avoiding ineffective investment. At the same time, the improved high-quality answers supplement the training data, which can help the policy model effectively learn the improvement logic from low quality to high quality, reduce the interference of low-quality samples on training, and ultimately improve the overall quality and stability of the model output.

[0102] In step 204, each of the answers to be modified is input into the initial strategy model, and the optimized answer after modification of each answer to be modified is output based on the modification requirements, so as to obtain a second preset number of optimized answers.

[0103] Among them, the optimized answer is the improved answer generated by the strategy model according to the modification requirements of the answer to be modified. The quality is higher than the original answer to be modified (such as changing the written language beginning to a conversational interactive beginning).

[0104] Specifically, by combining prompts with modification requirements and the answer to be modified into an answer modification template, and inputting it into the initial strategy model, an optimized answer is generated for each answer to be modified. This leverages the strategy model's contextual learning capability, allowing the model to reference the question of the answer to be modified (e.g., loose structure), and combine the task requirements and modification requirements of the prompts to generate a higher-quality optimized answer. Since the number of answers to be modified is a second preset number (…), If there are 10 ... (one), denoted as .

[0105] In some implementations, the evaluation result further includes evaluation opinions, wherein inputting each of the answers to be modified into the initial strategy model, and outputting an optimized answer after modification for each of the answers to be modified based on the modification requirements, to obtain a second preset number of optimized answers, includes:

[0106] (1) Obtain the target evaluation opinion for each of the answers to be modified;

[0107] (2) Based on the prompt words, each of the answers to be modified, and the target evaluation opinion for each of the answers to be modified, construct a first answer modification template;

[0108] The modification requirement for the first answer modification template is to make the initial strategy model modify the corresponding answer to be modified by referring to the target evaluation opinion of each answer to be modified.

[0109] (3) Input the first answer modification template into the initial strategy model, output the optimized answer after modification for each answer to be modified, and obtain a second preset number of optimized answers.

[0110] The evaluation results also include evaluation comments for each initial response, i.e. ; That is, assessment, i.e. The initial response corresponds to an evaluation comment (response_evaluation) for each initial response. The evaluation comment is the basis for assessing and scoring the initial response. The target evaluation comment is the evaluation result generated by the preset evaluation model for each response to be modified.

[0111] For example, if the short video script to be revised is "sweet rice dumplings vs. savory rice dumplings" (2 points), the target evaluation comments are: "The script is entirely written in formal language (such as the flavor of sweet rice dumplings lies in the fusion of sugar and glutinous rice), which does not conform to the application scenario of short videos that are conversational and interactive; it does not mention user pain points (such as the dilemma of choosing between sweet and savory), and cannot resonate with the audience; it deviates from the core of the topic and does not compare the differences between sweet and savory rice dumplings."

[0112] Specifically, after obtaining the target evaluation opinion for each of the answers to be modified, a first answer modification template is constructed based on the prompt words, each answer to be modified, and the target evaluation opinion for each answer to be modified. This allows the initial policy model to modify the corresponding answers to be modified by referring to the target evaluation opinion for each answer to be modified in the first answer modification template, thereby providing guidance for the initial policy model when modifying the answers to be modified. This addresses the technical pain point of low efficiency in optimizing training data in traditional online reinforcement learning (such as PPO). The first answer modification template is then determined as the answer modification template subsequently input into the initial policy model.

[0113] In this way, by obtaining the target evaluation opinions of the answer to be modified, the prompt words, the answer to be modified, and the target evaluation opinions are integrated to construct a first answer modification template, which is then used as the final template. This approach not only provides the policy model with clear guidance on defects and improvement directions through the target evaluation opinions, thus solving the pain point of traditional online reinforcement learning models having no direction in modifying the answer to be modified and blindly trying and failing, but also significantly improving the accuracy of modification; it also anchors the task benchmark through prompt words, ensuring that the modified answer does not deviate from the original requirements; at the same time, the high-quality modified answers generated by the answer modification template can supplement the training data, helping the policy model learn the corresponding logic between the defect types of training data samples and the improvement methods, enhancing the model's ability to actively avoid low-quality outputs and ultimately improving training efficiency and the overall output quality of the policy model.

[0114] In some implementations, the evaluation result further includes evaluation opinions, wherein inputting each of the answers to be modified into the initial strategy model, and outputting an optimized answer after modification for each of the answers to be modified based on the modification requirements, to obtain a second preset number of optimized answers, includes:

[0115] (1) Obtain the target evaluation opinion for each of the answers to be modified;

[0116] (2) Based on the prompt words, each of the answers to be modified, the target evaluation opinions of each of the answers to be modified, and the reference answers of the prompt words, construct a second answer modification template;

[0117] The modification requirement for the second answer modification template is to make the initial strategy model modify the corresponding answer to be modified by referring to the reference answer and the target evaluation opinion of each answer to be modified.

[0118] (3) Input the second answer modification template into the initial strategy model, output the optimized answer after modification for each answer to be modified, and obtain a second preset number of optimized answers.

[0119] In this embodiment of the application, a reference answer is set for each prompt word, and the reference answer is taken as the optimal answer for that prompt word. In some implementations, obtaining the reference answer for the prompt word includes:

[0120] The prompt words are input into the teacher model to obtain the reference answer output by the teacher model.

[0121] The reference response (teacher_response) is a high-quality answer pre-configured for each prompt word, serving as a benchmark for evaluation and modification. The reference response for each prompt word is obtained by acquiring at least one prompt word corresponding to a question and inputting it into an advanced teacher model (such as DeepSeek Full Version) to generate a high-quality reference response.

[0122] In addition, a reference answer can be obtained by manually configuring high-quality answers for the prompt words.

[0123] Therefore, by acquiring the corresponding prompts for the questions and obtaining reference answers through both advanced teacher model generation and manual configuration, this approach achieves efficient and large-scale generation of reference answers using the advanced teacher model, significantly reducing manual costs and meeting the reference needs of a large number of routine questions. Simultaneously, manual configuration provides high-quality reference answers that are precise and context-appropriate for specific and demanding questions, avoiding potential biases in model generation. This combination of methods ensures both the quality and diversity of reference answers and enhances the flexibility and applicability of reference answer acquisition. It lays a reliable foundation for subsequent evaluation template construction based on reference answers, precise selection of answers to be modified, and optimization of strategy model training, ultimately contributing to improved training effectiveness and output quality of the strategy model.

[0124] In this embodiment of the application, the evaluation results also include evaluation opinions for each initial response, i.e. ; That is, assessment, i.e. The initial response corresponds to an evaluation comment (response_evaluation) for each initial response. The evaluation comment is the basis for assessing and scoring the initial response. The target evaluation comment is the evaluation result generated by the preset evaluation model for each response to be modified.

[0125] Specifically, after obtaining the target evaluation opinion for each answer to be modified, a second answer modification template is constructed based on the prompt words, each answer to be modified, the target evaluation opinion for each answer to be modified, and the reference answer. This allows the initial policy model to modify the corresponding answers to be modified by referring to the reference answer in the second answer modification template and the target evaluation opinion for each answer to be modified. This provides guidance for the initial policy model when modifying the answers to be modified, addressing the technical pain point of low efficiency in optimizing training data in traditional online reinforcement learning (such as PPO). The second answer modification template is then used as the answer modification template subsequently input into the initial policy model.

[0126] In this way, by obtaining the target evaluation opinion of the answer to be modified, the second answer modification template is constructed by integrating the prompt words, the answer to be modified, the target evaluation opinion, and the reference answer. This template is then used as the final input answer modification template. It retains the guidance on the defects and improvement directions provided by the target evaluation opinion, while adding the reference answer as a high-quality example. This forms a dual guidance on how to modify (evaluation opinion) and what standard should be achieved after modification (reference answer). This further solves the problem of inadequate modification caused by the model's misunderstanding of evaluation opinions in traditional online reinforcement learning. At the same time, the prompt words anchor the task benchmark, ensuring that the modification does not deviate from the original requirements. After the high-quality optimized answers generated by the template are added to the training data, they can help the policy model learn the transformation logic from low quality to high quality more accurately, strengthen the absorption of high-quality features, and ultimately significantly improve the modification efficiency, optimize the training effect, and enhance the stability and quality rate of the policy model output.

[0127] In some implementations, the step of inputting each of the answers to be modified into the initial strategy model, and outputting an optimized answer after modification for each of the answers to be modified based on the modification requirements, to obtain a second preset number of optimized answers, includes:

[0128] (1) Based on the prompt words, each of the answers to be modified, and the reference answers to the prompt words, construct a third answer modification template;

[0129] The modification requirement for the third answer modification template is to make the initial strategy model modify each of the answers to be modified by referring to the reference answer;

[0130] (2) Input the third answer modification template into the initial strategy model, output the optimized answer after modification for each of the answers to be modified, and obtain a second preset number of optimized answers.

[0131] This can be achieved by directly integrating prompts, each answer to be modified, and a reference answer to construct a third answer modification template. This allows the initial strategy model to modify each answer to be modified by comparing it with the reference answer. Similarly, it provides guidance for the initial strategy model to modify the answers to be modified. Thus, the third answer modification template serves as the subsequent answer modification template input into the initial strategy model.

[0132] For example, the modified template for the third answer is:

[0133] "Based on the original task below and referring to high-quality examples, please optimize the specified text to be modified so that the modified text fully meets the task requirements."

[0134] [Prompt] Please generate a short video script about "Which is better, sweet or savory zongzi?" The script should be conversational and interactive; include user pain points (such as difficulty in choosing); and compare the core differences between sweet and savory zongzi.

[0135] [Text to be revised] Sweet rice dumplings are mainly made of glutinous rice and sugar, and have a sweet taste; savory rice dumplings contain meat or egg yolk, and have a salty taste. Both are traditional foods, but they appeal to different groups of people.

[0136] [Suggested Answer] Do you always struggle with the decision when buying zongzi? You're worried that sweet ones will be too cloying, and savory ones won't suit your taste! Actually, sweet zongzi are made with jujubes and red bean paste, and they have a soft and chewy texture with a hint of sweetness, making them suitable for those who prefer a refreshing taste; savory zongzi contain pork belly and salted egg yolk. Which one do you usually prefer? Please tell me (This answer should be conversational, address a pain point, and provide a comparison).

[0137] [Revision Requirements] Compare the copy to be revised with the high-quality example, learning from the strengths of the example (e.g., conversational expression, presentation of user pain points, and methods of comparison); retain the core discussion object of the copy to be revised (sweet and savory rice dumplings), but must be optimized entirely according to the style and structure of the high-quality example; output the revised copy without additional explanation.

[0138] Therefore, by integrating prompts, answers to be modified, and reference answers to construct a third answer modification template, which is then used as the final input template for answer modification. The reference answer serves as a high-quality example, directly providing specific standards that can be imitated (such as conversational style, pain point presentation, and difference comparison structure). This gives the strategy model a clear and intuitive reference for modifying the answers to be modified, avoiding misunderstandings that may arise from relying solely on abstract evaluation opinions. The prompts anchor the core requirements of the task, ensuring that the modifications do not deviate from the original needs. The optimized answers generated by the third answer modification template can accurately absorb the high-quality features of the reference answers. After being added to the training data, they can help the strategy model efficiently learn the transformation logic from low quality to high quality, significantly improving the accuracy and efficiency of modification, and ultimately enhancing the consistency and quality rate of the strategy model's output.

[0139] In step 205, a group of responses to the prompt words is constructed using a first preset number of initial responses and a second preset number of optimized responses. The initial strategy model is then trained based on the in-group reward value of each target response in the response group to obtain the trained strategy model.

[0140] Among them, the answer group is based on the initial answer ( (one) and optimized answers ( The sample group constructed from (a number of samples) is empirical data for reinforcement learning.

[0141] Specifically, in obtaining After optimizing the answer, you can directly... this One optimized answer and the initial answer Merge to obtain the answer group ,Right now Total One answer. Or... The answer to be modified has been replaced with An optimized answer yields a response group. That is, the total number of answers remains the same. The number is not limited here.

[0142] Therefore, the initial policy model is trained using prompt words and corresponding response groups to obtain the trained policy model.

[0143] In some implementations, training the initial policy model based on the in-group reward value of each target answer in the answer group to obtain the trained policy model includes:

[0144] (1) Calculate the intra-group reward value for each target answer in the answer group;

[0145] (2) Determine the relative advantage value of each target answer in the answer group by using the in-group reward value of each target answer;

[0146] (3) Calculate the objective function of the reinforcement learning algorithm based on the relative advantage value and KL divergence of each target answer;

[0147] (4) Run the gradient ascent algorithm that maximizes the objective function to update the parameters of the initial policy model and obtain the trained policy model.

[0148] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the model training framework provided in the embodiments of this application. Figure 3 The reinforcement learning training process based on group optimization combined with evaluation feedback aims to improve the ability of a policy model (such as a large language model) to generate high-quality content through a closed loop of generation, evaluation, modification, reward calculation, advantage analysis, and parameter updates. This will be explained below:

[0149] Generate initial response: The strategy model is the large language model to be fine-tuned (such as the topic selection copy generation model), which generates prompt words. The initial answer This is the core manifestation of the optimization of the group in this application, and the generated... An initial response can be used to build a set of samples instead of a single sample to train the model, which can reduce the noise interference of a single sample and make subsequent evaluations more reliable.

[0150] The pre-defined evaluation model outputs an evaluation score for each initial answer. ) and evaluation opinions ( ).filter An initial answer with a low evaluation score was optimized by combining evaluation comments and reference answers to obtain... An optimized answer The answer will be optimized. (number) and initial answer ( (one) and merged to form a group containing A group of answers targeting a specific target, or the initial answer ( (in) The answer that needs to be modified will be replaced with an optimized answer. (individual), forming A group of answers targeting a specific goal.

[0151] The reward model assigns a quality score to each target answer and calculates the within-group reward value. (e.g., assessing whether the text "conforms to the logic of short video introductory presentations"); then, through group calculations, obtain the relative advantage value of each target response. This measure, or relative quality, is used to evaluate the relative performance of each target response within the group. This is the core of the strategy model relative optimization in this application's embodiments: it does not pursue absolutely high rewards, but rather seeks to be more advantageous within the group, allowing the model to focus on directions that are better than other outputs in the same group.

[0152] Based on relative advantage value ( The objective function is constructed using policy distribution constraints (such as chi-square-Leibler divergence, KL divergence). The objective function is maximized using the gradient ascent algorithm to update the policy model parameters. When updating the policy model parameters, a reference model is introduced, typically a baseline model with the same architecture as the policy model but with its parameters frozen (such as a policy model trained using SFT). In this embodiment, the relative advantage value (...) is used... The policy model uses gradient ascent and KL divergence to update parameters, ensuring that the responses generated in the next training iteration are more inclined towards samples with high advantage and low KL divergence. KL divergence measures the difference between the current output distribution of the policy model and the output distribution of the reference model, constraining the magnitude of policy updates and preventing the model from deviating significantly from its original capabilities in pursuit of higher rewards. The updated policy model generates responses again, and the above process is repeated for continuous optimization until the model training converges, resulting in the trained policy model.

[0153] Answering groups Each result in The corresponding in-group reward value can be calculated using a preset evaluation model, reward function, or reward model. , can get ;or The reward function is set according to the specific task and can use reward rules or reward models; no restrictions are imposed here.

[0154] The reward model is a pre-trained model specifically designed for predicting within-group reward values. Since using a new reward model for reward value calculation involves introducing an additional model, significantly increasing the computational load on the computer, a pre-defined evaluation model can be used for within-group reward value calculation, thereby reducing the computational burden and thus the computational resource consumption. The reward function can be either the pre-defined evaluation model or a function within the reward model used for reward value calculation, i.e., reward_function().

[0155] Specifically, for each target answer within the answer group, the reward value is determined based on the group's reward value. Perform calculations within the answer group to obtain each answer in the answer group. Corresponding relative advantage value The specific calculation method for the relative advantage value can be found in the following formula:

[0156] ;

[0157] in, This is the average reward value within the group corresponding to all target answers. This is the standard deviation of the within-group reward values ​​for all target responses in the response group. The relative advantage value represents the quality of each target response relative to the mean in the response group; in the above steps, the lower-scoring responses are considered more favorable. The initial responses were modified, which improved the overall level of responses from the response group. This allowed the fine-tuning of the policy model to achieve better results, thus improving training effectiveness and efficiency.

[0158] After obtaining the relative advantage value of each target answer in the answer group, the objective function of the reinforcement learning algorithm is calculated. The model parameters of the initial policy model are updated by maximizing the objective function to obtain the trained policy model.

[0159] Specifically, the reinforcement learning algorithm in this application can adopt the objective function of GRPO. Essentially, it maximizes the relative advantage of the group by combining token-level policy optimization with KL divergence constraints, under the sampling expectations of the prompt words and group outputs. The KL divergence constraint is a stable constraint term of GRPO, preventing policy models from being manipulated during training. Excessive deviation from the reference model.

[0160] Therefore, by first calculating the intra-group reward value and relative advantage value of the target answer in the answer group, group optimization is used instead of single-sample optimization, effectively reducing single-sample noise interference. At the token level, by combining the generation probability ratio of the old and new strategies, the pruning function (to avoid excessive update amplitude), and the relative advantage value, high-quality token generation can be encouraged and low-quality tokens can be suppressed. At the same time, the KL divergence constraint term based on the reference model prevents the strategy model during training from deviating too much from the basic capabilities of the reference model (such as copywriting style and logical coherence). Furthermore, sampling based on the distribution of prompt words ensures that training covers diverse input scenarios and improves optimization robustness. In addition, the modification of low-quality initial answers improves the overall level of the answer group. Ultimately, the training effect and efficiency of the strategy model are improved, while also ensuring the stability of model updates and the preservation of initial basic capabilities.

[0161] In some implementations, after training the initial policy model using the prompt words and corresponding answer groups to obtain the trained policy model, the method further includes:

[0162] Obtain the prompt words corresponding to the input text, and input the prompt words into the trained policy model to obtain the answer corresponding to the input text.

[0163] After obtaining the trained policy model, inference can be applied using the trained policy model. When a prompt word corresponding to the input text is obtained, it is input into the trained policy model to obtain the answer corresponding to the input text.

[0164] As described above, this embodiment of the application obtains the prompt words corresponding to the input text and inputs the prompt words into an initial strategy model to obtain a first preset number of initial answers output by the initial strategy model; inputs the prompt words and each initial answer into a preset evaluation model to obtain an evaluation result for each initial answer; based on the evaluation results, selects a second preset number of answers to be modified from the initial answers; inputs each answer to be modified into the initial strategy model, and outputs an optimized answer after modification for each answer to be modified based on the modification requirements to obtain a second preset number of optimized answers; constructs an answer group for the prompt words using the first preset number of initial answers and the second preset number of optimized answers, and trains the initial strategy model based on the in-group reward value of each target answer in the answer group to obtain a trained strategy model. Therefore, after the initial strategy model outputs a first preset number of initial answers based on the prompt words, the prompt words and each initial answer are input into a preset evaluation model. The preset evaluation model evaluates each initial answer and, based on the evaluation results, selects a second preset number of answers to be modified from the initial answers. Each answer to be modified is then input into the initial strategy model, which outputs an optimized answer after modification based on the modification requirements, resulting in high-quality optimized answers. This leads to an answer group constructed from the initial answers and optimized answers. The initial strategy model is then trained based on the in-group reward value of each target answer in the answer group. Compared to related technologies where the self-exploration method during the enhanced training phase leads to insufficient exploration, deviation from the correct direction, lower model prediction accuracy, and higher computer resource consumption, this application helps the model avoid ineffective exploration and anchor to the correct direction, shortening the exploration path during self-exploration. This reduces the consumption of computer processing resources, improves processing power, and avoids the problem of the strategy model obtaining high rewards by circumventing the core requirements of the task, thus improving the inference accuracy of the strategy model.

[0165] In this embodiment, taking the strategy model as the copywriting generation model as an example, the complete process of this application is explained:

[0166] For at least one copywriting requirement (such as designing promotional copy for a new sports shoe), determine the key words (such as product features: lightweight, non-slip, suitable for marathons; target audience: long-distance running enthusiasts), and obtain the corresponding reference answers for the key words (such as high-quality copy written by humans: Light as a feather, non-slip protection, accompany you to conquer every kilometer, professional marathon running shoes, go all out for your passion).

[0167] Input the prompt words into the pre-trained copy generation model (strategy model) to obtain the first preset number (e.g., 10) of initial copy (initial response) output by the model.

[0168] Based on the prompts and 10 initial text inputs, a preset evaluation model is used to obtain the evaluation results (such as evaluation scores and evaluation comments) for each initial text.

[0169] Screening copy to be modified: Based on the evaluation results, select a second preset number (e.g., 3, less than 10) of poor-quality copy to be modified from the 10 initial copy samples.

[0170] Input the three texts to be modified into the text generation model, and output the optimized text for each text based on the modification requirements.

[0171] Training the model: The initial 10 texts and the optimized 3 texts are merged into a group of 13 responses for the prompt word, or the optimized 3 texts replace the 3 texts to be modified in the initial 10 texts to obtain a group of 10 responses for the prompt word. The text generation model is trained using reinforcement learning based on the in-group reward value of each target response in the response group to obtain the optimized text generation model.

[0172] The specific method for selecting text to be modified is as follows:

[0173] Sort the 10 initial copywritings in ascending order of their evaluation scores (e.g., score sorting: 3 points, 4 points, 5 points, ..., 9 points).

[0174] Select the three lowest-scoring initial copy (e.g., 3, 4, and 5 points) from the sorted list and identify them as the copy to be modified.

[0175] A response modification template can be created by combining prompts with modification requirements and the response to be modified. This template can then guide the initial strategy model to generate an optimized response. Methods for constructing response modification templates include:

[0176] Method 1: Revisions based on evaluation feedback:

[0177] The evaluation results include evaluation comments (such as the copy not highlighting the lightweight feature, or the language being stiff). When creating the template:

[0178] Obtain target evaluation feedback for each piece of text to be revised (i.e., specific improvement suggestions for the text).

[0179] Create a template for revising the first response. Example content: Hints: [Product features combined with target audience]; Text to be revised: [3-point copy]; Evaluation comments: [Failed to highlight lightweight features, language is stiff]. Please refer to the evaluation comments to optimize your copy.

[0180] Based on this template, the model makes targeted modifications to the copy based on evaluation feedback (such as adding lightweight descriptions and optimizing language fluency).

[0181] Method 2: Modify based on the reference answers and evaluation comments:

[0182] Building upon Method 1, a second answer modification template is further introduced, using reference answers. Example content: "Hint: [Product features combined with target audience]; Reference copy: [High-quality reference copy]; Copy to be modified: [3-point copy]; Evaluation comments: [Failed to highlight lightweight features, language is stiff]. Please refer to the reference copy and evaluation comments to optimize your copy."

[0183] The model also draws on the style and structure of the reference text, as well as the specific suggestions from the evaluation opinions, to improve the accuracy of the modifications.

[0184] Method 3: Modifications based solely on the reference answer:

[0185] The third answer template is built directly based on the reference answer. Example content: "Hint: [Product features combined with target audience]; Reference copy: [High-quality reference copy]; Copy to be revised: [3-point copy]. Please refer to the reference copy to optimize the copy to be revised."

[0186] The model modifies low-quality copy by imitating the high-quality features of the reference copy (such as keywords and sentence structure).

[0187] How to obtain the reference answer:

[0188] If you cannot directly obtain a manually written reference answer, you can generate one using the following methods:

[0189] Get prompts (such as designing promotional copy for new sneakers);

[0190] Input the prompts into a high-quality teacher model (such as DeepSeek Full Version) to obtain the model's output reference answer (such as high-quality copywriting that meets industry standards).

[0191] The specific process of model training is as follows:

[0192] The steps for training the copy generation model by answering a set of 13 text entries are as follows:

[0193] Calculate the intra-group reward value: Use a reward function (such as a comprehensive scoring function based on copy relevance, attractiveness, and conversion rate) to calculate the intra-group reward value (such as 7 points, 8 points, ..., 9.5 points) for each target answer in the answer group.

[0194] The relative advantage value of each piece of copy is determined based on its in-group reward value. The formula is: Relative Advantage Value = (In-group reward value of a single piece of copy - Mean of in-group reward values ​​in the group) / Standard deviation of in-group reward values ​​in the group. For example, if the in-group reward value of a piece of copy is higher than the mean, the relative advantage value is positive, indicating that its quality is better than the average level of the group.

[0195] The objective function of the reinforcement learning algorithm comprehensively considers the generation probability, relative advantage value and KL divergence constraint of each token (such as words like "light", "light", "protect") in each piece of copy (to prevent the model from deviating from the basic copy generation ability).

[0196] By maximizing the objective function and adjusting the parameters of the copy generation model, the model is made more inclined to generate tokens and copy with high relative advantage values. As a result, the trained copy generation model can generate high-quality copy in terms of reasoning accuracy.

[0197] Through the above approach, the copywriting generation model can continuously improve the quality and applicability of generated copywriting by specifically optimizing low-quality outputs and training with group sample features.

[0198] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of the strategy model training device provided in the embodiment of this application. The strategy model training device is applied to a server and may include a first input unit 501, a second input unit 502, a filtering unit 503, a third input unit 504, and a construction unit 505, etc.

[0199] The first input unit 501 is used to obtain the prompt words corresponding to the input text and input the prompt words into the initial strategy model to obtain the first preset number of initial answers output by the initial strategy model.

[0200] The second input unit 502 is used to input the prompt words and each initial answer into a preset evaluation model to obtain the evaluation result of each initial answer;

[0201] The filtering unit 503 is used to filter out a second preset number of answers to be modified from the initial answers based on the evaluation results.

[0202] The third input unit 504 is used to input each of the answers to be modified into the initial strategy model, and output the optimized answer after modification of each of the answers to be modified based on the modification requirements, so as to obtain a second preset number of optimized answers.

[0203] The construction unit 505 is used to construct a group of answers for the prompt words with the first preset number of initial answers and the second preset number of optimized answers, and to train the initial strategy model based on the in-group reward value of each target answer in the group of answers to obtain the trained strategy model.

[0204] In some embodiments, the evaluation result includes at least an evaluation score, and the screening unit 503 includes:

[0205] The sorting subunit is used to sort each of the initial answers in ascending order of evaluation scores to obtain a sorted sequence;

[0206] The first determining subunit is used to determine the initial answers that are in the first second preset number in the sorting sequence as answers to be modified, thereby obtaining the second preset number of answers to be modified.

[0207] In some implementations, the evaluation result further includes evaluation opinions, and the third input unit 504 is used for:

[0208] Obtain the target evaluation opinion for each of the aforementioned answers to be modified;

[0209] Based on the prompt words, each of the answers to be modified, and the target evaluation opinions for each of the answers to be modified, a first answer modification template is constructed;

[0210] The modification requirement for the first answer modification template is to make the initial strategy model modify the corresponding answer to be modified by referring to the target evaluation opinion of each answer to be modified.

[0211] The first answer modification template is input into the initial strategy model, and the optimized answer after modification for each answer to be modified is output to obtain a second preset number of optimized answers.

[0212] In some embodiments, the evaluation result further includes evaluation opinions, and the third input unit 504 is further used for:

[0213] Obtain the target evaluation opinion for each of the aforementioned answers to be modified;

[0214] Based on the prompt words, each of the answers to be modified, the target evaluation opinion for each of the answers to be modified, and the reference answers for the prompt words, a second answer modification template is constructed;

[0215] The modification requirement for the second answer modification template is to make the initial strategy model modify the corresponding answer to be modified by referring to the reference answer and the target evaluation opinion of each answer to be modified.

[0216] The second answer modification template is input into the initial strategy model, and the optimized answer after modification for each answer to be modified is output to obtain a second preset number of optimized answers.

[0217] In some embodiments, the third input unit 504 is further configured to:

[0218] Based on the prompt words, each of the answers to be modified, and the reference answers for the prompt words, a third answer modification template is constructed;

[0219] The modification requirement for the third answer modification template is to make the initial strategy model modify each of the answers to be modified by referring to the reference answer;

[0220] The third answer modification template is input into the initial strategy model, and the optimized answer after modification for each answer to be modified is output to obtain a second preset number of optimized answers.

[0221] In some embodiments, the third input unit 504 is further configured to:

[0222] The prompt words are input into the teacher model to obtain the reference answer output by the teacher model.

[0223] In some embodiments, the building unit 505 is used for:

[0224] Calculate the in-group reward value for each target answer in the answer group;

[0225] The relative advantage value of each target answer in the answer group is determined by the in-group reward value of each target answer;

[0226] The objective function of the reinforcement learning algorithm is calculated based on the relative advantage value and KL divergence of each target answer;

[0227] The gradient ascent algorithm that maximizes the objective function is run to update the parameters of the initial policy model, resulting in the trained policy model.

[0228] The specific implementation of each of the above units can be found in the previous embodiments, and will not be repeated here.

[0229] As described above, in this embodiment, the first input unit 501 obtains the prompt words corresponding to the input text and inputs the prompt words into the initial strategy model to obtain a first preset number of initial answers output by the initial strategy model; the second input unit 502 inputs the prompt words and each initial answer into a preset evaluation model to obtain an evaluation result for each initial answer; the filtering unit 503 filters out a second preset number of answers to be modified from the initial answers based on the evaluation results; the third input unit 504 inputs each answer to be modified into the initial strategy model and outputs an optimized answer after modification for each answer to be modified based on the modification requirements to obtain a second preset number of optimized answers; the construction unit 505 constructs an answer group of the prompt words using the first preset number of initial answers and the second preset number of optimized answers, and trains the initial strategy model based on the intra-group reward value of each target answer in the answer group to obtain a trained strategy model.

[0230] Therefore, after the initial strategy model outputs a first preset number of initial answers based on the prompt words, the prompt words and each initial answer are input into a preset evaluation model. The preset evaluation model evaluates each initial answer and, based on the evaluation results, selects a second preset number of answers to be modified from the initial answers. Each answer to be modified is then input into the initial strategy model, which outputs an optimized answer after modification based on the modification requirements, resulting in high-quality optimized answers. This leads to an answer group constructed from the initial answers and optimized answers. The initial strategy model is then trained based on the in-group reward value of each target answer in the answer group. Compared to related technologies where the self-exploration method during the enhanced training phase leads to insufficient exploration, deviation from the correct direction, lower model prediction accuracy, and higher computer resource consumption, this application helps the model avoid ineffective exploration and anchor to the correct direction, shortening the exploration path during self-exploration. This reduces the consumption of computer processing resources, improves processing power, and avoids the problem of the strategy model obtaining high rewards by circumventing the core requirements of the task, thus improving the inference accuracy of the strategy model.

[0231] The specific implementation of each of the above units can be found in the previous embodiments, and will not be repeated here.

[0232] Figure 5 This is a partial structural block diagram of a computer device 1000 implementing an embodiment of this application. The computer device 1000 can vary significantly due to different configurations or performance characteristics, and may include one or more central processing units (CPUs) 422 (e.g., one or more processors) and a memory 432, and one or more storage media 430 (e.g., one or more mass storage devices) storing application programs 442 or data 444. The memory 432 and storage media 430 may be temporary or persistent storage. The program stored in the storage media 430 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 500. Furthermore, the CPU 422 may be configured to communicate with the storage media 430 and execute the series of instruction operations in the storage media 430 on the server 500.

[0233] Computer device 1000 may also include one or more power supplies 425, one or more wired or wireless network interfaces 450, one or more input / output interfaces 458, and / or one or more operating systems 441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0234] The central processing unit 422 in the computer device 1000 can be used to execute the strategy model training method of this application embodiment: acquiring prompt words corresponding to input text and inputting the prompt words into an initial strategy model to obtain a first preset number of initial answers output by the initial strategy model; inputting the prompt words and each of the initial answers into a preset evaluation model to obtain an evaluation result for each of the initial answers; selecting a second preset number of answers to be modified from the initial answers based on the evaluation results; inputting each of the answers to be modified into the initial strategy model, and outputting an optimized answer after modification for each of the answers to be modified based on modification requirements to obtain a second preset number of optimized answers; constructing an answer group for the prompt words with the first preset number of initial answers and the second preset number of optimized answers, and training the initial strategy model based on the in-group reward value of each target answer in the answer group to obtain a trained strategy model.

[0235] This application also provides a computer-readable storage medium for storing program code for executing the strategy model training methods of the foregoing embodiments: obtaining prompt words corresponding to input text and inputting the prompt words into an initial strategy model to obtain a first preset number of initial answers output by the initial strategy model; inputting the prompt words and each initial answer into a preset evaluation model to obtain an evaluation result for each initial answer; selecting a second preset number of answers to be modified from the initial answers based on the evaluation results; inputting each answer to be modified into the initial strategy model and outputting an optimized answer after modification for each answer to be modified based on modification requirements to obtain a second preset number of optimized answers; constructing an answer group for the prompt words using the first preset number of initial answers and the second preset number of optimized answers, and training the initial strategy model based on the in-group reward value of each target answer in the answer group to obtain a trained strategy model.

[0236] This application also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the strategy model training method described above.

[0237] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0238] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0239] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0240] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0241] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0242] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0243] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0244] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0245] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0246] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for training a policy model, characterized in that, include: Obtain the prompt words corresponding to the input text and input the prompt words into the initial strategy model to obtain the first preset number of initial answers output by the initial strategy model; The prompt words and each initial answer are input into a preset evaluation model to obtain an evaluation result for each initial answer; Based on the evaluation results, a second preset number of answers to be modified are selected from the initial answers; Each of the answers to be modified is input into the initial strategy model, and the optimized answer after modification of each answer to be modified is output based on the modification requirements, so as to obtain a second preset number of optimized answers; Construct a response group for the prompt word using the first preset number of initial responses and the second preset number of optimized responses, and calculate the intra-group reward value for each target response in the response group; The relative advantage value of each target answer in the answer group is determined by the in-group reward value of each target answer; The objective function of the reinforcement learning algorithm is calculated based on the relative advantage value and KL divergence of each target answer; The gradient ascent algorithm that maximizes the objective function is run to update the parameters of the initial policy model, resulting in the trained policy model.

2. The strategy model training method according to claim 1, characterized in that, The evaluation result includes at least an evaluation score, and the step of selecting a second preset number of answers to be modified from the initial answers based on the evaluation result includes: The initial answers are sorted in ascending order of their evaluation scores to obtain a sorted sequence. The initial answers that are in the first second preset number in the sorted sequence are identified as answers to be modified, thus obtaining the second preset number of answers to be modified.

3. The strategy model training method according to claim 2, characterized in that, The evaluation results also include evaluation opinions. The process involves inputting each of the answers to be modified into the initial strategy model, outputting optimized answers after modification for each answer to be modified based on the modification requirements, and obtaining a second preset number of optimized answers, including: Obtain the target evaluation opinion for each of the answers to be modified; Based on the prompt words, each of the answers to be modified, and the target evaluation opinions for each of the answers to be modified, a first answer modification template is constructed; The modification requirement for the first answer modification template is to make the initial strategy model modify the corresponding answer to be modified by referring to the target evaluation opinion of each answer to be modified. The first answer modification template is input into the initial strategy model, and the optimized answer after modification for each answer to be modified is output to obtain a second preset number of optimized answers.

4. The strategy model training method according to claim 2, characterized in that, The evaluation results also include evaluation opinions. The process involves inputting each of the answers to be modified into the initial strategy model, outputting optimized answers after modification for each answer to be modified based on the modification requirements, and obtaining a second preset number of optimized answers, including: Obtain the target evaluation opinion for each of the answers to be modified; Based on the prompt words, each of the answers to be modified, the target evaluation opinion for each of the answers to be modified, and the reference answers for the prompt words, a second answer modification template is constructed; The modification requirement for the second answer modification template is to make the initial strategy model modify the corresponding answer to be modified by referring to the reference answer and the target evaluation opinion of each answer to be modified. The second answer modification template is input into the initial strategy model, and the optimized answer after modification for each answer to be modified is output to obtain a second preset number of optimized answers.

5. The strategy model training method according to claim 1, characterized in that, The process involves inputting each of the answers to be modified into the initial strategy model, and outputting an optimized answer after modification for each answer to be modified based on the modification requirements, to obtain a second preset number of optimized answers, including: Based on the prompt words, each of the answers to be modified, and the reference answers for the prompt words, a third answer modification template is constructed; The modification requirement for the third answer modification template is to make the initial strategy model modify each of the answers to be modified by referring to the reference answer; The third answer modification template is input into the initial strategy model, and the optimized answer after modification for each answer to be modified is output to obtain a second preset number of optimized answers.

6. The strategy model training method according to claim 1, characterized in that, Obtaining the reference answer for the prompt word includes: The prompt words are input into the teacher model to obtain the reference answer output by the teacher model.

7. A strategy model training device, characterized in that, include: The first input unit is used to obtain the prompt words corresponding to the input text and input the prompt words into the initial strategy model to obtain a first preset number of initial answers output by the initial strategy model. The second input unit is used to input the prompt words and each initial answer into a preset evaluation model to obtain the evaluation result of each initial answer; A filtering unit is used to filter out a second preset number of answers to be modified from the initial answers based on the evaluation results; The third input unit is used to input each of the answers to be modified into the initial strategy model, and output the optimized answer after modification of each of the answers to be modified based on the modification requirements, so as to obtain a second preset number of optimized answers. A construction unit is configured to construct a group of responses to the prompt words using the first preset number of initial responses and the second preset number of optimized responses, and to calculate the intra-group reward value for each target response in the group of responses. The relative advantage value of each target answer in the answer group is determined by the in-group reward value of each target answer; The objective function of the reinforcement learning algorithm is calculated based on the relative advantage value and KL divergence of each target answer; The gradient ascent algorithm that maximizes the objective function is run to update the parameters of the initial policy model, resulting in the trained policy model.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the policy model training method according to any one of claims 1 to 6.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the policy model training method according to any one of claims 1 to 6.

Images