Strategy model training method and device, strategy generation method and device, storage medium and terminal

Abstract

The embodiment of the invention discloses a strategy model training method and device, a strategy generation method and device, a storage medium and a terminal. Firstly, a set of sample answers are generated for a target question through a preset large language model to serve as references. And during training, inputting an answer generated by the current strategy model and a sample answer into the evaluation model for comparison, and outputting a good and bad sorting result. The ranking is quantized as a reward value whose size is positively correlated with the degree to which the current answer is superior to the sample answer. And finally, the system adjusts strategy model parameters according to the reward value and guides the strategy model parameters to be continuously optimized. According to the method, scoring according to standard answers is replaced with relative sorting, and the problem that open questions lack clear judgment standards is solved. The method has the beneficial effects that the training data cost and labeling dependence are remarkably reduced, so that the model can realize stable and autonomous efficiency improvement in the vertical field, and a self-driven benign evolution cycle is formed.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and terminal for strategy model training and strategy generation. Background Technology

[0002] In traditional business operations, complex decisions are primarily driven by experts over long periods. However, with advancements in artificial intelligence, large language models (LRMs) trained using reinforcement learning and verifiable reasoning (RLVR) are gradually achieving autonomous operation. Currently, the training of large language models typically relies on tasks with explicit answers, such as mathematics and coding. Since much operational knowledge is implicit and difficult to translate into verifiable, uniquely standardized data, this makes it challenging to align such methods with real-world optimization goals in specific business sectors. This not only limits their practical application effectiveness but also increases decision-making risks. Summary of the Invention

[0003] This application provides a method, apparatus, storage medium, and terminal for training a strategy model and generating a strategy, which can solve the technical problem that existing model training paradigms in related technologies are difficult to apply to open strategy scenarios.

[0004] In a first aspect, embodiments of this application provide a strategy model training method, the method comprising: Extract at least one target question from the vertical open question answering dataset, input the target question into a preset large language model, and control the preset large language model to generate multiple sample answers corresponding to the target question offline; During the training of the current policy model, the target question is input into the current policy model, and the current policy model is controlled to generate multiple current answers corresponding to the target question online. Construct a pre-trained evaluation model, input each current answer and the above sample answers into the evaluation model, and use the evaluation model to compare each current answer with the above sample answers, and output the ranking results of each current answer relative to the above sample answers; The reward value for each current answer is calculated by ranking the current answers. The magnitude of the reward value is positively correlated with the degree to which each current answer is superior to each of the sample answers. Based on the above reward value, construct the policy gradient estimate, update the parameters of the current policy model, and train through multiple rounds of iteration until the performance of the current policy model on the pre-set validation set reaches the preset performance threshold.

[0005] In one possible implementation, the above-mentioned input of each current answer and the above-mentioned multiple sample answers into the above-mentioned evaluation model, and the comparison of each current answer with the above-mentioned multiple sample answers by the above-mentioned evaluation model, includes: combining each current answer with each sample answer in pairs to form multiple answer pairs; inputting each answer pair into the above-mentioned evaluation model, and the above-mentioned evaluation model outputting the superiority or inferiority judgment result of each answer pair.

[0006] In one possible implementation, the ranking result is determined based on the number of times the current answer wins among all the ranking results.

[0007] In one possible implementation, the above-mentioned input of each current answer and the above-mentioned multiple sample answers into the above-mentioned evaluation model, and the comparison of each current answer with the above-mentioned multiple sample answers by the above-mentioned evaluation model, includes: forming each current answer and all the above-mentioned sample answers into an answer set corresponding to each current answer, and inputting it into the above-mentioned evaluation model; the above-mentioned evaluation model performs an overall evaluation of all answers in each answer set, and outputs a ranking list of the superiority and inferiority of each answer set.

[0008] In one possible implementation, the ranking result is determined based on the current answer's position in the ranking list.

[0009] In one possible implementation, the above-mentioned calculation of the reward value corresponding to each current answer based on the ranking of the current answers includes: obtaining the reward value corresponding to each current answer after processing by a preset smoothing function based on the ranking of the current answers.

[0010] In one possible implementation, the process of adjusting the parameters of the current policy model based on all reward values ​​is achieved through a reinforcement learning algorithm based on group-relative policy optimization.

[0011] In one possible implementation, the above-mentioned construction of policy gradient estimation based on the reward value and updating of the parameters of the current policy model includes: smoothing and normalizing all reward values, constructing a policy gradient estimate based on the processed reward values, calculating the gradient direction in the parameter space of the current policy model that increases the probability of generating answers with high reward values, and fine-tuning the parameters of the current policy model based on the gradient direction.

[0012] In one possible implementation, the evaluation model is a reward model trained based on contrastive learning, and the evaluation model is configured to judge the merits of a pair of input answers.

[0013] In one possible implementation, the evaluation model is a vertical domain inference model trained based on the domain knowledge of the scenario to which the target problem belongs.

[0014] In one possible implementation, the aforementioned preset large language model is a general large language model; or, the aforementioned preset large language model is a large language model that has been fine-tuned with knowledge of the scenario domain to which the aforementioned target problem belongs.

[0015] In one possible implementation, the aforementioned objective problem is an open-ended decision problem.

[0016] Secondly, embodiments of this application provide a strategy generation method, the method comprising: In response to a policy generation request triggered by a user in the interactive interface, the policy generation request shall at least include a policy question in the form of a natural language description. Input the above strategy problem into the current strategy model, and control the strategy answer output by the current strategy model for the above strategy problem; Generate a natural language description of the strategy answer to the strategy question based on the above strategy answer, and display the above strategy answer text on the above interactive interface; The current strategy model mentioned above refers to the current strategy model in any of the above embodiments.

[0017] Thirdly, embodiments of this application provide a strategy model training apparatus, the apparatus comprising: The sample acquisition module is used to extract at least one target question from the vertical open question answering dataset, input the target question into the preset large language model, and control the preset large language model to generate multiple sample answers corresponding to the target question offline. The online sampling module is used to input the target question into the current policy model during the training process of the current policy model, and control the current policy model to generate multiple current answers corresponding to the target question online. The answer ranking module is used to build a pre-trained evaluation model. Each current answer and the above sample answers are input into the evaluation model. The evaluation model compares each current answer with the above sample answers and outputs the ranking results of each current answer relative to the above sample answers. The reward calculation module is used to calculate the reward value corresponding to each current answer based on the ranking of the current answers. The magnitude of the reward value is positively correlated with the degree to which each current answer is better than each of the sample answers in the above sample answers. The model parameter tuning module is used to construct policy gradient estimates based on the above reward values, update the parameters of the above current policy model, and train through multiple rounds of iterations until the performance of the above current policy model on the pre-set validation set reaches the preset performance threshold.

[0018] Fourthly, embodiments of this application provide a strategy generation apparatus, the apparatus comprising: The request receiving module is used to respond to a strategy generation request triggered by the user in the interactive interface, wherein the strategy generation request contains at least a strategy question in the form of a natural language description. The strategy generation module is used to input the above strategy problem into the current strategy model and control the strategy answer output by the current strategy model for the above strategy problem; The interactive response module is used to generate a strategy response text in natural language form corresponding to the strategy question based on the above strategy answer, and to display the above strategy response text on the above interactive interface. The current strategy model mentioned above refers to the current strategy model in any of the above embodiments.

[0019] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to perform the steps of the method described above.

[0020] Sixthly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method described above.

[0021] In a seventh aspect, embodiments of this application provide a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is adapted to be loaded by the processor and to execute the steps of the method described above.

[0022] The beneficial effects of the technical solutions provided in some embodiments of this application include at least the following: This application provides a strategy model training method. It extracts at least one target question from a vertical open question-answering dataset, inputs the target question into a preset large language model, and controls the preset large language model to generate multiple sample answers corresponding to the target question offline. During the training of the current strategy model, the target question is input into the current strategy model, and the current strategy model generates multiple current answers corresponding to the target question online. A pre-trained evaluation model is constructed, and each current answer and multiple sample answers are input into the evaluation model. The evaluation model compares each current answer with the multiple sample answers and outputs a ranking result of the current answer relative to the multiple sample answers. A reward value is calculated for each current answer based on the ranking result, where the reward value is positively correlated with the degree to which each current answer is superior to each of the multiple sample answers. A strategy gradient estimate is constructed based on the reward value, and the parameters of the current strategy model are updated. Multiple rounds of iterative training are performed until the performance of the current strategy model on a preset validation set reaches a preset performance threshold. In the sample answer acquisition phase, a pre-defined large language model generates a diverse set of answers for the target question, serving as a benchmark for subsequent training. This significantly reduces the reliance on training data, making the method applicable to scenarios where data is scarce and there is no single standard answer. During training, the system retrieves a set of current answers generated for the same target question from the current policy model, which is currently in optimization. By dynamically sampling the real-time performance of the policy model during training iterations, changes in its policy space are captured. Using multiple answers for model training provides more robust data support for policy evaluation, effectively reducing random fluctuations during training and making the adjustment of model parameters more accurate and reliable, thereby accelerating the convergence of the training process. Next, the evaluation model compares each current answer with all sample answers and outputs a ranking result. The core of this approach is to transform the complex and subjective task of answer quality evaluation from a scoring mechanism based on a standard answer to a relative ranking mechanism. The evaluation model only needs to determine which of two or more given answers is better. This relative comparison reduces the difficulty and ambiguity of the evaluation task itself, allowing the model to bypass the fundamental challenge of defining precise quantitative standards for such problems and achieve automated evaluation of open-ended and subjective questions. The step of determining the reward value based on the ranking results transforms the relative ranking information into a continuous reward value that can be directly used by the reinforcement learning algorithm. This reward value is positively correlated with the degree to which the current answer is better than the sample answers. This reward mechanism provides the policy model with a clear and continuous optimization direction. The model is explicitly guided towards the goal of generating better answers. This reward design ensures the achievability of the training objective and the stability of the training process, effectively motivating the model to conduct beneficial explorations.Finally, based on the reward values ​​corresponding to all current answers, the parameters of the current policy model are adjusted through an optimization algorithm, and the above training process is repeated until the model converges. This step realizes the automated and targeted evolution of the policy model's capabilities, enabling the policy model to achieve stable and autonomous performance improvement under the guidance of reward signals, forming a self-driven virtuous evolutionary cycle, thereby better solving problems in open policy scenarios. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 An exemplary system architecture diagram of a strategy model training method provided in this application embodiment; Figure 2 A flowchart illustrating a strategy model training method provided in an embodiment of this application; Figure 3 A logical diagram illustrating the reward value calculation mechanism in a strategy model training method provided in this application embodiment; Figure 4 A flowchart illustrating a strategy model training method provided in an embodiment of this application; Figure 5 A flowchart illustrating a strategy generation method provided in an embodiment of this application; Figure 6 A structural block diagram of a strategy model training device provided in an embodiment of this application; Figure 7 A structural block diagram of a strategy generation device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0025] To make the features and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0026] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims. Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the association relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, in the description of the embodiments of this application, "multiple" refers to two or more.

[0027] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0028] In traditional business operations, companies typically rely on experienced experts to formulate reasonable decisions and solutions for complex operational problems. With the rapid development of artificial intelligence, especially the emergence of Large Language Models (LRMs) trained using reinforcement learning and verifiable reasoning (RLVR), machines are gradually acquiring the ability to make autonomous decisions, thus replacing human decision-making to some extent. Currently, LRM training in general scenarios generally relies on verifiable tasks with clear answers, such as mathematical calculations, programming, and scientific reasoning, to improve the model's reasoning and generalization abilities. These tasks, due to their clear right-or-wrong boundaries and verifiability, provide stable and reliable reward signals for model optimization. However, when these methods are applied to the vertical domain of business operations, significant challenges arise: because the amount of data that can be transformed into verifiable forms (such as deterministic calculations and option judgments) in this vertical domain is relatively limited, much operational knowledge is inherently implicit, experience-driven, and not uniquely accurate, making it difficult to effectively structure and incorporate into existing supervised or reinforcement learning frameworks.

[0029] To address the aforementioned issues, existing technical approaches mainly include two approaches: one is to introduce a domain-biased reward model, which scores the model's output answers through manual annotation or expert feedback, thereby constructing a reward mechanism suitable for specific domains; the other is to rely on a general-purpose large language model as an alternative evaluator to allocate rewards to different answers generated for open-ended questions. However, both approaches have significant drawbacks. While the reward model method is domain-specific, its construction heavily relies on large-scale, high-quality manually annotated data, resulting in high training costs. Furthermore, this method is prone to "reward hacking" due to model overfitting to annotation rules or data distribution biases. In this case, the model obtains high reward values ​​through shortcuts during reinforcement learning rather than truly learning beneficial behaviors, causing the model to merely conform to the established scoring mechanism without truly understanding the essence of the task. On the other hand, while using a general-purpose large model as an evaluator reduces data dependence, it requires extremely sophisticated scoring rules and prompt engineering to guide the model to make evaluations that align with the objectives. However, because general models lack in-depth understanding of specific application scenarios, their reward outputs often deviate significantly from actual operational optimization goals, making it difficult to guarantee the effectiveness and feasibility of decisions.

[0030] Fundamentally, the openness and complexity of operational problems in vertical domains mean that there is usually no single standard for the answer; instead, it depends on multiple factors such as application context, market dynamics, and long-term strategies. Existing training paradigms often rely on verifiable, closed-ended tasks, lacking the ability to model unstructured knowledge and implicit experience, resulting in insufficient generalization ability of models in real-world application environments. Furthermore, the design of reward mechanisms is difficult to effectively align with actual application goals, further exacerbating the gap between model output and operational results. This problem limits the in-depth application of LRMs in key business scenarios, not only affecting the progress of enterprises in promoting intelligent operations but also potentially leading to decision-making risks due to unreliable model output, thereby weakening the efficiency and cost advantages of artificial intelligence technology in vertical domains.

[0031] Therefore, this application provides a policy model training method to solve the technical problem that the existing model training paradigm is difficult to apply to open policy scenarios.

[0032] Please see Figure 1 , Figure 1 This is an exemplary system architecture diagram of a strategy model training method provided in an embodiment of this application.

[0033] like Figure 1As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0034] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.

[0035] In this embodiment, at least one target question is extracted from a vertical open question-answering dataset. Terminal 101 first inputs the target question into a preset large language model, controlling the model to generate multiple sample answers corresponding to the target question offline. During the training of the current policy model, terminal 101 also inputs the target question into the current policy model, controlling it to generate multiple current answers corresponding to the target question online. Further, terminal 101 constructs a pre-trained evaluation model, inputting each current answer and multiple sample answers into the model. The evaluation model compares each current answer with each sample answer, outputting a ranking result of the current answer relative to the sample answers. Then, terminal 101 calculates the reward value for each current answer based on the ranking result, where the reward value is positively correlated with the degree to which each current answer is superior to each of the sample answers. Finally, terminal 101 constructs a policy gradient estimate based on the reward value, updates the parameters of the current policy model, and iterates through multiple rounds of training until the current policy model's performance on a preset validation set reaches a preset performance threshold.

[0036] Server 103 can be a business server providing various services. It should be noted that server 103 can be hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0037] Alternatively, the system architecture may not include server 103. In other words, server 103 may be an optional device in the embodiments of this application. That is, the method provided in the embodiments of this application can be applied to a system structure that only includes terminal 101. The embodiments of this application do not limit this.

[0038] It should be understood that Figure 1 The number of terminals, networks, and servers shown is only illustrative; the number can be any number of terminals, networks, and servers depending on the implementation requirements.

[0039] Please see Figure 2 , Figure 2 This is a flowchart illustrating a policy model training method provided in an embodiment of this application. The execution entity in this embodiment can be a terminal executing policy model training, a processor within the terminal executing the policy model training method, or a policy model training service within the terminal executing the policy model training method. For ease of description, the following example uses a processor within the terminal as the execution entity to illustrate the specific execution process of the policy model training method.

[0040] like Figure 2 As shown, policy model training methods can include at least: S202. Extract at least one target question from the vertical open question answering dataset, input the target question into the preset large language model, and control the preset large language model to generate multiple sample answers corresponding to the target question offline.

[0041] Optionally, offline data preparation is required before formal training begins. The purpose is to establish a relatively stable and comparable baseline or reference system for the subsequent reinforcement learning training phase. In this embodiment, to provide a feasible evaluation basis for open-ended questions lacking clear standard answers, before training begins, a pre-selected large language model is used to generate a diverse set of answers for an open-ended target question through offline sampling, and these answers are then solidified as a comparison benchmark for subsequent reinforcement learning training.

[0042] Specifically, in some complex decision-making scenarios, the target problem encountered by the model is often an open-ended decision problem, such as business strategy analysis or creative idea generation in a merchant operation scenario, or complex diagnostic suggestion problems in a medical scenario. These types of problems do not have a single or standard correct answer; the quality of the solution often depends on the context, domain knowledge, and logical coherence. Please refer to [link / reference needed]. Figure 3 , Figure 3 This is a logical diagram illustrating the reward value calculation mechanism in a policy model training method provided in an embodiment of this application. Figure 3 As shown, in this embodiment, at least one target question is extracted from a vertical open question-answering dataset. The target question is input into a preset large language model, which then generates multiple sample answers corresponding to the target question offline. By using the answers output by the preset large language model as the sample benchmark, an achievable and reasonable optimization objective is set: generating answers better than existing answers. This contributes to the stable convergence of the training process. Simultaneously, the offline sampling method decouples data preparation from model training, improving the efficiency and operability of the entire process and laying a solid data foundation for subsequent online optimization. Furthermore, since the sample answers also originate from the artificial intelligence model, they can provide a reasonable starting point for subsequent comparisons, ensuring the feasibility and stability of the optimization process.

[0043] Specifically, a vertical domain refers to an industry or discipline with a specialized knowledge system, specific terminology, and unique problem patterns, such as healthcare, legal consulting, and financial investment. In contrast, an open question-answering dataset refers to a collection of data consisting of "questions" and their corresponding "answers." When extracting target questions from such datasets, extraction methods can combine rule-based filtering (such as keyword matching), semantic embedding-based clustering analysis (grouping similar questions and selecting representatives from different clusters), or rely on manual annotation by domain experts to ensure that the selected questions are both representative and have generative potential.

[0044] Furthermore, the selection of the pre-set large language model is equally important when obtaining sample answers. One feasible implementation is to directly use a general-purpose large language model. The advantage of this approach lies in its convenience and low cost, enabling a rapid start to the training process. Another feasible implementation is to use a large language model fine-tuned with knowledge of the target problem's domain. Fine-tuning refers to using specialized data from a specific domain to further train the pre-trained general model, making its internal representations and knowledge distribution more closely match the language patterns and problem requirements of that domain. Sample answers generated by such models are typically more professional and targeted.

[0045] Furthermore, at the operational level, the sample acquisition process is typically offline sampling. Sampling involves inputting the target question multiple times into a selected pre-defined large language model, independently obtaining a generated answer each time. The number of samples is an adjustable hyperparameter, usually collected within the limits of computational power to form a statistically significant set of answers. This process is independent of subsequent online reinforcement learning training loops and is a one-time preprocessing step, which helps reduce the complexity and computational cost of the overall training process. These sampled answers are used as sample answers in subsequent processes, becoming the comparison objects that the policy model needs to surpass in iterations.

[0046] S204. During the training process of the current policy model, the target question is input into the current policy model, and the current policy model is controlled to generate multiple current answers corresponding to the target question online.

[0047] Optionally, after establishing the sample answers generated by the pre-defined model as a benchmark, the iterative optimization phase, i.e., the online reinforcement learning training process, can begin. Please continue reading. Figure 3 The core operation at this stage is to obtain a new set of answers generated by the current policy model for the same target question in each training iteration; these answers are the current answers. Specifically, this process unfolds in a continuously interactive online learning environment. Within a specific reinforcement learning training algorithm framework, the current policy model is a model that needs to be optimized, and its parameters need to be continuously adjusted. The current policy model will receive the exact same open-ended question (the "target question") as when generating sample answers, and based on its current capability state, it will perform inference and generate a set of responses.

[0048] It is important to note that the generation of the current answer is completed online. "Online" means that data generation and model learning occur synchronously, forming a dynamic part of the training loop. After each parameter update, the policy model's internal state changes; therefore, the "current answer" generated in a new iteration is an immediate reflection of its latest capabilities. The reinforcement learning training algorithm framework used in this embodiment serves as a set of computational rules guiding the entire training process, defining how to safely and efficiently update its parameters using the model-generated data. Its core objective is to encourage the model to explore and discover better answers while avoiding overly drastic and unstable changes to model behavior caused by a single update.

[0049] Furthermore, by generating multiple current answers in parallel, rather than a single answer, the training system can obtain a more representative sample of the model's current policy performance in this iteration. This diversity effectively reduces evaluation noise caused by the randomness of a single generation. This series of current answers, along with the previously collected offline static sample answers, will then be sent to the evaluation stage.

[0050] S206. Construct a pre-trained evaluation model, input each current answer and multiple sample answers into the evaluation model, compare each current answer with multiple sample answers through the evaluation model, and output the ranking results of each current answer relative to multiple sample answers.

[0051] Optionally, after the policy model generates the current answer, the training process enters the crucial evaluation and feedback phase. The core task of this phase is to build a pre-trained evaluation model. Each current answer and multiple sample answers are input into the evaluation model, which then systematically evaluates the quality of the policy model's output. For more details, please refer to [link to relevant documentation]. Figure 3 The evaluation model needs to compare and analyze each current answer with all previously sampled offline answers. This comparison does not require the evaluation model to perform detailed semantic breakdown or rule-based verification of the answer content, but rather focuses on performing a relative overall judgment. This relative overall judgment means using sample answers as a reference to judge the performance of the current answer relative to each sample answer from overall dimensions such as "whether it accurately responds to the task requirements," "whether the information completeness meets the standards," and "the logic and clarity of expression." For example, in a question-and-answer task, if the current answer A accurately covers the core knowledge points of the question and is concisely expressed, while sample answer B has some missing information, the evaluation model will conclude that "the current answer A is superior to sample answer B" through overall judgment. This judgment method avoids the high cost and detail bias of detailed semantic breakdown and solves the problem of insufficient flexibility in rule-based verification. Therefore, judging whether "answer A is superior to answer B" yields a far more intuitive and reliable comparison result than independently assigning specific scores to A and B. The final output of this step is the ranking of each current answer relative to the sample answer set. This quantitative metric clearly indicates the relative position of the current answer's quality within the sample output. By systematically comparing, relatively judging, and ranking the answers against the sample answers, the evaluation model provides more transparent feedback signals for the black-box operation, ultimately driving the model to gradually approach the ideal output standard of the task through continuous iteration.

[0052] In this embodiment, the selection of the evaluation model can revolve around the model's training paradigm, knowledge adaptability, and task matching degree. In one feasible implementation, a general reward model trained based on contrastive learning can be used as the evaluation model, or a vertical domain reasoning model trained in combination with knowledge of the target problem's context can be used as the evaluation model. The design logic and application scenarios of the two types of models have different focuses, and the final selection should be made according to the specific task requirements.

[0053] On the one hand, general reward models trained based on contrastive learning are widely used tools in evaluation scenarios, and their core function is explicitly configured to directly judge the merits of a pair of input answers. From the training logic perspective, the core of contrastive learning is to allow the model to learn the difference features in a large number of paired samples of "high-quality answers and low-quality answers," ultimately forming the ability to judge "which answer is better." General open-source models, trained on large-scale general corpora through contrastive learning, already possess the basic ability to identify the merits of answers in most general scenarios (such as daily question answering and common sense reasoning), and their open-source nature gives them the advantages of low-cost deployment and rapid adaptation to general tasks.

[0054] On the other hand, vertical domain reasoning models are specialized evaluation tools designed for specific scenario-domain problems. Their core advantage lies in their deep adaptation to the knowledge of the target problem's domain. Theoretically, while general-purpose models cover a wide range of general knowledge, in vertical domains (such as business marketing, medical treatment, legal compliance, and industrial technology), the density and depth of domain knowledge in the training corpus are often insufficient, making it difficult to accurately determine whether the answer conforms to the professional standards within the domain. Vertical domain reasoning models, by incorporating professional corpus from that domain during training (such as clinical guidelines and case literature in the medical field, and legal interpretations and case analyses in the legal field), can establish judgment logic that conforms to the rules of the domain, thereby making more accurate judgments on the professionalism, accuracy, and logical compliance of answers within the domain. Therefore, in vertical domain tasks, their evaluation results are more reliable. In practical applications, the selection of different evaluation models can also be dynamically adjusted strictly according to task needs. From the perspective of the overall training process, the selection of evaluation models is not an isolated decision, but rather guided by the optimization goals of the strategy model, thus scientifically selecting the evaluation model.

[0055] S208. Calculate the reward value corresponding to each current answer by ranking the merits of each current answer. The reward value is positively correlated with the degree to which each current answer is better than each of the sample answers in the multiple sample answers.

[0056] Alternatively, please continue reading Figure 3The system needs to convert this ranking result into reward values ​​that the reinforcement learning framework can directly utilize. These reward values ​​guide the direction of policy model behavior optimization. The design principle ensures that the reward value is strictly positively correlated with the degree to which each current answer is superior to the sample answers, thus achieving the effect that the better the current answer, the higher the reward value. An intuitive and robust implementation is to calculate the percentage of times the current answer wins in pairwise comparisons, i.e., the percentage of times the answer is superior to the sample answers out of the total number of comparisons. For example, if a current answer wins 6 out of 8 sample answers, its win rate of 75% can be used directly or after processing with a smoothing function as its reward value. This reward mechanism design allows the model to be explicitly guided to generate answers that consistently outperform existing model performance, rather than pursuing an abstract and potentially unattainable standard, thus achieving stable and measurable iteration and progress.

[0057] S210. Construct a policy gradient estimate based on the reward value, update the parameters of the current policy model, and train it through multiple rounds of iterations until the performance of the current policy model on the pre-set validation set reaches the preset performance threshold.

[0058] Optionally, the reward values ​​corresponding to all these current answers are pooled together and used to adjust the internal parameters of the current policy model. This step is typically performed by reinforcement learning algorithms such as Proximal Policy Optimization (PPO). Reinforcement learning training is a machine learning method, specifically a machine learning paradigm where an agent interacts with its environment and adjusts its behavioral policy based on reward signals. These algorithms can calculate the direction and magnitude of the model parameters to be adjusted based on the obtained reward signals, and then update them using optimization techniques such as gradient descent. The goal is to significantly increase the probability of the model generating high-reward answers when faced with the same problem in the future. Specifically, policy gradient estimates can be constructed based on these reward values. In the reinforcement learning framework, the gradient indicates in the model parameter space which small adjustments to the parameters can effectively improve the expected value. However, in complex generative tasks, the relationship between the reward function and the output of the policy model is usually a non-differentiable black box, and the expected value cannot be calculated precisely. Therefore, it is necessary to construct approximate values ​​of the gradient through sampling and estimation methods. By sampling multiple generated results from the current policy model, the gradient of the log probability of each result relative to the model parameters is calculated. This gradient is then weighted and averaged using the reward value obtained, resulting in an unbiased but potentially high-variance gradient estimate. Gradient directions corresponding to high-reward samples are reinforced, and the model parameters are adjusted to generate more similar high-reward samples; conversely, gradient directions corresponding to low-reward samples are suppressed. This process is repeated until the model reaches the expected convergence state. When the policy model reaches convergence, it means that after sufficient training, its performance has stabilized at a high level, and the quality of its generated answers and corresponding reward values ​​no longer show significant improvement or drastic fluctuations with continued training. At this point, the policy model has successfully evolved into a highly specialized intelligent model capable of producing better solutions to the target problem. The entire process tightly couples relative comparison, reward calculation, and parameter updates, constructing an efficient autonomous learning loop that enables machine intelligence to achieve continuous and targeted capability evolution even in complex domains lacking clear standards.

[0059] In this embodiment, a strategy model training method is provided. At least one target question is extracted from a vertical open question-answering dataset. The target question is input into a preset large language model, which is then controlled to generate multiple sample answers corresponding to the target question offline. During the training of the current strategy model, the target question is input into the current strategy model, which is then controlled to generate multiple current answers corresponding to the target question online. A pre-trained evaluation model is constructed, and each current answer and multiple sample answers are input into the evaluation model. The evaluation model compares each current answer with each sample answer and outputs a ranking result of the current answer relative to the multiple sample answers. A reward value is calculated based on the ranking result of each current answer, where the reward value is positively correlated with the degree to which each current answer is superior to each of the multiple sample answers. A strategy gradient estimate is constructed based on the reward value, and the parameters of the current strategy model are updated. The model is iteratively trained through multiple rounds until the performance of the current strategy model on a preset validation set reaches a preset performance threshold. During training, the system obtains a set of current answers generated by the current policy model, which is in an optimized state, for the same target problem. By dynamically sampling the real-time performance of the policy model during training iterations, it captures changes in its policy space. Using multiple answers for model training provides more robust data support for policy evaluation, effectively reducing random fluctuations during training and making the adjustment of model parameters more accurate and reliable, thereby accelerating the convergence of the training process. Next, the evaluation model compares each current answer with all sample answers and outputs a ranking result. The core is to transform the complex and subjective task of answer quality evaluation from a scoring mechanism based on standard answers to a relative ranking mechanism. The evaluation model only needs to determine which of the given two or a set of answers is better. This relative comparison reduces the difficulty and ambiguity of the evaluation task itself, allowing the model to bypass the fundamental problem of defining precise quantitative standards for such problems and achieve automated evaluation of open-ended and subjective problems. The step of determining the reward value based on the ranking results transforms the relative ranking information into a continuous reward value that can be directly used by the reinforcement learning algorithm. This reward value is positively correlated with the degree to which the current answer is better than the sample answers. This reward mechanism provides the policy model with a clear and continuous optimization direction. The model is explicitly guided towards the goal of generating better answers. This reward design ensures the feasibility of the training objective and the stability of the training process, effectively motivating the model to conduct beneficial explorations.Finally, based on the reward values ​​corresponding to all current answers, the parameters of the current policy model are adjusted through an optimization algorithm, and the above training process is repeated until the model converges. This step realizes the automated and targeted evolution of the policy model's capabilities, enabling the policy model to achieve stable and autonomous performance improvement under the guidance of reward signals, forming a self-driven virtuous evolutionary cycle, thereby better solving problems in open policy scenarios.

[0060] Please see Figure 4 , Figure 4 This is a flowchart illustrating a strategy model training method provided in an embodiment of this application.

[0061] like Figure 4 As shown, policy model training methods can include at least: S402. Obtain multiple sample answers generated by a preset large language model for the target question.

[0062] S404. During the training of the current policy model, obtain multiple current answers generated by the current policy model for the target problem.

[0063] For details regarding steps S402-S404, please refer to the detailed descriptions in steps S202-S204, which will not be repeated here.

[0064] S406. Pair each current answer with each sample answer to form multiple answer pairs; input each answer pair into the evaluation model, and the evaluation model outputs the superiority or inferiority judgment result of each answer pair; and output the superiority or inferiority ranking result of each current answer relative to the multiple sample answers.

[0065] In this embodiment, the core of the evaluation mechanism lies in transforming abstract quality judgments into quantifiable relative comparisons. To achieve this goal, a feasible implementation method is to use a "pairwise comparison" approach. The operation of this method is as follows: The system first systematically pairs each current answer to be evaluated with each sample answer from offline sampling, thereby generating a set of answer pairs containing all possible combinations. Subsequently, each independent answer pair is input into the evaluation model sequentially or in parallel. The evaluation model performs a binary judgment on the input answer pairs and outputs a judgment result representing "which is better or worse".

[0066] Furthermore, after all answer pairs have been evaluated, the system enters the data aggregation phase. For each current answer, the system counts the number of times it wins in all pairwise comparisons with related answers; this number of wins constitutes the original ranking score for that current answer. This ranking method based on the number of wins ensures that each current answer competes with all benchmark answers, thereby guaranteeing that the final ranking objectively and comprehensively reflects its relative quality position within the entire answer group. This pairwise comparison mechanism significantly reduces the demands on the evaluation model's capabilities; the model only needs to distinguish relative superiority or inferiority, without needing to master the difficult skill of precisely calibrating the standard scores. Secondly, by decomposing the global ranking into a large number of independent local judgments and then re-aggregating them, the final ranking result has stronger statistical stability and reliability, providing a solid and less noisy basis for subsequent reward calculations, thus driving the targeted optimization of the strategy model.

[0067] Alternatively, in addition to one feasible implementation in step S406 above, the ranking results can also be obtained according to another feasible implementation in step S408. The specific content of step S408 includes: S408. Construct answer sets corresponding to each current answer and all sample answers, and input them into the evaluation model. The evaluation model performs an overall evaluation of all answers in each answer set and outputs a ranking list of the best and worst answers for each answer set. It also outputs the ranking results of each current answer relative to multiple sample answers.

[0068] In this application embodiment, in addition to the pairwise comparison-based evaluation mechanism, a more comprehensive alternative implementation method, namely the "list sorting" method, is provided. The core of this method lies in performing a holistic quality assessment and sorting of the answer group. Its specific implementation process is as follows: The system first dynamically constructs a dedicated answer set for each current answer that needs to be evaluated. This set consists of the specific current answer and all offline sampled answers. Subsequently, this complete set, containing both new and old answers, is input into the evaluation model for processing.

[0069] Furthermore, in this model, the evaluation model's responsibility is to comprehensively consider and compare all answers in a given list. Ultimately, the output of the evaluation model is a complete ranking list for the input set. This list explicitly indicates a quality sequence from best to worst. For each current answer, its final ranking is determined by its position within this overall ranking list. For example, it can directly use the reciprocal of its position or quantify its relative performance based on its percentile in the list. The higher the ranking, the higher its relative quality within the set. Therefore, using this list ranking mechanism can improve computational efficiency and reduce computational overhead.

[0070] S410. Based on the ranking of the current answers, the reward value corresponding to each current answer is obtained after processing by a preset smoothing function.

[0071] Optionally, after ranking the current answers, the training process enters the reward quantification and model optimization stage. First, the ranking results need to be converted into reward signals usable in reinforcement learning. Since the original ranking data (such as the number of wins or ranking position) may exhibit discrete or uneven distribution characteristics, directly using it as a reward value may lead to unstable gradients or overly sharp optimization directions during training. Therefore, the system introduces a preset smoothing function to mathematically process this original ranking result. Commonly used smoothing functions include the Sigmoid function or the Softmax function. These functions can map the original ranking values ​​to a continuous, smooth, and differentiable range of reward values. For example, the number of wins can be compressed to the range (0,1) using the Sigmoid function and used as a reward value; or the Softmax function can be used to convert a set of ranking scores for the current answers into a probability distribution. The core purpose is to make the distribution of reward values ​​more uniform and smooth while preserving the relative ranking relationships. This process ensures that the reward signal can accurately reflect the relative differences in answer quality and provide good mathematical properties for subsequent gradient calculation, preventing training oscillations or divergences caused by sudden changes or inappropriate magnitudes of reward values.

[0072] S412. Smooth and normalize all reward values, construct a policy gradient estimate based on the processed reward values, calculate the gradient direction in the parameter space of the current policy model that increases the probability of generating answers with high reward values, and fine-tune the parameters of the current policy model based on the gradient direction until the current policy model converges.

[0073] Optionally, after obtaining the smoothed reward value for each current answer, these values ​​are used to guide the parameter updates of the current policy model. The core of this process is calculating the policy gradient. The policy gradient is an important method in reinforcement learning; its basic idea is to estimate the direction of parameter adjustment by evaluating the rewards obtained for each action (i.e., the generated answer) under the current policy. Specifically, the system calculates a gradient vector with respect to the model parameters based on this series of reward values. This gradient points in the parameter space in a direction that increases the probability of the model generating high-reward answers in the future. Subsequently, the millions or even billions of parameters of the policy model are fine-tuned along this gradient direction using gradient descent or its variants (such as the Adam optimizer).

[0074] In this embodiment, this complex parameter adjustment process is implemented using a Group Relative Policy Optimization (GRPO) reinforcement learning algorithm. GRPO estimates the dominance function by comparing policy performance within a group, thus guiding policy updates more stably and efficiently. This algorithm is an improvement and extension of the classic proximal policy optimization algorithm. Its core feature is that it does not consider each answer and its reward value in isolation, but focuses on the answer's relative performance within the group. The GRPO algorithm uses the reward values ​​obtained in previous steps, based on relative comparisons, to calculate a more robust policy gradient estimate. This algorithm reduces the variance of the gradient estimate by introducing a baseline composed of the average performance within the group, and ensures the stability and safety of the training process by constraining the magnitude of each parameter update, avoiding the destruction of valuable knowledge already learned by the model due to excessively large single updates. The entire iterative process is repeated until the current policy model reaches convergence. Convergence is a key milestone marking the completion of training. The criteria for determining convergence can be that the average reward value of the policy model on a series of validation questions no longer shows a significant upward trend and stabilizes at a certain high level, or that the quality of the answers generated by the model no longer shows a substantial improvement in multiple iterations.

[0075] This application provides a method for training a strategy model. The pairwise comparison evaluation method reduces potential systematic biases during the evaluation process, ultimately yielding ranking results with stronger statistical stability and reliability, thereby more effectively driving the targeted optimization of the strategy model. The list-ranking evaluation method effectively captures the continuous distribution of answer quality within the population, providing the strategy model with a reward signal that has a smoother gradient and higher discriminative power, guiding it to make more precise behavioral adjustments. Through the model iterative optimization mechanism in this application, the strategy model can gradually and stably optimize its internal parameters under the continuous guidance of relative evaluation signals, ultimately achieving a significantly higher quality of generated answers on a specific task than the initial sample answer benchmark.

[0076] Please see Figure 5 , Figure 5 This is a flowchart illustrating a strategy generation method provided in an embodiment of this application.

[0077] like Figure 5 As shown, the strategy generation method may include at least: S502, responding to a policy generation request triggered by the user in the interactive interface, wherein the policy generation request contains at least a policy question in the form of a natural language description.

[0078] Optionally, when a user raises a question about strategy generation, the system first needs to capture and parse the user's explicit intent. Specifically, when a user interacts with the system's front-end interface (such as a web application, chat window, or API client), they can trigger a strategy generation request by clicking a button, sending a message, or calling a specific interface. The core payload of the strategy generation request includes at least a strategy question described in natural language. Natural language description means that users do not need to learn any specific programming language, query syntax, or structured templates; instead, they can express their complex needs in the manner of everyday human communication. This design greatly lowers the barrier to entry, expands the potential user base, and directly reflects the inclusiveness of artificial intelligence technology. The system's back-end service continuously listens for such requests, and once captured, immediately activates the subsequent processing pipeline. This step establishes a user-centric, natural language-based, and user-friendly entry point, accurately translating unstructured user intent into computationally readable tasks within the system.

[0079] S504. Input the policy problem into the current policy model and control the policy answer output by the current policy model for the policy problem.

[0080] Optionally, in the core policy generation phase, the current policy model is used to output the user's question. The current policy model used here refers to the version of the model that has undergone optimization through methods such as reinforcement learning and has reached a preset performance threshold. Specifically, this step involves inputting the policy question into the current policy model and controlling the policy answer output by the current policy model. Typically, the natural language question in the request is given necessary system prompts and contextual instructions (e.g., "You are a corporate strategy consultant; please answer in a clear and structured manner"), and tokenized into a sequence of numerical vectors that the model can understand. Subsequently, the model enters the forward inference process. Based on its large number of parameters and deep attention mechanisms such as the Transformer architecture, the model generates the policy answer word by word (token).

[0081] S506. Generate a strategy response text in natural language form corresponding to the strategy question based on the strategy answer, and display the strategy response text on the interactive interface.

[0082] Optionally, the system ultimately generates a natural language description of the strategy answer to the strategy question based on the strategy answer, and displays the strategy answer text on the interactive interface. Generating the natural language description of the strategy answer to the strategy question based on the strategy answer includes necessary post-processing and formatting operations. The raw text directly output by the model may need to be standardized in terms of punctuation, paragraph breaks, and list formatting. It may also need to be quickly checked against the knowledge base to embed the latest data or cited sources, and even perform sensitive information filtering and security review. Subsequently, the system displays the strategy answer text on the interactive interface, thus providing the user with the results of the system's intelligent calculation in an intuitive and user-friendly way, thereby completing this interactive cycle.

[0083] In this embodiment, a strategy generation method is provided, responding to a strategy generation request triggered by a user in an interactive interface. The strategy generation request includes at least a strategy question in natural language description form. The strategy question is input into a current strategy model, and the current strategy model is controlled to output a strategy answer to the strategy question. Based on the strategy answer, a strategy response text in natural language description form corresponding to the strategy question is generated, and the strategy response text is displayed on the interactive interface. The current strategy model is the current strategy model in any of the above embodiments. First, the user's question in natural language description form is received; then, a trained and converged strategy model is invoked to transform the unstructured question into structured strategy insights and solution text; finally, the generated strategy answer is ensured to be converted into natural language description form and fed back to the user in an intuitive way. This process not only allows for convenient invocation of model capabilities but also ensures the stability, security, and predictability of each interaction through a reliable engineering closed loop.

[0084] Please see Figure 6 , Figure 6 This is a structural block diagram of a strategy model training device provided in an embodiment of this application. Figure 6 As shown, the policy model training device 600 includes: The sample acquisition module 610 is used to acquire multiple sample answers generated by a preset large language model for the target question; The online sampling module 620 is used to obtain multiple current answers generated by the current policy model for the target problem during the training process of the current policy model; The answer ranking module 630 is used to compare each current answer with multiple sample answers through the evaluation model, and output the ranking results of each current answer relative to the multiple sample answers. The reward calculation module 640 is used to determine the reward value corresponding to each current answer based on the ranking results of each current answer. The size of the reward value is positively correlated with the degree to which each current answer is better than each of the sample answers in the multiple sample answers. The model tuning module 650 is used to adjust the parameters of the current strategy model based on all reward values ​​until the current strategy model converges.

[0085] Optionally, the answer sorting module 630 is also used to combine each current answer with each sample answer in pairs to form multiple answer pairs; input each answer pair into the evaluation model, and output the evaluation model to determine the merits of each answer pair.

[0086] Optionally, the ranking result is determined based on the number of times the current answer wins among all the ranking results.

[0087] Optionally, the answer sorting module 630 is also used to construct the answer set corresponding to each current answer and all sample answers respectively, and input them into the evaluation model; the evaluation model performs an overall evaluation of all answers in each answer set, and outputs a ranking list of the superiority and inferiority of each answer set.

[0088] Optionally, the ranking result is determined based on the current answer's position in the sorted list.

[0089] Optionally, the reward calculation module 640 is also used to obtain the reward value corresponding to each current answer after processing by a preset smoothing function based on the ranking results of each current answer.

[0090] Optionally, the process of adjusting the parameters of the current policy model based on all reward values ​​is implemented by a reinforcement learning algorithm based on group-relative policy optimization.

[0091] Optionally, the model tuning module 650 is also used to calculate the policy gradient of the current policy model based on all reward values ​​in order to update the parameters of the current policy model.

[0092] Optionally, the evaluation model is a reward model trained based on contrastive learning, and the evaluation model is configured to judge the merits of a pair of input answers.

[0093] Optionally, the evaluation model is a vertical domain inference model trained based on the domain knowledge of the scenario to which the target problem belongs.

[0094] Optionally, the default large language model is a general large language model; or, the default large language model is a large language model that has been fine-tuned with knowledge of the scenario domain to which the target problem belongs.

[0095] Optionally, the target problem can be an open-ended decision problem.

[0096] Please see Figure 7 , Figure 7 This is a structural block diagram of a strategy generation device provided in an embodiment of this application. Figure 7 As shown, the strategy generation device 700 includes: The request receiving module 710 is used to respond to a strategy generation request triggered by the user in the interactive interface, wherein the strategy generation request contains at least a strategy question in the form of a natural language description. The strategy generation module 720 is used to input the strategy problem into the current strategy model and control the strategy answer output by the current strategy model for the strategy problem; The interactive response module 730 is used to generate a strategy response text in natural language form corresponding to the strategy question based on the strategy answer, and to display the strategy response text on the interactive interface; The current strategy model is the current strategy model in any of the above embodiments.

[0097] This application provides a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to perform the steps of any of the methods described in the above embodiments.

[0098] This application also provides a computer storage medium that can store multiple instructions adapted for loading by a processor and executing the steps of any of the methods described in the above embodiments.

[0099] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Figure 8 As shown, terminal 800 may include: at least one terminal processor 801, at least one network interface 804, user interface 803, memory 805, and at least one communication bus 802.

[0100] The communication bus 802 is used to enable communication between these components.

[0101] The user interface 803 may include a display screen and a camera. Optionally, the user interface 803 may also include a standard wired interface and a wireless interface.

[0102] The network interface 804 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0103] The terminal processor 801 may include one or more processing cores. The terminal processor 801 connects to various parts within the terminal 800 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 805, and by calling data stored in the memory 805. Optionally, the terminal processor 801 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The terminal processor 801 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the terminal processor 801.

[0104] The memory 805 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 805 may include a non-transitory computer-readable storage medium. The memory 805 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 805 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 805 may also be at least one storage device located remotely from the aforementioned terminal processor 801. Figure 8 As shown, the memory 805, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a policy model training and / or policy generation program.

[0105] exist Figure 8In the terminal 800 shown, the user interface 803 is mainly used to provide an input interface for the user and to obtain the user's input data; while the terminal processor 801 can be used to call the policy model training program stored in the memory 805 and specifically perform the following operations: Obtain multiple sample answers generated by a pre-defined large language model for the target question; During the training of the current policy model, obtain multiple current answers generated by the current policy model for the target problem; The evaluation model compares each current answer with multiple sample answers and outputs a ranking of the current answer relative to the multiple sample answers. The reward value for each current answer is determined based on the ranking of the current answers. The size of the reward value is positively correlated with the degree to which each current answer is better than each of the sample answers in the multiple sample answers. Adjust the parameters of the current policy model based on all reward values ​​until the current policy model converges.

[0106] In some embodiments, when the terminal processor 801 inputs each current answer and multiple sample answers into the evaluation model and compares each current answer with multiple sample answers through the evaluation model, it specifically performs the following steps: combining each current answer and each sample answer in pairs to form multiple answer pairs; inputting each answer pair into the evaluation model, and having the evaluation model output the superiority or inferiority judgment result of each answer pair.

[0107] In some embodiments, the ranking result is determined based on the number of times the current answer wins in all ranking results.

[0108] In some embodiments, when the terminal processor 801 inputs each current answer and multiple sample answers into the evaluation model, and compares each current answer with multiple sample answers through the evaluation model, it specifically performs the following steps: each current answer and all sample answers constitute an answer set corresponding to each current answer, and inputs it into the evaluation model; the evaluation model performs an overall evaluation of all answers in each answer set, and outputs a ranking list of the superiority and inferiority of each answer set.

[0109] In some embodiments, the ranking result is determined based on the current answer's position in the ranking list.

[0110] In some embodiments, when the terminal processor 801 calculates the reward value corresponding to each current answer by ranking the merits of each current answer, it specifically performs the following steps: based on the ranking of merits of each current answer, the reward value corresponding to each current answer is obtained after processing by a preset smoothing function.

[0111] In some embodiments, the process of adjusting the parameters of the current policy model based on all reward values ​​is implemented by a reinforcement learning algorithm based on group-relative policy optimization.

[0112] In some embodiments, when the terminal processor 801 constructs a policy gradient estimate based on the reward value and updates the parameters of the current policy model, it specifically performs the following steps: smoothing and normalizing all reward values, constructing a policy gradient estimate based on the processed reward values, calculating the gradient direction in the parameter space of the current policy model that increases the probability of generating answers with high reward values, and fine-tuning the parameters of the current policy model based on the gradient direction.

[0113] In some embodiments, the evaluation model is a reward model trained based on contrastive learning, and the evaluation model is configured to judge the merits of a pair of input answers.

[0114] In some embodiments, the evaluation model is a vertical domain inference model trained based on knowledge of the scenario domain to which the target problem belongs.

[0115] In some embodiments, the preset large language model is a general large language model; or, the preset large language model is a large language model finely tuned with knowledge of the scenario domain to which the target problem belongs.

[0116] In some embodiments, the target problem is an open-ended decision problem.

[0117] exist Figure 8 In the terminal 800 shown, the user interface 803 is mainly used to provide an input interface for the user and to obtain the user's input data; while the terminal processor 801 can also be used to call the policy generation program stored in the memory 805 and specifically perform the following operations: In response to a strategy generation request triggered by a user in the interactive interface, the strategy generation request shall at least contain a strategy question in the form of a natural language description; Input the policy problem into the current policy model, and control the policy answer output by the current policy model for the policy problem; Generate a strategy response text in natural language form corresponding to the strategy question based on the strategy answer, and display the strategy response text on the interactive interface; The current strategy model is the current strategy model in any of the above embodiments.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0119] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0120] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0121] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this application.

[0122] The information, data, and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0123] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0124] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0125] The above is a description of a strategy model training and strategy generation method, apparatus, storage medium and terminal provided in the embodiments of this application. For those skilled in the art, based on the ideas of the embodiments of this application, there will be changes in the specific implementation and application scope. Therefore, the content of this application should not be construed as a limitation on the embodiments of this application.

Claims

1. A method for training a policy model, characterized in that, The method includes: Extract at least one target question from the vertical open question answering dataset, input the target question into a preset large language model, and control the preset large language model to generate multiple sample answers corresponding to the target question offline; During the training process of the current strategy model, the target question is input into the current strategy model, and the current strategy model is controlled to generate multiple current answers corresponding to the target question online; Construct a pre-trained evaluation model, input each current answer and the multiple sample answers into the evaluation model, and compare each current answer with the multiple sample answers through the evaluation model, and output the ranking result of each current answer relative to the multiple sample answers; The reward value for each current answer is calculated by ranking the current answers according to their merits. The magnitude of the reward value is positively correlated with the degree to which each current answer is superior to each of the multiple sample answers. Based on the reward value, a policy gradient estimate is constructed, the parameters of the current policy model are updated, and the model is trained through multiple rounds of iteration until the performance of the current policy model on the preset validation set reaches a preset performance threshold.

2. The method according to claim 1, characterized in that, The step of inputting each current answer and the plurality of sample answers into the evaluation model, and comparing each current answer with the plurality of sample answers through the evaluation model, includes: Each current answer is paired with each sample answer to form multiple answer pairs; Each answer pair is input into the evaluation model, and the evaluation model outputs the judgment result of the merits of each answer pair.

3. The method according to claim 2, characterized in that, The ranking result is determined based on the number of times the current answer wins among all the ranking results.

4. The method according to claim 1, characterized in that, The step of inputting each current answer and the plurality of sample answers into the evaluation model, and comparing each current answer with the plurality of sample answers through the evaluation model, includes: Each current answer is combined with all the sample answers to form the answer set corresponding to each current answer, and then input into the evaluation model; The evaluation model performs an overall evaluation of all answers in each answer set and outputs a ranking list of the best and worst answers for each answer set.

5. The method according to claim 4, characterized in that, The ranking result is determined based on the current answer's position in the ranking list.

6. A strategy generation method, characterized in that, The method includes: In response to a strategy generation request triggered by a user in an interactive interface, the strategy generation request includes at least a strategy question in the form of a natural language description; Input the policy problem into the current policy model, and control the policy answer output by the current policy model for the policy problem; Generate a strategy answer text in natural language form corresponding to the strategy question based on the strategy answer, and display the strategy answer text on the interactive interface; Wherein, the current strategy model is the current strategy model as described in any one of claims 1 to 5.

7. A strategy model training device, characterized in that, The device includes: The sample acquisition module is used to extract at least one target question from the vertical open question answering dataset, input the target question into a preset large language model, and control the preset large language model to generate multiple sample answers corresponding to the target question offline; An online sampling module is used to input the target question into the current policy model during the training process of the current policy model, and control the current policy model to generate multiple current answers corresponding to the target question online; The answer ranking module is used to construct a pre-trained evaluation model. Each current answer and the multiple sample answers are input into the evaluation model. The evaluation model compares each current answer with the multiple sample answers and outputs the ranking result of each current answer relative to the multiple sample answers. The reward calculation module is used to calculate the reward value corresponding to each current answer based on the ranking of the current answers. The reward value is positively correlated with the degree to which each current answer is better than each of the sample answers in the plurality of sample answers. The model parameter tuning module is used to construct a policy gradient estimate based on the reward value, update the parameters of the current policy model, and train it through multiple rounds of iterations until the performance of the current policy model on the preset validation set reaches a preset performance threshold.

8. A strategy generation apparatus, characterized in that, The device includes: The request receiving module is used to respond to a strategy generation request triggered by a user in the interactive interface, wherein the strategy generation request contains at least a strategy question in the form of a natural language description. The strategy generation module is used to input the strategy problem into the current strategy model and control the current strategy model to output the strategy answer to the strategy problem; An interactive response module is used to generate a strategy response text in natural language form corresponding to the strategy question based on the strategy answer, and to display the strategy response text on the interactive interface; Wherein, the current strategy model is the current strategy model as described in any one of claims 1 to 5.

9. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as claimed in any one of claims 1 to 5 or 6.

10. A terminal, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any one of claims 1 to 5 or 6.

Images