Method and device for training business model based on reinforcement learning

Abstract

The embodiment of the invention provides a method and a device for training a business model based on reinforcement learning, which can determine reinforcement learning task weights of samples under corresponding tasks according to the performance of the business model on each task under the condition that the business model of multiple tasks is trained based on a reinforcement learning principle. And the contribution of each task sample in the model loss is determined through the task weight. Therefore, the importance of the reward values corresponding to the simple tasks and the complex tasks can be distinguished according to the task weights determined dynamically and objectively in the model loss, so that the performance of the multi-task business model on various tasks can be improved more uniformly.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to a method and device for training a business model based on reinforcement learning. BACKGROUND

[0002] Reinforcement Learning (RL) as an important branch of machine learning has made significant progress in both theory and application in recent years. The core goal is to learn the optimal strategy to maximize long-term cumulative rewards through the interaction between the agent and the environment. With the development of RL, it has shown strong adaptability in complex tasks.

[0003] The traditional reinforcement learning theory mainly relies on the value function method (such as Q-learning, SARS A, etc.) and the policy gradient method (such as REINFORCE, etc.). The research hotspots of reinforcement learning include: Multi-Agent Reinforcement Learning (MARL), which focuses on game theory and cooperative tasks, studies the competition and cooperation mechanism between agents, and typical applications include multi-UAV cooperation, traffic signal optimization, etc.; Meta-RL, which improves the adaptability of agents in new tasks by learning the method of learning, such as the rapid migration ability in the Meta-World benchmark test; imitation learning and inverse reinforcement learning, which use expert demonstration data to reduce the exploration cost and are applied to robot operation and autonomous driving fields; Safe RL, which avoids dangerous behavior of agents in training through constraint optimization or safety-constrained policy, such as fault avoidance in industrial control; and so on.

[0004] Deep Reinforcement Learning (DRL) is the combination of deep neural networks and reinforcement learning. During the training process of deep neural networks (such as large language models), reinforcement learning principles can be used to give rewards to the results, thereby training the deep neural network. DRL relies on a large number of environment interactions, resulting in high training costs, especially for multi-task scenarios (such as vertical application scenarios of large language models in specific fields). In the reinforcement learning process, the reward signal is usually used to guide the model to generate responses that better meet professional standards and user expectations. However, multi-task scenarios have significant complexity differences, and different tasks have different requirements and challenges for the model.

[0005] Traditional reinforcement learning methods use the same learning strategy for all tasks in the model training process without distinguishing the differences in task difficulty. This unified training method leads to the model quickly converging on simple tasks, but not learning adequately on complex and critical professional tasks. For example, in the legal field, simple statute query tasks are relatively easy to master, but complex case analysis, legal reasoning, and other tasks require deeper professional knowledge and logical reasoning capabilities. Therefore, how to train neural networks based on reinforcement learning on various tasks is a technical problem worth studying. SUMMARY

[0006] One or more embodiments of the present specification describe a method and device for training a business model based on reinforcement learning to solve one or more problems mentioned in the background.

[0007] According to a first aspect, a method for training a business model based on reinforcement learning is provided, the method comprising a plurality of update cycles, in a single update cycle: obtaining a plurality of training samples, each training sample comprising input data, a task type, a correct response, and an incorrect response; determining a predicted response obtained by processing each training sample based on the business model, and determining a task weight corresponding to each task type, wherein a single task weight is determined based on the correctness of the predicted response corresponding to each training sample of the corresponding task type; predicting a reward difference of the predicted response corresponding to each training sample relative to the correct response and the incorrect response through a reward model; determining a model loss based on each task weight and each reward difference, the model loss being used to update the business model, the model loss comprising a reinforcement learning loss obtained by: using the task weight corresponding to each training sample as a weight coefficient to weight each sample loss, and a single sample loss being determined based on the reward difference.

[0008] In one embodiment, the business model is a large language model, and the input data is prompt information for the large language model, including at least one of the following: text, audio, and image.

[0009] In an embodiment, the plurality of training samples includes a first sample, the first sample corresponding to a first predicted response, a first correct response, and a first task type, the correctness of the first predicted response including: in a case where the first task type is a classification task, detecting whether a classification category corresponding to the first predicted response is the same as a classification category corresponding to the first correct response, and determining whether the first predicted response is correct according to a detection result; in a case where the first task type is a question answering task, the first predicted response and the first correct response being texts, and determining whether the first predicted response is correct by an embedding vector similarity of the first predicted response and the first correct response; in a case where the first task type is an image generation task, the first predicted response and the first correct response being images, and determining whether the first predicted response is correct by a pixel-by-pixel comparison or a feature vector similarity of the first predicted response and the first correct response; in a case where the first task type is a code generation task, the first predicted response and the first correct response being codes, and determining whether the first predicted response is correct by comparing whether respective running results of the first predicted response and the first correct response are consistent.

[0010] In an embodiment, the plurality of training samples includes a second sample, the second sample corresponding to second input data, and the correctness of a predicted response corresponding to the second sample being determined by: performing k times processing on the second data by using the business model to obtain k second predicted responses; and determining the correctness of the predicted response corresponding to the second sample according to a pass@k score of at least one correct one of the k second predicted responses.

[0011] In an embodiment, a single task weight corresponding to a single task type is determined by: determining a prediction accuracy of the business model on the single task type based on the correctness of respective predicted responses corresponding to respective training samples; determining a current weight corresponding to the single task type according to the prediction accuracy; and determining the single task weight based on the current weight.

[0012] In a further embodiment, for a single task type, the prediction accuracy is one of: a ratio of a number of training samples corresponding to a predicted response correct for the single task type to a total number of training samples corresponding to the single task type; and a Pass@k score average of respective training samples corresponding to the single task type.

[0013] In another further embodiment, the current weight corresponding to the single task type is positively correlated with a prediction error rate of the single task type, and the prediction error rate is a difference between 1 and the prediction accuracy of the business model on the single task type.

[0014] In a further embodiment, the current weight corresponding to the single task type is further positively correlated with a difference between a prediction accuracy of the reference model on the single task type and a prediction accuracy of the business model on the single task type, and 0 if the difference is less than 0; and the prediction accuracy of the reference model on the single task type is a mean of the prediction accuracy of each reference model on the single task type.

[0015] In an embodiment, determining the single task weight based on the current weight comprises: in a case where the single task type first appears, taking the current weight or a normalized value of the current weight based on the current weight of each task type as the single task weight; in a case where the single task type first appears, performing a sliding average on the current weight and the task weight determined last time to obtain a sliding average value, or taking a normalized value of the sliding average value of the single task weight of each task type as the single task weight.

[0016] According to a second aspect, an apparatus for training a business model based on reinforcement learning is provided, the apparatus comprising an obtaining unit, a determining unit, a reward unit, and an updating unit; the apparatus performs reinforcement learning-based training on the business model through multiple updating cycles, and in a single updating cycle:

[0017] The obtaining unit is configured to obtain a plurality of training samples, and each training sample comprises input data, a task type, a correct response, and an incorrect response;

[0018] The determining unit is configured to determine a plurality of task weights corresponding to each task type based on a predicted response obtained by processing each training sample based on the business model, wherein a single task weight is determined based on the correctness of the predicted response corresponding to each training sample of the corresponding task type;

[0019] The reward unit is configured to predict a reward difference of the predicted response corresponding to each training sample relative to the correct response and the incorrect response through a reward model;

[0020] The updating unit is configured to determine a model loss based on each task weight and each reward difference, and to update the business model based on the model loss, wherein the model loss comprises a reinforcement learning loss, and each sample loss is weighted by using the task weight corresponding to each training sample as a weight coefficient, and a single sample loss is determined based on the reward difference.

[0021] According to a third aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program, when the computer program is executed in a computer, the computer program causes the computer to execute the method of the first aspect.

[0022] According to a fourth aspect, a computing device is provided, comprising a memory and a processor, characterized in that the memory stores executable code, and the processor, when executing the executable code, implements the method of the first aspect.

[0023] By the method and device provided by the embodiments of the present specification, in the case of training a multi-task business model based on the principle of reinforcement learning, the reinforcement learning task weight of the sample under the corresponding task can be determined according to the performance of the business model on each task, so as to determine the loss contribution of various task samples in the model loss through the task weight. In this way, in the process of adjusting the model parameters in the direction of minimizing the model loss, the importance of the reward values corresponding to simple tasks and complex tasks can be distinguished according to the weight, which is used to reduce the loss contribution of simple tasks and increase the loss contribution of complex tasks, so that the multi-task business model can more evenly improve the performance on various tasks. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0025] Figure 1 A specific application scenario architecture schematic diagram of the present specification is shown;

[0026] Figure 2 A basic principle schematic diagram of reinforcement learning is shown;

[0027] Figure 3 A flowchart of training a business model based on reinforcement learning according to an embodiment is shown;

[0028] Figure 4 A schematic block diagram of a device for training a business model based on reinforcement learning according to an embodiment is shown. DETAILED DESCRIPTION

[0029] First, some terms that may be used in this paper are explained.

[0030] Large Language Model (LLM) can also be referred to as large model. Large language model is a natural language processing model based on deep learning technology, whose parameter order of magnitude usually reaches tens of billions to hundreds of billions or even higher, with strong language understanding and generation ability. Large language model can adopt Transformer architecture or its variants (such as GPT, BERT, etc.), which uses attention mechanism to realize global modeling of sequence data, and can efficiently handle long-distance dependencies, thus performing well in natural language tasks. Large language model learns the statistical characteristics and semantic relevance of language by pre-training on large-scale corpus, so it has strong generalization ability. The core capabilities of large language model include but are not limited to: understanding context semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage methods usually include direct inference and fine-tuning. In direct inference mode, users guide large language model to generate specific output by designing prompts. The prompt can be a text form task description or instruction to stimulate the semantic understanding and generation ability of large language model. In fine-tuning mode, large language model is further trained on small-scale data sets in specific fields to optimize its performance on specific tasks. The strong generalization ability and flexibility of large language model make it an important tool in the field of artificial intelligence technology, providing efficient and accurate solutions for automated text generation and understanding.

[0031] Domain-Specific Large Language Models: Large language models that are specifically trained and optimized for a particular industry or domain (such as finance, healthcare, law, etc.). These models integrate industry knowledge, business rules, and scenario data to achieve precise application in specific tasks, with stronger professional, compliance, and scenario adaptability features, and lower development costs than general large models.

[0032] Pass@k score: A measure of model accuracy, specifically the probability of at least one correct answer in k attempts, which is an important standard for evaluating model performance.

[0033] The technical solutions provided in the specification will be described below in conjunction with the drawings.

[0034] Figure 1 A specific implementation architecture diagram of the specification is shown. As Figure 1As shown, in the implementation architecture, a server and a client are included. The client can run on a smart terminal held by a user, such as a smartphone, a notebook computer, a smart watch, and the like, which can perform various businesses, such as query business, question and answer business, shopping business, and the like, through interaction with the server. The server can be located in the cloud, a business server, or the like. The server can be deployed with a vertical large model, which is used to provide business support, such as question and answer business support, query business support, shopping business support, and the like, for the corresponding client based on the business request of the client, and feedback the business processing result to the client.

[0035] It can be understood that the vertical large model is a business model obtained by optimizing and fine-tuning a large language model using domain knowledge of a specific domain. The optimization and fine-tuning of the large language model can be performed on the server, or can be completed by other computers or devices. Here, the platform for optimizing and fine-tuning the large language model is uniformly referred to as a computing platform.

[0036] It is worth noting that, Figure 1 The specific implementation architecture is only an example. In practice, the number of clients can be any reasonable number, the server can correspond to a single device or a device cluster, and the business model can be a vertical large model or other business model. In particular, the business model can also be applied to a single machine mode, that is, relative to the server-client model, the business model is deployed on a single device to perform various businesses. The business model discussed in this specification generally has multi-task processing capability, such as query task, planning task, and the like.

[0037] This specification discusses the training process of the business model. Based on the reinforcement learning theory, the business model can be guided to generate responses that better meet professional standards and user expectations through reward signals. Figure 2 The basic principle of reinforcement learning is shown in the schematic diagram. As Figure 2 shown, in the reinforcement learning process, the reward feedback from the environment adjusts the action of the agent, so that the agent continuously adjusts the action to optimize the long-term reward. As Figure 2 shown, assuming that the agent is in the current state S t , through the reward R t of the previous round, the action a t to be performed by the agent can be predicted, through the environment feedback, the state S t after performing the action a t+1 can be obtained, and the reward R t+1 of this round can be obtained, so that the next action is predicted by the agent through the reward R t+1 of this round, and so on. In the initial state, the corresponding initial action can be directly predicted.

[0038] It can be understood that in the process of training the business model based on reinforcement learning, the business model corresponds to an agent, the state of the business model is described by the specific value of the parameter, the parameter update rule of the business model is described by the policy, in a single update process, based on the current parameter state of the business model, the specific value in the parameter update rule is determined according to the reward, forming a parameter update action, and the next state of the business model is formed after updating the parameters. The reward is usually determined by the reward model.

[0039] The conventional reinforcement learning method often uses the same learning strategy for all tasks in the training process, and adjusts the reward function to affect the model behavior. This way may have the following problems in the process of training the multi-task business model: reward sparsity, complex tasks often have sparse reward signals, making it difficult to provide effective learning guidance; reward design difficulty, designing appropriate reward functions for tasks of different difficulty requires a lot of professional knowledge and experience; training instability, directly modifying the reward may cause the training process to be unstable, affecting model convergence.

[0040] In addition, the conventional reinforcement learning method also attempts to artificially determine the task difficulty, however, the artificially set task difficulty often has subjectivity, ignores the objectivity and dynamics of task difficulty evaluation, cannot dynamically adjust according to the current ability of the model, and may result in low training efficiency.

[0041] Therefore, the present specification provides a technical solution for training a business model based on reinforcement learning. The training samples correspond to input data, task types, correct responses, and error responses. The correct response is the expected output of the business model, which can be understood as the label data of the positive sample together with the input data. The error response is the unexpected output of the business model, which can be understood as the label data of the negative sample together with the input data. The process of training the business model based on reinforcement learning can include multiple update cycles. In a single update cycle, multiple training samples are obtained, and the respective task weights corresponding to each task type are determined based on the processing of the training samples corresponding to each task type by the business model. Then, the reward difference of the predicted response corresponding to each training sample relative to the correct response and the error response is predicted by the reward model, and the model loss is determined based on the respective task weights and the respective reward differences. The pending parameters in the business model are updated using the model loss.

[0042] In the above process, the task weight can be determined based on the correctness of the predicted response of the corresponding task type obtained by the business model processing the training sample, and the model loss is obtained by weighting each sample loss determined based on the reward difference using the task weight of each training sample as a weight coefficient. In this way, the task state can be dynamically and objectively evaluated through the processing of the current batch of training samples by the business model, and the task weight can be used to measure it, avoiding the introduction of subjective factors and increasing effectiveness. Introducing the task difficulty weight at the level of the loss function instead of directly modifying the reward function can make the training of the business model based on reinforcement learning more stable and effective. At the same time, each task weight is dynamically determined through the current batch of training samples, maintaining objectivity and dynamics, and is more consistent with actual data rules. Thus, the training efficiency and model stability of the business model based on reinforcement learning can be improved, the computing resources can be saved, and the business model obtained can provide better business support and improve user experience.

[0043] The technical concept of training a business model based on reinforcement learning is described in detail below with reference to the accompanying drawings.

[0044] Figure 3 A flow of training a business model based on reinforcement learning is shown according to one embodiment. The execution subject of the flow can be any computer, device or server with certain computing power. More specifically, as shown in the figure, Figure 1 The computing platform is shown. The business model here can be various neural networks capable of processing multiple tasks, such as natural language processing models such as large language models, etc. The tasks supported by the business model may, for example, include multiple items among classification tasks, prediction tasks, question and answer tasks, query tasks, etc. Generally, the training of the business model (in the case of the business model being a large language model, the training here can refer to fine-tuning) often iteratively updates the pending parameters in the model through multiple update cycles, Figure 3 An example of any one of the update cycles is described.

[0045] As shown in the figure, Figure 3As shown, the process of training the business model based on reinforcement learning can include the following steps: step 301, obtaining a plurality of training samples, each training sample including input data, task type, correct response and error response; step 302, determining the respective task weights corresponding to each task type based on the predicted responses obtained by processing each training sample based on the business model, wherein a single task weight is determined based on the correctness of the predicted responses corresponding to each training sample of the corresponding task type; step 303, predicting the reward difference of the predicted response corresponding to each training sample relative to the correct response and the error response through the reward model; step 304, determining the model loss based on each task weight and each reward difference, which is used to update the business model, the model loss including a reinforcement learning loss determined by weighting each sample loss using the task weight corresponding to each training sample as a weight coefficient, and a single sample loss being determined based on the reward difference.

[0046] First, in step 301, a plurality of training samples are obtained.

[0047] It can be understood that the training samples are various data used to train the business model. Here, each training sample can at least include input data and output response supervision data.

[0048] The input data can be various data provided to the business model for processing, such as feature data, attribute data, natural language description information (such as user questions, search terms Query, etc.), and the like. In the case of a large language model as the business model, the input data can be a prompt information (Prompt) input into the large language model, which can include but is not limited to search terms, task instructions, user questions, and other natural language description information. As a specific example, such as "Is it legal for a landlord to ask tenants to move out before the lease expires?" Optionally, the input data can include at least one modality data such as text, audio, image, etc.

[0049] Under the technical concept of the present specification, the output response data can include correct responses and error responses. The correct response and the error response are preferred responses determined according to the user's expected answer to the input data, which can be determined manually, generated by a large language model with better performance, or determined through user feedback. The correct response is usually a response data that the business model can accurately identify the user's intention in the input data and give the answer expected by the user, while the error response can be a response data including at least one of the following: unable to identify the user's intention in the input data, unable to give the answer expected by the user, logical error in the answer, common sense error, not in accordance with laws and regulations, irrelevant to the input data, etc.

[0050] Taking the input data "Is it legal for a landlord to ask tenants to move out before the lease expires?" as an example, a correct response is "According to Article 730 of the Civil Code, the landlord needs to bear breach of contract liability when asking tenants to move out before the lease expires. You can require the landlord to continue to perform the contract or claim compensation for breach of contract.", and an incorrect response is "The landlord can ask tenants to move out at any time, and the law does not protect the rights and interests of tenants. You can move out directly."

[0051] In order to determine the importance of the task according to the type of the task during the training process, each training sample can also correspond to a task type. The task type is the category of the task to be completed by the business model in the current training sample, for example, the task types classified according to the business scenario can include at least one of the following: question and answer task, search task, generation task, classification task, etc. The task type can be manually annotated, or it can be the type during the training sample collection process, or it can be determined in other reasonable ways, which will not be described here. In the training sample, the task type can be represented by a type name, or by a corresponding symbol or number (such as 1, 2, 3 corresponding to three task types), which is not limited here.

[0052] In addition, the way to obtain multiple training samples here can be random, according to the storage order, according to the proportion of the task type (such as equal proportion 1:1:1……), or according to other reasonable ways, which will not be limited in this specification.

[0053] Then, via step 302, the prediction response obtained by processing each training sample by the business model is used to determine the task weight corresponding to each task type.

[0054] The processing of the training sample by the business model is actually the processing of the input data. The business model processes the input data through its internal structure and corresponding business logic to obtain the corresponding output result, which is denoted as the prediction response. The prediction response can be a classification result, a question answer, a search result, a generation result, etc.

[0055] It can be understood that the correct response in the output response data can be used as the expected output of the prediction response. Generally, for a single task type training sample, the closer the prediction response is to the correct response, the better the data processing performance of the business model in the current state for that task type (such as a simpler query task, etc.), which can occupy a smaller component in the model pending parameter adjustment. On the contrary, the farther the prediction response is from the correct response, the weaker the data processing performance of the business model in the current state for that task type (such as a more complex planning task, etc.), which can occupy a larger component in the model pending parameter adjustment, so that the update of the business model is balanced in various task types.

[0056] The component of a single task type in the model pending parameter adjustment can be quantified by a task weight. According to the above principle, a single task weight corresponding to a single task type can be determined based on the correctness of the predicted response of the business model to the training sample processing of the corresponding task type.

[0057] The judgment on the correctness of the predicted response can be performed in various reasonable ways. In some optional implementation manners, the judgment on the correctness of the predicted response can be related to a specific task type. Assuming that any training sample is denoted as a first sample, the corresponding predicted response is denoted as a first predicted response, the correct response is denoted as a second predicted response, and the task type is denoted as a first task type.

[0058] In one embodiment, the first task type is a classification task, and whether the classification category corresponding to the first predicted response is the same as the classification category corresponding to the first correct response is detected, and whether the first predicted response is correct is determined according to the detection result. Generally, in the case where the classification categories corresponding to the two are the same (for example, both are category C), it is determined that the first predicted response is correct.

[0059] In another embodiment, the first task type is a question and answer task, and the first predicted response and the first correct response are texts. Whether the first predicted response is correct is determined by vector similarity judgment between embedding vectors of the first predicted response and the first correct response. Generally, in the case where the vector similarity is greater than a predetermined similarity threshold, it can be determined that the first predicted response is correct. The vector similarity of the embedding vector is, for example, at least one of the following: cosine similarity, Jaccard similarity, Pearson correlation coefficient, and the like. The vector similarity can also be measured by the distance (which can be denoted as vector distance) between the points represented by the vectors, for example, at least one of the following: Euclidean distance, Hamming distance, KL divergence, and the like. The vector similarity is generally negatively correlated with the distance between the points represented by the vectors.

[0060] In the case where the first task type is an image generation task, the first predicted response and the first correct response are images, and whether the first predicted response is correct is determined by pixel-by-pixel comparison of the first predicted response and the first correct response or feature vector similarity. The feature vector can be extracted by an image processing network such as a convolutional neural network. Generally, in the case where the feature vector similarity is greater than a similarity or the proportion of consistent pixels is greater than a predetermined proportion threshold, it can be determined that the first predicted response is correct.

[0061] In the case that the first task type is a code generation task, the first predicted response and the first correct response are in the form of code, and whether the first predicted response is correct is determined by comparing whether the respective running results of the first predicted response and the first correct response are consistent. Generally, in the case that the running results are consistent, it can be determined that the first predicted response is correct.

[0062] In yet some optional implementations, the correctness of the predicted response can be determined by a pass@k score. Specifically, the corresponding input data can be processed by the business model for k times, and the ratio of the number of times that the output predicted response is correct (determined in any of the foregoing manners) to k is obtained. For example, the input data of a certain training sample A (denoted as a second sample, which can be the same as or different from the first sample) is processed for k times to obtain k predicted responses (denoted as k second predicted responses), and according to the vector correlation degree, it is determined that a of the k predicted responses are correct, and then the pass@k score is a / k. The pass@k score itself represents the correctness of the predicted response. Alternatively, the correctness of the predicted response of the business model for the single training sample A can also be determined according to the comparison between the pass@k score and a predetermined score value.

[0063] In other implementations, the correctness of the predicted response can also be determined by various other reasonable manners, which will not be described here. The various implementations described above can also be freely superimposed or combined without conflict.

[0064] According to the correctness of the predicted response of the business model for each training sample, the respective task weights corresponding to each task type can be further determined.

[0065] Generally, the task weight can be positively correlated with the correctness rate s. The correctness rate is simply understood as the ratio of the case that the predicted response is correct to the total number of samples for a single task type in the current batch of samples. For example, in an embodiment, the correctness rate is the ratio of the number of samples for which the predicted response is correct to the total number of samples for the single task type. In practice, the correctness rate can also be determined in other manners, for example, in an optional embodiment, for a single task type, the mean of the pass@k scores of the respective training samples can be used as the correctness rate corresponding to the task type. Assuming that the number of training samples corresponding to the single task type is n, the sum of the pass@k scores of the n training samples is divided by n to obtain the result as the correctness rate corresponding to the task type.

[0066] After determining the prediction accuracy of the business model on a single task type based on the correctness of each prediction response corresponding to each training sample, the current weight corresponding to the single task type can be determined according to the prediction accuracy, and the corresponding single task weight can be determined based on the current weight. The following describes the determination manner of the corresponding current weight and task weight by taking a single task type t as an example.

[0067] In some optional implementations, the current weight of the single task type t is, for example, the difference (which can also be understood as the error rate) between 1 and the prediction accuracy, denoted as 1-s, or a multiple of 1-s, denoted as the current weight w raw = a (1-s).

[0068] In other optional implementations, a reference model can also be introduced, and the performance (still taking the accuracy rate as an example) of the business model under the current parameter strategy on the same task is compared with the performance of the reference model, so as to more objectively evaluate the relative strength of the current strategy, and the current weight is determined according to the comparison result of the performance of the reference model. Here, the reference model can be a model with different hyperparameter configurations set at different training stages, or an industry standard model, etc. The performance of the reference model is usually better than or not worse than the business model being trained. In order to guide the training of the business model, for a single task type, the corresponding current weight can be positively correlated with the difference between the prediction accuracy (denoted as s ref ) of the reference model and the prediction accuracy s of the current business model, such as w raw = β (s ref -s), β is a preset positive number. The determination manner of the prediction accuracy of the reference model and the prediction accuracy of the current business model is consistent, and will not be described here. In order to avoid that the prediction accuracy of the reference model is lower than the current business model, a lower limit 0 can be set for (s ref -s), such as w raw = β × max [0, (s ref -s)], max represents the larger value of multiple numbers.

[0069] In practice, the number of reference models can be one or multiple. In the case where the number of reference models is multiple, the prediction accuracy s ref of the reference model can be replaced by the mean of the prediction accuracies of the reference models, such as

[0070] In yet other optional implementations, a single current weight can be a weighted sum of the following two items: 1 is the difference of the prediction accuracy, and the difference between the prediction accuracy of the reference model and the prediction accuracy s of the current business model. Denoted as: w raw = a (1-s) + β (s ref-s), or In some optional manners, to ensure that all tasks (including simple tasks) obtain the minimum attention, a minimum weight γ (a preset small positive number, such as 0.001, etc.) can be set for the single current weight, and used as an additional term of the weighted value, denoted as: w raw = a(1-s) + β(s ref -s) + γ, or

[0071] Based on the current weight w raw , the task weight w(t) of the current update cycle can be determined.

[0072] In some optional implementation manners, the current weight or the normalized value thereof based on the current weight of each task type can be directly used as the task weight of the current update cycle. The normalization manner is, for example: wherein T is the task type set of the current batch of training samples, and |T| is the number of task types in the task type set

[0073] In another optional implementation manner, to control the inertia of weight update and avoid drastic fluctuations in the training process, a smoothing mechanism can be used to update the current weight. In the case where the current task type is first appeared, the current task weight can be directly determined as w raw , denoted as w(t) = w prev = w raw . In the case where the current task type is not first appeared, a sliding average mechanism can be used to determine the current task weight, for example: w(t) = w prev = ρw prev +(1-ρ)w raw . ρ is a preset hyperparameter, controlling the smoothing degree, and w prev is the task weight determined in the last time.

[0074] In a possible design, the task weight determined by the above manners can be normalized for each task type, and the normalized task weight can be used as wherein T is the task type set of the current batch of training samples, and |T| is the number of task types in the task type set.

[0075] In more implementation manners, the task weight corresponding to each task type can also be determined by more manners, which will not be described herein.

[0076] To make the determination manner of the task weight more clear, a specific example of determining the task weight of the task type t in the current update cycle is given as follows.

[0077] Assume that among the multiple training samples collected in the current update cycle, the number of training samples for task type t is n. First, for each of the n training samples, we can process it k times to obtain k predicted responses, resulting in a corresponding pass@k score. Then, we calculate the average of the n pass@k scores to obtain the prediction accuracy s of the current business model for task type t. On the other hand, through m reference models, we can obtain m corresponding accuracy rates s1, s2…s. m Let j be a single reference model in the reference model set, and s be the corresponding accuracy. j The average accuracy of the reference model can be obtained as follows: Thus, the initial task weight of the current period (i.e., the current weight) can be obtained, denoted as: w raw (t)=α(1-s)+β×max[0,(s ... ref -s)]+γ.

[0078] When task type t appears for the first time, the initial task weight w for the current period is set. raw (t) is denoted as the final task weight for the current period, such as w. final (t)=w raw (t). When task type t is not appearing for the first time, the task weight w determined in the current period is smoothed by exponential average. final (t)=ρw prev (t)+(1-ρ)w raw (t). Where, w prev (t) represents the task weight of task type t after the previous update. Without task weight normalization, we have: w prev (t) and w final (t) is consistent. Under the condition of task weight normalization, we can have:

[0079] In this specific example, the Pass@k scoring metric is used in conjunction with multi-model comparison to achieve objective and dynamic task difficulty assessment, avoiding the subjectivity and static issues of manual annotation. Furthermore, an exponential smoothing mechanism is introduced to update weights exponentially, preventing training oscillations caused by drastic changes in task weights and ensuring the smoothness of the learning process.

[0080] Next, based on step 303, the reward difference between the predicted response and the correct and incorrect responses for each training sample is predicted using the reward model.

[0081] Here, the reward model can be a pre-trained model that can process the input text pair and give a corresponding reward score. It can be pre-trained supervised by samples containing text pairs and labels of reward scores, which will not be described here.

[0082] For any training sample, the correct response is denoted as y w , the error response is denoted as y l , and the predicted response is denoted as y, the predicted response y and the correct response y w can form a data pair (y, y w ), and the predicted response y and the error response y l can form a text pair (y, y l ). Inputting the data pair (y, y w ) into the reward model can obtain the reward score of the predicted response relative to the correct response r w = r (y, y w ), and inputting the data pair (y, y l ) into the reward model can obtain the reward score of the predicted response relative to the error response r l = r (y, y l ). Then the reward difference of the predicted response relative to the correct response and the error response of the training sample can be denoted as: r w -r l = r (y, y w )- r (y, y l ).

[0083] In this way, the reward difference corresponding to each training sample in the current batch can be determined.

[0084] Further, in step 304, based on the task weights and the reward differences, the model loss is determined for updating the business model.

[0085] It can be understood that under the technical concept of the present specification, the correct response and the error response are used as the supervision signal of reinforcement learning together. The size of the supervision signal is described by the aforementioned reward difference. The initial loss caused by a single training sample can be determined based on the reward difference, such as negative correlation with the corresponding reward difference r w -r l , cross entropy log(r w -r l ), etc. In a specific example, the reward difference can be further processed, such as mapping the reward difference r w -r l to the interval (-1, 1) using the sigmiod function. At this time, the initial loss caused by a single training sample is related to logσ(r w -r l) negative correlation, wherein σ is a sigmiod function.

[0086] Since the task weight describes the importance of the initial loss (denoted as sample loss) of the training sample under the corresponding task type, the task weight of each task type can be used as a weight coefficient to weight each sample loss to obtain

[0087] For example, the reinforcement learning loss is wherein the summation is performed for each training sample in the current batch, is the task weight of the task type corresponding to the single training sample, and E represents the expected direction, such as maximization.

[0088] Here, the reward function is not directly modified, but a weight is introduced at the level of the loss function to ensure that the original semantics of the reward signal are not destroyed and the model can correctly understand the essential requirements of the task. In the case of determining the reinforcement learning loss, the original output of the reward model is used to maintain the semantic integrity of the reward model output, ensuring the reliability and consistency of the reward signal. In combination with the determination of the reinforcement learning loss by the weighting of the task weight, the key learning of the difficult task can be realized. By applying the weight in the loss function rather than the reward function, the influence of the reward scale change on the policy gradient estimation is avoided, and the stability and convergence of the training process are ensured.

[0089] The model loss at least includes the reinforcement learning loss. In an optional implementation, the model loss can also include a label supervision loss. The label here is usually a correct response. The label supervision loss can be determined by comparing the predicted corresponding y with y w , which can be negatively correlated with the vector correlation of the semantic embedding vectors of the two, and can be measured in various reasonable ways such as cross-entropy loss, KL divergence, etc., which will not be repeated here. It is worth noting that the label supervision loss can also weight the original loss of each training sample by the task weight.

[0090] Through the model loss, the gradient of each pending parameter in the business model can be determined, so that the value of the pending parameter is adjusted in a manner such as gradient descent. Taking the gradient descent method as an example, the adjustment process of the pending parameter θ is, for example: θ = θ - λΔθ, wherein λ is a pre-set update step, and Δθ is the partial derivative of the model loss with respect to the parameter θ, i.e. the gradient.

[0091] The model parameter update can also be optimized by methods such as GRPO algorithm and PPO algorithm, which will not be repeated here.

[0092] Reviewing the above process, in the scheme provided in the present specification for training a business model based on reinforcement learning, in the case of training a multi-task business model based on the principle of reinforcement learning, the reinforcement learning task weight of the sample under the corresponding task can be determined according to the performance of the business model on each task, so as to determine the contribution of various task samples in the model loss through the task weight. In this way, in the process of adjusting the model parameters in the direction of minimizing the model loss, the importance of the reward values corresponding to simple tasks and complex tasks can be distinguished according to the weight, which is used to reduce the loss contribution of simple tasks and increase the loss contribution of complex tasks, so that the multi-task business model can more evenly improve the performance on various tasks.

[0093] Through experiments, the inference model trained by reinforcement learning with task weight weighting in the present specification has achieved significant performance improvement on complex vertical field tasks. The optimized business model has stronger task adaptation ability, especially in complex reasoning, multi-step analysis and other high difficulty tasks. The business model not only improves on difficult tasks, but also maintains good performance on simple tasks, achieving balanced development of overall performance. Specific data set comparison, due to the task weight balancing the learning of tasks of different difficulty, the success rate of the business model on complex professional reasoning, multi-step analysis and other high difficulty tasks can be improved by 15-20%, the performance of the business model on various vertical field tasks is more balanced, the performance variance can be reduced by 25-35%, the original semantics of the reward function is maintained, the training process is easier to understand and debug, and the problem positioning efficiency can be improved by more than 30%.

[0094] According to another aspect of the embodiment, an apparatus for training a business model based on reinforcement learning is also provided. The apparatus can be provided in any computer, device or server with certain computing power, for example, Figure 1 the computing platform in Figure 4 An apparatus 400 for generating images from text according to an embodiment of the present specification is shown.

[0095] As Figure 4 shown, the apparatus 400 for training a business model based on reinforcement learning can include an acquisition unit 401, a determination unit 402, a reward unit 403 and an update unit 404. The apparatus 400 can train the business model based on reinforcement learning through multiple update cycles.

[0096] In a single update cycle: the acquisition unit 401 can be configured to acquire a plurality of training samples, and each training sample includes input data, a task type, a correct response, and an incorrect response; the determination unit 402 can be configured to determine a prediction response of each training sample based on the business model, and determine a task weight corresponding to each task type, wherein the single task weight is determined based on the correctness of the prediction response corresponding to each training sample of the corresponding task type; the reward unit 403 can be configured to predict a reward difference of the prediction response corresponding to each training sample relative to the correct response and the incorrect response through the reward model; and the update unit 404 can be configured to determine a model loss based on each task weight and each reward difference, and use the model loss to update the business model. The model loss can include the following reinforcement learning loss: each sample loss is weighted by using the task weight corresponding to each training sample as a weight coefficient, and a single sample loss is determined based on the reward difference.

[0097] It should be noted that, Figure 4 The apparatus 400 shown corresponds to the method described, Figure 3 The description of the method embodiments is equally applicable to the apparatus 400, and will not be repeated here. Figure 2 The description of the method embodiments is equally applicable to the apparatus 400, and will not be repeated here.

[0098] According to another aspect of the embodiments, a computer readable storage medium is also provided, which stores a computer program. When the computer program is executed in a computer, the computer executes the method described in combination with the foregoing Figure 3 and the like.

[0099] According to another aspect of the embodiments, a computer readable storage medium is also provided, which stores a computer program. When the computer program is executed in a computer, the computer executes the method described in combination with the foregoing Figure 3 and the like.

[0100] Those skilled in the art should be aware that, in one or more examples described above, the functions described in the embodiments of the present specification can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, these functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0101] The above specific embodiments described further detail the purpose, technical solutions and beneficial effects of the technical concept of the present specification. It should be understood that the above description is only a specific embodiment of the technical concept of the present specification, and does not limit the protection scope of the technical concept of the present specification. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the embodiments of the present specification shall be included in the protection scope of the technical concept of the present specification.

Claims

1. A method for training a business model based on reinforcement learning, the method comprising a plurality of update cycles, in a single update cycle: obtaining a plurality of training samples, each training sample comprising input data, a task type, a correct response, and an incorrect response; processing each training sample based on the business model to obtain a predicted response, and determining a task weight corresponding to each task type, wherein a single task weight is determined based on the correctness of the predicted response corresponding to each training sample of the corresponding task type; predicting a reward difference of the predicted response corresponding to each training sample relative to the correct response and the incorrect response through a reward model; determining a model loss based on each task weight and each reward difference, the model loss being used to update the business model, the model loss comprising a reinforcement learning loss obtained by weighting each sample loss using the task weight corresponding to each training sample as a weight coefficient, a single sample loss being determined based on the reward difference.

2. The method of claim 1, wherein, The business model is a large language model, and the input data is prompt information of the large language model, comprising at least one of the following: text, audio, and image.

3. The method of claim 1, wherein, The plurality of training samples comprises a first sample, the first sample corresponding to a first predicted response, a first correct response, and a first task type, the correctness of the first predicted response comprising: in the case of a classification task of the first task type, detecting whether the classification category corresponding to the first predicted response is the same as the classification category corresponding to the first correct response, and determining whether the first predicted response is correct according to the detection result; in the case of a question and answer task of the first task type, the first predicted response and the first correct response are texts, and whether the first predicted response is correct is determined by the similarity of the embedding vectors of the first predicted response and the first correct response; in the case of an image generation task of the first task type, the first predicted response and the first correct response are images, and whether the first predicted response is correct is determined by pixel-by-pixel comparison or feature vector similarity of the first predicted response and the first correct response; in the case of a code generation task of the first task type, the first predicted response and the first correct response are in code form, and whether the first predicted response is correct is determined by comparing whether the running results corresponding to the first predicted response and the first correct response are consistent.

4. The method of claim 1, wherein, The plurality of training samples comprises a second sample, the second sample corresponding to second input data, and the correctness of the predicted response corresponding to the second sample being determined by the following method: processing the second data k times through the business model to obtain k second predicted responses; determining the correctness of the predicted response corresponding to the second sample according to a pass@k score of at least one correct second predicted response in the k second predicted responses.

5. The method of claim 1, wherein, A single task weight corresponding to a single task type is determined by the following method: determining a prediction accuracy of the business model on the single task type based on the correctness of each predicted response corresponding to each training sample; determining a current weight corresponding to the single task type according to the prediction accuracy; determining the single task weight based on the current weight.

6. The method of claim 5, wherein, For a single task type, the prediction accuracy is one of the following: The ratio of the number of training samples corresponding to the single task type and correctly predicted to the total number of training samples corresponding to the single task type; The average Pass@k score of each training sample corresponding to the single task type.

7. The method of claim 5, wherein, The current weight corresponding to the single task type is positively correlated with the prediction error rate of the single task type, and the prediction error rate is the difference between 1 and the prediction accuracy of the business model on the single task type.

8. The method of claim 7, wherein, The current weight corresponding to the single task type is also positively correlated with the difference between the prediction accuracy of the reference model on the single task type and the prediction accuracy of the business model on the single task type, and 0 is taken in the case that the difference is less than 0. The prediction accuracy of the reference model on the single task type is the average prediction accuracy of each reference model on the single task type.

9. The method of claim 5, wherein, Determining the single task weight based on the current weight, comprising: In the case that the single task type first appears, taking the current weight or a normalized value of the current weight based on the current weight of each task type as the single task weight; In the case that the single task type first appears, taking a sliding average value obtained by using the current weight and the task weight determined in the last time or a normalized value of the sliding average value of the weight of each task type as the single task weight.

10. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-9. The memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-9.

Images