Large model adjusting method and device, computer equipment and readable storage medium

Abstract

The invention relates to the technical field of computers, in particular to a large model adjusting method and device, computer equipment and a computer readable storage medium. The large model adjustment method comprises the steps of obtaining an initial large model, wherein the initial large model is used for processing a question and answer task; based on the initial large model, a plurality of bypass branches related to the initial large model are generated, the bypass branches at least comprise an award model, an evaluation model and a strategy model, the award model is used for evaluating the quality of answers output by the initial large model, and the evaluation model is used for evaluating whether the answers output by the initial large model are accurate or not; the strategy model is used for adjusting an answer path output by the initial large model; and obtaining a preset data set, respectively substituting the preset data set into the reward model, the approval model and / or the strategy model, and adjusting parameters of bypass branches of the initial large model to obtain a target large model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a large model adjustment method, device, computer equipment and computer-readable storage medium. Background Art

[0002] Large models perform well in general tasks, few-sample learning, generation tasks, and other tasks, and have strong learning capabilities and flexibility. However, they have some problems with task accuracy in specific scenarios, computing resource consumption, and dealing with ethical and bias issues. The purpose of model fine-tuning is to solve these problems, reduce the demand for computing and storage resources for large models, reduce the bias or unsafe content generated by the model, and improve the performance and accuracy of the model on specific tasks.

[0003] However, due to the large number of parameters in large models, the computational and storage requirements are too high when fine-tuning large models, and the computational and time costs of model fine-tuning are both high, which reduces the efficiency of fine-tuning large models. Summary of the invention

[0004] Based on this, it is necessary to provide a large model adjustment method, device, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of large model fine-tuning in response to the above technical problems.

[0005] In a first aspect, the present application provides a large model adjustment method, comprising:

[0006] Obtaining an initial large model, where the initial large model is used to process a question-answering task;

[0007] Based on the initial large model, a plurality of bypass branches related to the initial large model are generated, wherein the bypass branches at least include a reward model, a criticism model and a strategy model, wherein the reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model;

[0008] A preset data set is obtained, and the preset data set is respectively introduced into the reward model, the criticism model and / or the strategy model, and the parameters of the bypass branch of the initial large model are adjusted to obtain the target large model.

[0009] In one embodiment, the obtaining of a preset data set, bringing the preset data set into the reward model, the criticism model and / or the policy model respectively, and adjusting the parameters of the bypass branch of the initial large model to obtain the target large model includes:

[0010] A preference data set is obtained, and the preference data set is brought into the reward model to update the parameters of the bypass branch corresponding to the reward model.

[0011] In one embodiment, the obtaining of a preset data set, bringing the preset data set into the reward model, the criticism model and / or the policy model respectively, and adjusting the parameters of the bypass branch of the initial large model to obtain the target large model includes:

[0012] Obtain a prompt data set, and bring the prompt data set into the strategy model, the reward model, the criticism model and the initial large model respectively to obtain a strategy value, a reward value, a benefit value and a question-and-answer value;

[0013] Calculating a policy model loss function and a critical model loss function based on the policy value, the reward value, and the benefit value;

[0014] Parameters of bypass branches corresponding to the policy model and the critical model are updated based on the policy model loss and the critical model loss.

[0015] In one embodiment, updating the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss includes:

[0016] Obtaining the strategy value and the question-answer value;

[0017] Calculating a difference value between the strategy value and the question-answer value;

[0018] Based on the question-answer value and the difference value, as well as the policy model loss, the parameters of the bypass branch corresponding to the policy model are updated.

[0019] In one embodiment, updating the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss includes:

[0020] Obtaining the policy model loss and the critic model loss;

[0021] Based on the policy model loss and the critical model loss, the parameters of the bypass branches corresponding to the policy model and the critical model are updated until the policy model loss and the critical model loss converge to obtain the target large model.

[0022] In one embodiment, updating the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss includes:

[0023] The parameters of the corresponding bypass branch of the reward model are not updated, and the parameters of the initial large model are not updated.

[0024] In one embodiment, obtaining the initial large model includes:

[0025] Get the original large model and question-answering dataset;

[0026] An initial large model is generated based on the question-answer data set and the original large model.

[0027] In a second aspect, the present application also provides a large model adjustment device, including:

[0028] An acquisition module, used to acquire an initial large model, where the initial large model is used to process the question-answering task;

[0029] A generation module, configured to generate a plurality of bypass branches related to the initial large model based on the initial large model, wherein the bypass branches at least include a reward model, a criticism model, and a strategy model, wherein the reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model;

[0030] The adjustment module is used to obtain a preset data set, bring the preset data set into the reward model, the criticism model and / or the strategy model respectively, and adjust the parameters of the bypass branch of the initial large model to obtain the target large model.

[0031] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0032] Obtaining an initial large model, where the initial large model is used to process a question-answering task;

[0033] Based on the initial large model, a plurality of bypass branches related to the initial large model are generated, wherein the bypass branches at least include a reward model, a criticism model and a strategy model, wherein the reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model;

[0034] A preset data set is obtained, and the preset data set is respectively introduced into the reward model, the criticism model and / or the strategy model, and the parameters of the bypass branch of the initial large model are adjusted to obtain the target large model.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0036] Obtaining an initial large model, where the initial large model is used to process a question-answering task;

[0037] Based on the initial large model, a plurality of bypass branches related to the initial large model are generated, wherein the bypass branches at least include a reward model, a criticism model and a strategy model, wherein the reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model;

[0038] A preset data set is obtained, and the preset data set is respectively introduced into the reward model, the criticism model and / or the strategy model, and the parameters of the bypass branch of the initial large model are adjusted to obtain the target large model.

[0039] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0040] Obtaining an initial large model, where the initial large model is used to process a question-answering task;

[0041] Based on the initial large model, a plurality of bypass branches related to the initial large model are generated, wherein the bypass branches at least include a reward model, a criticism model and a strategy model, wherein the reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model;

[0042] A preset data set is obtained, and the preset data set is respectively introduced into the reward model, the criticism model and / or the strategy model, and the parameters of the bypass branch of the initial large model are adjusted to obtain the target large model.

[0043] The above-mentioned large model adjustment method, device, computer equipment, computer-readable storage medium and computer program product generate multiple bypass branches (Lora branches) related to the initial large model, and the bypass branches are used to represent the reward model, strategy model and criticism model required in the fine-tuning stage of the initial large model, thereby reducing the complexity of calculation and improving the fine-tuning efficiency of the initial large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a schematic diagram of the adjustment method of the large model provided in the first embodiment;

[0046] Figure 2 A schematic diagram of a method for adjusting a large model provided in a second embodiment;

[0047] Figure 3 A schematic diagram of a method for adjusting a large model provided in a third embodiment;

[0048] Figure 4 A schematic diagram of a large model adjustment method provided in a fourth embodiment;

[0049] Figure 5 A schematic diagram of a method for adjusting a large model provided in a fifth embodiment;

[0050] Figure 6 A schematic diagram of a large model adjustment method provided in a sixth embodiment;

[0051] Figure 7 A schematic diagram of a method for adjusting a large model provided in a seventh embodiment;

[0052] Figure 8 A schematic diagram of a large model adjustment method provided in an eighth embodiment;

[0053] Fig. 9 A structural block diagram of a large model adjustment device provided in one embodiment;

[0054] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] As mentioned in the background technology, due to the large number of parameters of large models, when fine-tuning large models, the computing and storage requirements are too high, and the computing and time costs of model fine-tuning are both large. Specifically, in the related technology, the human feedback reinforcement learning (Reinforcement Learning from Human Feedback, RLHF) model is often used to improve the quality and relevance of conversations, reduce bias and harmful content, and stabilize the training process and improve efficiency. Among them, the reinforcement learning algorithm (Proximal Policy Optimization, PPO) algorithm is a commonly used reinforcement learning algorithm in RLHF. Since the number of parameters of the large language model to be fine-tuned is large, when the PPO stage loads the above-mentioned multiple models at the same time, it will cause the model calculation and storage requirements to be too high, and the computing cost and time cost of model fine-tuning will increase significantly, which increases the difficulty of model fine-tuning.

[0057] Based on this, the present application provides a method for adjusting a large model. It can be understood that the method for adjusting a large model provided by multiple embodiments of the present application can be applied to a terminal, a server, and a system in which the terminal and the server interact. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and Internet of Things devices. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0058] In one embodiment, Figure 1 As shown, a large model adjustment method is provided. This embodiment uses the method applied to a terminal as an example for illustration, and it is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server.

[0059] In this embodiment, the method includes the following steps:

[0060] Step S100: Obtain an initial large model, which is used to process question-answering tasks.

[0061] In step S100, the initial large model may be capable of processing basic question-answering tasks. As an example, the initial large model may be a pre-trained general language model or a model trained for a specific field or task.

[0062] Step S200: Based on the initial large model, multiple bypass branches related to the initial large model are generated, and the bypass branches include at least a reward model, a criticism model and a strategy model. The reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model.

[0063] In step S200, three bypass branches are generated based on the initial large model. The bypass branches may include at least a reward model, a criticism model, and a strategy model. Of course, this embodiment does not limit the specific number and form of the bypass branches. Specifically, the bypass branch may be a low-rank adaptation (LoRa) branch.

[0064] Specifically, the reward model can be used to evaluate the quality of the answers output by the initial large model. For example, the reward model can score the answers according to preset evaluation criteria (such as the completeness, accuracy, relevance, etc. of the answers). The scores can be used to adjust the reward model parameters later. Specifically, the reward model provides a reward signal to the generation model so that the initial large model is close to the optimal strategy or generates content that is more in line with human preferences during the training process. The critical model can be used to evaluate the accuracy of the answers output by the initial large model. The critical model can analyze the answers to determine whether they solve the problem. The critical model helps to ensure that the content of the answers output by the initial large model is accurate. Of course, the critical model generates feedback information based on the results of the critical analysis and makes improvements. The policy model can select the model of the optimal behavior based on the current state or input, or the policy model can select the appropriate answer path based on the current state. The policy model helps to increase the adaptability of the initial large model.

[0065] Step S300: Obtain a preset data set, bring the preset data set into the reward model, the criticism model and / or the strategy model respectively, and adjust the parameters of the bypass branch of the initial large model to obtain the target large model.

[0066] In step S300, as an example, the preset data set may include a preference data set, a prompt data set, etc. Of course, specifically, the preset data set may include various question-and-answer scenarios and corresponding answers. Afterwards, in a first possible example, the preset data set is input into the reward model, and the parameters of the initial large model are adjusted according to the output of the reward model (i.e., the quality score of the answer) to improve the quality of the answer. In a second possible example, the preset data set may be input into the critical model, and the parameters of the initial large model may be adjusted according to the output of the critical model (i.e., the accuracy evaluation of the answer) to ensure that the content of the answer is accurate. In a third possible example, the preset data set may be input into the strategy model, and the parameters of the initial large model may be adjusted according to the output of the strategy model (i.e., the path selection of the answer) to make the model more flexible and adaptable.

[0067] In this embodiment, multiple bypass branches (lora branches) related to the initial large model are generated, and the bypass branches are used to represent the reward model, strategy model and criticism model required for the fine-tuning stage of the initial large model, thereby reducing the complexity of the calculation and improving the fine-tuning efficiency of the initial large model. Moreover, the specific form of the bypass branch can also be added or deleted as needed, so that this embodiment can adapt to different tasks and scenarios to meet other needs.

[0068] In one embodiment, Figure 2 As shown, step S300 includes:

[0069] Step S310: Obtain a preference data set, and bring the preference data set into the reward model to update the parameters of the bypass branch corresponding to the reward model.

[0070] In step S310, the preference data set may include user preference information for different behaviors, results or outputs. The preference information may come from user feedback, historical behavior data, expert evaluation, etc.

[0071] Bringing the preference data set into the reward model to update the parameters of the bypass branch corresponding to the reward model may include inputting the collected preference data set into the reward model, and training or fine-tuning the parameters of the bypass branch based on the loss of the reward model so that the parameters of the bypass branch can more accurately reflect the state of the user's preference, thereby improving the accuracy and effectiveness of the reward model.

[0072] In this embodiment, the parameters of the bypass branch corresponding to the reward model are updated through the preference data set, so that the reward model can obtain the preference more accurately, so that the target large model can obtain an output that is more in line with human logic. Moreover, the lora branch can enable the reward model to quickly adapt to the new preference data set, so that it can be adjusted quickly, thereby shortening the fine-tuning time of the initial large model.

[0073] In one embodiment, Figure 3 As shown, step S300 includes:

[0074] Step S320: Obtain a prompt data set, and bring the prompt data set into the strategy model, reward model, criticism model and initial large model respectively to obtain the strategy value, reward value, benefit value and question-answer value.

[0075] In step S320, the prompt data set can be brought into the strategy model, reward model, criticism model and initial large model respectively, and four outputs can be obtained in one forward propagation: the output without lora branch operation (question and answer value), the output with the participation of lora branch of reward model (reward value), the output with the participation of lora branch of criticism model (profit value, or future profit value), and the output with the participation of lora branch of strategy model (strategy value).

[0076] Step S321: Based on the policy value, reward value and benefit value, calculate the policy model loss function and the critical model loss function.

[0077] In step S321, the policy model loss function and the critical model loss function can be calculated based on the policy value, the reward value and the benefit value. By comparing the policy model loss function and the critical model loss function, it can be determined which model needs to be optimized and the specific optimization goal.

[0078] Step S332: Update the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss.

[0079] In step S332, the parameters of the bypass branches corresponding to the policy model and the critical model may be updated based on the policy model loss and the critical model loss, thereby updating the model parameters multiple times.

[0080] In this embodiment, by obtaining a prompt data set and bringing the prompt data set into multiple models, the model can be trained in multiple scenarios. Moreover, by calculating the policy model loss function and the critical model loss function, this embodiment can accurately locate the problems existing in the model during the training process, thereby improving the policy model and the critical model. Compared with retraining the entire model, only updating the parameters of the bypass branch can reduce the consumption of computing resources and time, thereby improving training efficiency and reducing training costs.

[0081] Specifically, Figure 4 As shown, in one possible example, step S332 may include:

[0082] Step S3320: Obtain the strategy value and question-and-answer value.

[0083] In step S3320, the policy value and the question-answer value may be outputs for the same question, or outputs in the same batch of data, thereby helping to maintain data consistency.

[0084] Step S3321: Calculate the difference between the strategy value and the question-answer value.

[0085] In step S3321, the difference value may include KL divergence, etc. By calculating the difference value between the strategy value and the question-answer value, the degree of deviation between the model decision and the output result can be further evaluated.

[0086] Step S3322: Based on the question-answer value and the difference value, as well as the policy model loss, update the parameters of the bypass branch corresponding to the policy model.

[0087] In step S3322, by updating the parameters of the bypass branch corresponding to the policy model based on the question-answer value and the difference value, the actual output result can be more accurately reflected, thereby improving the accuracy of the output. As an example, at this time, the difference value (KL divergence) can be calculated based on the policy value and the question-answer value, and the difference value can be added to the policy model loss as a penalty term of the policy model loss to prevent the model from deviating from the original output and generating hallucination output.

[0088] In another possible example, Figure 5 As shown, step S332 may include:

[0089] Step S3323: Obtain the policy model loss and the critical model loss.

[0090] In step S3323, the policy model loss (the loss function value of the policy model) and the critical model loss (the loss function value of the critical model) can be used to represent the performance of the model during the training process, and then find out the problems that may exist in the model during the training process. As an example, if the policy model loss is high, it may mean that the decision logic of the model is not accurate enough. If the critical model loss is high, it may mean that the evaluation criteria of the model are not reasonable enough.

[0091] Step S3324: Update the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss until the policy model loss and the critical model loss converge to obtain the target large model.

[0092] In step S3324, specifically, the loss function value of the policy model can be calculated, and the parameters of the Lora branch of the policy model can be updated. The loss function value of the critical model can also be calculated, and the parameters of the Lora branch of the critical model can be updated. By continuously iterating and updating the parameters until the loss of the policy model and the loss of the critical model converge, a stable and high-performance target large model can be obtained.

[0093] In this example, the parameters of the bypass branches corresponding to the policy model and the critical model are updated through the policy model loss and the critical model loss, and the performance of the model can be improved through iterative optimization. Moreover, this embodiment only updates the parameters of the bypass branch instead of the entire model, which can reduce the consumption of computing resources and time, and reduce the cost of model optimization. At the same time, this embodiment can automatically adjust the model parameters by setting appropriate loss functions and convergence conditions without manual intervention, thereby improving the efficiency and accuracy of model optimization.

[0094] In another possible example, Figure 6 As shown, step S332 may include:

[0095] Step S3325: Do not update the parameters of the corresponding bypass branch of the reward model, and do not update the parameters of the initial large model.

[0096] In step S3325, the reward model lora branch parameters and the supervised model parameters may be frozen without performing parameter updates.

[0097] In this example, not updating the parameters of the reward model and the initial large model helps maintain the stability of these models, thereby reducing the probability of problems such as overfitting or underfitting, and preventing interference with the training of other models, ensuring that the training process can proceed smoothly.

[0098] In one embodiment, Figure 7 As shown, step S100 may include:

[0099] Step S110: Obtain the original large model and question-answering dataset.

[0100] In step S110, the question and answer data set may include preset questions and corresponding answers. As an example, the data in the question and answer data set may include knowledge questions and answers, dialogue generation, etc.

[0101] Step S120: Generate an initial large model based on the original large model based on the question and answer dataset.

[0102] As an example, the original large model can be fine-tuned in a supervised manner to obtain an initial supervised fine-tuning model, which can be used as the initial large model.

[0103] In this embodiment, the initial large model is a supervised fine-tuning model, thereby reducing the cost of subsequent fine-tuning of the initial large model.

[0104] like Figure 8As shown, the following exemplifies the specific application of the adjustment method of the large model provided in the present application. First, the model can be supervised fine-tuned based on the question-answering dataset to obtain an initial supervised fine-tuning model S. Afterwards, three lora bypass branches R-lora, P-lora, and C-lora can be added to each transformer structure of the model S, which are used to represent the reward model R, the policy model P, and the criticism model C, respectively, for subsequent model low-parameter fine-tuning. For example, the original model S parameter is expressed as W, then the other model parameters are the original model parameters plus the corresponding lora branch parameters. Specifically, the output calculations of the lth lora branch of the models R, P, and C are shown in the following formulas 1, 2, and 3, respectively. The output of the original model S is shown in formula 4, where the matrix e is a hyperparameter, α is a hyperparameter, set to r in the first experiment. f is the activation function, l represents the lth lora branch, and h 0 =x,r 0 =x.

[0105]

[0106]

[0107] y l =f(W l h l-1 )(Formula 4)

[0108] Based on the human preference dataset, the model S is trained to learn human preferences. The fine-tuning process only updates the lora branch parameters corresponding to the reward model R, and the reward model R is trained. Specifically, for the same prompt, the model uses the ranking loss function for different answers, as shown in the following formula, r θ represents the model, x is the input, y w and l There are two answers, D is the data set, and k is the number of answers.

[0109]

[0110] Input the fine-tuning data into the model S, use the policy model lora branch to get the model input p, use the reward model lora branch to calculate the reward value r for the model output, use the criticism model lora branch to calculate the future return c, and output p, r, c according to the above model. The calculation of the policy model loss function and the criticism model loss function are shown below, which are used to update the P-lora and C-lora weight parameters respectively. Among them, S t It is the token sequence that has been generated by the strategy model at the current moment.

[0111] P loss =-(rt +γ*c t+1 -c t )logP(p t |S t )

[0112] C loss =(r t +γ*c t+1 -c t ) 2

[0113] Afterwards, the KL divergence is calculated based on the model output y and the policy model output p, and added to the policy model loss as a penalty term to prevent the model from deviating too much from the original output and producing hallucination output. Then, the parameters are updated based on the policy model loss and the critical model loss until the loss converges. Finally, reasoning can be performed based on the fine-tuned model, and the prompt can be input into the policy model to obtain a logical output.

[0114] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a large model adjustment device for implementing the large model adjustment method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more large model adjustment device embodiments provided below can refer to the limitations of the large model adjustment method above, and will not be repeated here.

[0116] In one embodiment, Fig. 9 As shown, a large model adjustment method is provided, including: an acquisition module, a generation module and an adjustment module, wherein:

[0117] The acquisition module is used to obtain the initial large model, which is used to process question-answering tasks.

[0118] A generation module is used to generate multiple bypass branches related to the initial large model based on the initial large model. The bypass branches include at least a reward model, a criticism model and a strategy model. The reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model.

[0119] The adjustment module is used to obtain a preset data set, bring the preset data set into the reward model, the criticism model and / or the strategy model respectively, and adjust the parameters of the bypass branch of the initial large model to obtain the target large model.

[0120] In one embodiment, the adjustment module is used to obtain a preference data set, and bring the preference data set into the reward model to update the parameters of the bypass branch corresponding to the reward model.

[0121] In one embodiment, the adjustment module is used to obtain a prompt data set, and bring the prompt data set into the policy model, reward model, critical model and initial large model respectively to obtain the policy value, reward value, benefit value and question-answer value; based on the policy value, reward value and benefit value, calculate the policy model loss function and the critical model loss function; based on the policy model loss and the critical model loss, update the parameters of the bypass branches corresponding to the policy model and the critical model.

[0122] In one embodiment, the adjustment module is used to obtain a policy value and a question-and-answer value; calculate a difference between the policy value and the question-and-answer value; and update the parameters of the bypass branch corresponding to the policy model based on the question-and-answer value and the difference, as well as the policy model loss.

[0123] In one embodiment, the adjustment module is used to obtain the policy model loss and the critical model loss; based on the policy model loss and the critical model loss, the parameters of the bypass branches corresponding to the policy model and the critical model are updated until the policy model loss and the critical model loss converge to obtain the target large model.

[0124] In one embodiment, the adjustment module is used to not update the parameters of the corresponding bypass branch of the reward model, and not update the parameters of the initial large model.

[0125] In one embodiment, the acquisition module is used to acquire the original large model and the question and answer dataset; and generate an initial large model based on the question and answer dataset for the original large model.

[0126] Each module in the adjustment device of the large model can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0127] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store adjustment data of a large model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a large model adjustment method is implemented.

[0128] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0130] Step S100: Obtain an initial large model, which is used to process question-answering tasks.

[0131] Step S200: Based on the initial large model, multiple bypass branches related to the initial large model are generated, and the bypass branches include at least a reward model, a criticism model and a strategy model. The reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model.

[0132] Step S300: Obtain a preset data set, bring the preset data set into the reward model, the criticism model and / or the strategy model respectively, and adjust the parameters of the bypass branch of the initial large model to obtain the target large model.

[0133] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0134] Step S310: Obtain a preference data set, and bring the preference data set into the reward model to update the parameters of the bypass branch corresponding to the reward model.

[0135] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0136] Step S320: Obtain a prompt data set, and bring the prompt data set into the strategy model, reward model, criticism model and initial large model respectively to obtain the strategy value, reward value, benefit value and question-answer value.

[0137] Step S321: Based on the policy value, reward value and benefit value, calculate the policy model loss function and the critical model loss function.

[0138] Step S332: Update the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss.

[0139] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0140] Step S3320: Obtain the strategy value and question-and-answer value.

[0141] Step S3321: Calculate the difference between the strategy value and the question-answer value.

[0142] Step S3322: Based on the question-answer value and the difference value, as well as the policy model loss, update the parameters of the bypass branch corresponding to the policy model.

[0143] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0144] Step S3323: Obtain the policy model loss and the critical model loss.

[0145] Step S3324: Update the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss until the policy model loss and the critical model loss converge to obtain the target large model.

[0146] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0147] Step S3325: Do not update the parameters of the corresponding bypass branch of the reward model, and do not update the parameters of the initial large model.

[0148] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0149] Step S110: Obtain the original large model and question-answering dataset.

[0150] Step S120: Generate an initial large model based on the original large model based on the question and answer dataset.

[0151] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0152] Step S100: Obtain an initial large model, which is used to process question-answering tasks.

[0153] Step S200: Based on the initial large model, multiple bypass branches related to the initial large model are generated, and the bypass branches include at least a reward model, a criticism model and a strategy model. The reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model.

[0154] Step S300: Obtain a preset data set, bring the preset data set into the reward model, the criticism model and / or the strategy model respectively, and adjust the parameters of the bypass branch of the initial large model to obtain the target large model.

[0155] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0156] Step S310: Obtain a preference data set, and bring the preference data set into the reward model to update the parameters of the bypass branch corresponding to the reward model.

[0157] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0158] Step S320: Obtain a prompt data set, and bring the prompt data set into the strategy model, reward model, criticism model and initial large model respectively to obtain the strategy value, reward value, benefit value and question-answer value.

[0159] Step S321: Based on the policy value, reward value and benefit value, calculate the policy model loss function and the critical model loss function.

[0160] Step S332: Update the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss.

[0161] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0162] Step S3320: Obtain the strategy value and question-and-answer value.

[0163] Step S3321: Calculate the difference between the strategy value and the question-answer value.

[0164] Step S3322: Based on the question-answer value and the difference value, as well as the policy model loss, update the parameters of the bypass branch corresponding to the policy model.

[0165] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0166] Step S3323: Obtain the policy model loss and the critical model loss.

[0167] Step S3324: Update the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss until the policy model loss and the critical model loss converge to obtain the target large model.

[0168] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0169] Step S3325: Do not update the parameters of the corresponding bypass branch of the reward model, and do not update the parameters of the initial large model.

[0170] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0171] Step S110: Obtain the original large model and question-answering dataset.

[0172] Step S120: Generate an initial large model based on the original large model based on the question and answer dataset.

[0173] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0174] Step S100: Obtain an initial large model, which is used to process question-answering tasks.

[0175] Step S200: Based on the initial large model, multiple bypass branches related to the initial large model are generated, and the bypass branches include at least a reward model, a criticism model and a strategy model. The reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model.

[0176] Step S300: Obtain a preset data set, bring the preset data set into the reward model, the criticism model and / or the strategy model respectively, and adjust the parameters of the bypass branch of the initial large model to obtain the target large model.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0178] Step S310: Obtain a preference data set, and bring the preference data set into the reward model to update the parameters of the bypass branch corresponding to the reward model.

[0179] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0180] Step S320: Obtain a prompt data set, and bring the prompt data set into the strategy model, reward model, criticism model and initial large model respectively to obtain the strategy value, reward value, benefit value and question-answer value.

[0181] Step S321: Based on the policy value, reward value and benefit value, calculate the policy model loss function and the critical model loss function.

[0182] Step S332: Update the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0184] Step S3320: Obtain the strategy value and question-and-answer value.

[0185] Step S3321: Calculate the difference between the strategy value and the question-answer value.

[0186] Step S3322: Based on the question-answer value and the difference value, as well as the policy model loss, update the parameters of the bypass branch corresponding to the policy model.

[0187] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0188] Step S3323: Obtain the policy model loss and the critical model loss.

[0189] Step S3324: Update the parameters of the bypass branches corresponding to the policy model and the critical model based on the policy model loss and the critical model loss until the policy model loss and the critical model loss converge to obtain the target large model.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] Step S3325: Do not update the parameters of the corresponding bypass branch of the reward model, and do not update the parameters of the initial large model.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0193] Step S110: Obtain the original large model and question-answering dataset.

[0194] Step S120: Generate an initial large model based on the original large model based on the question and answer dataset.

[0195] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0196] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0197] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0198] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for adjusting a large model, characterized in that: The method comprises: Obtaining an initial large model, where the initial large model is used to process a question-answering task; Based on the initial large model, a plurality of bypass branches related to the initial large model are generated, wherein the bypass branches at least include a reward model, a criticism model and a strategy model, wherein the reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model; A preset data set is obtained, and the preset data set is respectively introduced into the reward model, the criticism model and / or the strategy model, and the parameters of the bypass branch of the initial large model are adjusted to obtain the target large model.

2. The method for adjusting a large model according to claim 1, characterized in that: The obtaining of a preset data set, bringing the preset data set into the reward model, the criticism model and / or the strategy model respectively, and adjusting the parameters of the bypass branch of the initial large model to obtain a target large model, includes: A preference data set is obtained, and the preference data set is brought into the reward model to update the parameters of the bypass branch corresponding to the reward model.

3. The large model adjustment method according to claim 1, characterized in that: The obtaining of a preset data set, bringing the preset data set into the reward model, the criticism model and / or the strategy model respectively, and adjusting the parameters of the bypass branch of the initial large model to obtain a target large model, includes: Obtain a prompt data set, and bring the prompt data set into the strategy model, the reward model, the criticism model and the initial large model respectively to obtain a strategy value, a reward value, a benefit value and a question-and-answer value; Calculating a policy model loss function and a critical model loss function based on the policy value, the reward value, and the benefit value; Parameters of bypass branches corresponding to the policy model and the critical model are updated based on the policy model loss and the critical model loss.

4. The method for adjusting a large model according to claim 3, characterized in that: The updating of the parameters of the bypass branches corresponding to the strategy model and the critical model based on the strategy model loss and the critical model loss includes: Obtaining the strategy value and the question-answer value; Calculating a difference value between the strategy value and the question-answer value; Based on the question-answer value and the difference value, as well as the policy model loss, the parameters of the bypass branch corresponding to the policy model are updated.

5. The method for adjusting a large model according to claim 3, characterized in that: The updating of the parameters of the bypass branches corresponding to the strategy model and the critical model based on the strategy model loss and the critical model loss includes: Obtaining the policy model loss and the critic model loss; Based on the policy model loss and the critical model loss, the parameters of the bypass branches corresponding to the policy model and the critical model are updated until the policy model loss and the critical model loss converge to obtain the target large model.

6. The method for adjusting a large model according to claim 3, characterized in that: The updating of the parameters of the bypass branches corresponding to the strategy model and the critical model based on the strategy model loss and the critical model loss includes: The parameters of the corresponding bypass branch of the reward model are not updated, and the parameters of the initial large model are not updated.

7. The large model adjustment method according to claim 1, characterized in that: The step of obtaining the initial large model comprises: Get the original large model and question-answering dataset; An initial large model is generated based on the question-answer data set and the original large model.

8. A large model adjustment device, characterized in that: The device comprises: An acquisition module, used to acquire an initial large model, wherein the initial large model is used to process the question-answering task; A generation module, configured to generate a plurality of bypass branches related to the initial large model based on the initial large model, wherein the bypass branches at least include a reward model, a criticism model, and a strategy model, wherein the reward model is used to evaluate the quality of the answer output by the initial large model, the criticism model is used to evaluate whether the answer output by the initial large model is accurate, and the strategy model is used to adjust the path of the answer output by the initial large model; The adjustment module is used to obtain a preset data set, bring the preset data set into the reward model, the criticism model and / or the strategy model respectively, and adjust the parameters of the bypass branch of the initial large model to obtain the target large model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images