Large model alignment method based on equilibrium solution
Through the large model alignment method based on balanced solution, the predictive model is used to divide and build the training data set, and the language model is trained, which solves the problem of poor response effect in different scenarios, achieving better response effect and preference alignment.
Patent Information
- Application Number
- CN202411689955.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-02-28
AI Technical Summary
In the prior art, language models have poor response effects in different scenarios, and it is difficult to select the best response among multiple responses, resulting in poor response effects from the model output.
A large model alignment method based on balanced solution is adopted. By obtaining the problem data set and prediction model, responding to multiple problems based on the preset language model, using the prediction model to divide the response text, construct the training data set, and train the language model to obtain the target model to improve the response effect.
Through this method, the model can better output text with better response effects in different scenarios, improve the response effect of the language model, and achieve better preference alignment of the large model.
Smart Images
Figure CN119539058A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a large model alignment method based on equilibrium solution. Background Art
[0002] Preference alignment of large models is a key technology to improve the performance of large models by adjusting the generation strategy of large language models so that their output results meet human preferences and expectations. In related technologies, the speech model is trained through preset training questions and training responses to obtain a language model after preference alignment. However, in related technologies, preferences are non-transitive, and the best responses are different in multiple responses in different scenarios, resulting in the language model trained by preset training questions and training responses being difficult to respond differently in different scenarios, and there is a problem of poor response effect of the language model output.
[0003] It can be seen that there is a problem in the related technology that the response effect of the language model output is poor. Summary of the invention
[0004] The embodiment of the present invention provides a large model alignment method based on balanced solution to solve the problem of poor response effect of language model output existing in the prior art.
[0005] To solve the above problems, the present invention is achieved as follows:
[0006] In a first aspect, an embodiment of the present invention provides a large model alignment method based on balanced solution, comprising:
[0007] Obtaining a question data set and a prediction model, wherein the question data set includes a plurality of questions, and the prediction model is used to judge the response effect of a response in different scenarios;
[0008] Respond to each of the multiple questions based on a preset language model to obtain two response texts corresponding to each question;
[0009] Dividing the two response texts corresponding to each question into a first text and a second text based on the prediction model, wherein the response effect of the first text is better than the response effect of the second text;
[0010] Building a training data set based on the first text and the second text corresponding to each question;
[0011] The preset language model is trained based on the training data set to obtain a target model, and the target model is used to respond to the question.
[0012] In a second aspect, an embodiment of the present invention further provides a large model alignment device based on balanced solution, comprising:
[0013] A first acquisition module is used to acquire a question data set and a prediction model, wherein the question data set includes multiple questions, and the prediction model is used to judge the response effect of a response in different scenarios;
[0014] A first response module, used to respond to each of the multiple questions based on a preset language model to obtain two response texts corresponding to each question;
[0015] A first division module, configured to divide the two response texts corresponding to each question into a first text and a second text based on the prediction model, wherein the response effect of the first text is better than the response effect of the second text;
[0016] A first construction module, used to construct a training data set based on the first text and the second text corresponding to each question;
[0017] The first training module is used to train the preset language model based on the training data set to obtain a target model, and the target model is used to respond to the question.
[0018] In a third aspect, an embodiment of the present application further provides an electronic device, including a transceiver and a processor.
[0019] The transceiver is used to obtain a question data set and a prediction model, wherein the question data set includes a plurality of questions, and the prediction model is used to determine the response effect of a response in different scenarios;
[0020] The processor is used to respond to each of the multiple questions based on a preset language model to obtain two response texts corresponding to each question;
[0021] The processor is further configured to divide the two response texts corresponding to each question into a first text and a second text based on the prediction model, wherein the response effect of the first text is better than the response effect of the second text;
[0022] The processor is further configured to construct a training data set based on the first text and the second text corresponding to each question;
[0023] The processor is further used to train the preset language model based on the training data set to obtain a target model, and the target model is used to respond to the question.
[0024] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the large model alignment method based on equilibrium solution as described in the first aspect above are implemented.
[0025] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the large model alignment method based on balanced solution as described in the first aspect above are implemented.
[0026] In a sixth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the large model alignment method based on balanced solution as described in the first aspect above.
[0027] In an embodiment of the present application, a question data set and a prediction model are obtained, the question data set includes multiple questions, and the prediction model is used to judge the response effect of the response in different scenarios; each of the multiple questions is responded to based on a preset language model to obtain two response texts corresponding to each question; the two response texts corresponding to each question are divided into a first text and a second text based on the prediction model, and the response effect of the first text is better than the response effect of the second text; a training data set is constructed based on the first text and the second text corresponding to each question; the preset language model is trained based on the training data set to obtain a target model, and the target model is used to respond to the question. In this way, the two response texts corresponding to each question are divided into a first text and a second text through the prediction model, and then a training data set is constructed through the first text and the second text, so that the target model obtained by training the training data set can output more of the first text, that is, output a response text with a better response effect, thereby improving the response effect of the model and achieving better preference alignment of the large model. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0029] Figure 1 is a flow chart of a large model alignment method based on balanced solution provided by an embodiment of the present invention;
[0030] Figure 2 It is a structural diagram of a large model alignment device based on equilibrium solution provided by an embodiment of the present invention;
[0031] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] See also Figure 1 , Figure 1 is a flow chart of a large model alignment method based on balanced solution provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:
[0034] Step 101: Obtain a question data set and a prediction model, wherein the question data set includes multiple questions, and the prediction model is used to determine the response effect of a response in different scenarios.
[0035] The above-mentioned question data set includes multiple different questions, and the above-mentioned prediction model is used to judge the response effect of the response. It should be noted that multiple responses can be generated for the same question, and different responses have different effects in different scenarios. For example, multiple responses include response A, response B, and response C. In the case of scenario one, the response effect is response A>response B>response C, and in the case of scenario two, the response effect is response C>response A>response B, so a prediction model is set in this application to realize the judgment of the response effect in different scenarios.
[0036] Among them, the prediction model can be a model set separately for each scenario. The corresponding scenario can be set for the question in the question data set. When it is necessary to judge the response to the question later, the prediction model can be obtained through the scenario corresponding to the question, and then the response can be judged by the prediction model to determine the response with better effect.
[0037] Step 102: respond to each of the multiple questions based on a preset language model to obtain two response texts corresponding to each question.
[0038] The above-mentioned preset language model is a language model to be trained, and the preset language model can generate multiple response texts for a question. In this application, the preset language model generates only two response texts for each question, so that the prediction model can judge the response effect of the two response texts.
[0039] Furthermore, the questions in the question data set are set with corresponding scenarios, and multiple scenarios can be set for the same question. The preset language model can generate different response texts for different scenarios of the same question, so that the subsequently trained target model can generate response texts that meet the response effects of the scenarios for different scenarios.
[0040] Step 103: Divide the two response texts corresponding to each question into a first text and a second text based on the prediction model, wherein the response effect of the first text is better than the response effect of the second text.
[0041] The two response texts corresponding to each question are divided into a first text and a second text through a prediction model. Specifically, the two response texts are input into the prediction model as input data, and the prediction model then outputs the response text with a better response effect among the two response texts. At this time, the response text is the first text, and the other response text is the second text.
[0042] In another embodiment, two response texts may be input into the prediction model as input data, and the prediction model then outputs the response text with a worse response effect among the two response texts. In this case, the response text is the second text, and the other response text is the first text.
[0043] Step 104: construct a training data set based on the first text and the second text corresponding to each question;
[0044] Step 105: Train the preset language model based on the training data set to obtain a target model, and the target model is used to respond to the question.
[0045] It should be noted that the two response texts corresponding to each question are divided into a first text and a second text through the prediction model, and then a training data set is constructed through the first text and the second text, so that the target model trained by the training data set can output more first texts, that is, output response texts with better response effects, thereby improving the response effect of the model.
[0046] Among them, the first text and the second text are response texts for the same question. Through the prediction model, the first text and the second text can reflect the response preference, and then a training data set for training can be constructed.
[0047] Exemplarily, the preset language model is The problem dataset is D q , the question dataset includes multiple questions x, first through the preset language model Generate two response texts y1 and y2 for each question x; then use the prediction model p to judge the two response texts y1 and y2, and determine the response text with better response effect as the first text y w , the other response text is the second text y l , that is, for each question x, a data (x, y w ,y l). After dividing the first text and the second text corresponding to each question through the prediction model p, the training data set D can be obtained. t ,The training data set includes the first text and the second text corresponding to each question.
[0048] In an embodiment of the present application, a question data set and a prediction model are obtained, the question data set includes multiple questions, and the prediction model is used to judge the response effect of the response in different scenarios; each of the multiple questions is responded to based on a preset language model to obtain two response texts corresponding to each question; the two response texts corresponding to each question are divided into a first text and a second text based on the prediction model, and the response effect of the first text is better than the response effect of the second text; a training data set is constructed based on the first text and the second text corresponding to each question; the preset language model is trained based on the training data set to obtain a target model, and the target model is used to respond to the question. In this way, the two response texts corresponding to each question are divided into a first text and a second text through the prediction model, and then a training data set is constructed through the first text and the second text, so that the target model obtained by training the training data set can output more of the first text, that is, output a response text with a better response effect, thereby improving the response effect of the model and achieving better preference alignment of the large model.
[0049] In one embodiment, the training of the preset language model based on the training data set to obtain a target model includes:
[0050] Training the preset language model based on the training data set to obtain an intermediate training model;
[0051] Based on the intermediate training model and the preset language model, calculating a loss value corresponding to the intermediate training model;
[0052] When the loss value corresponding to the intermediate training model is less than or equal to a preset loss threshold, the intermediate training model is updated to the target model.
[0053] In the embodiment of the present application, the preset language model is trained based on the training data set to obtain an intermediate training model; based on the intermediate training model and the preset language model, the loss value corresponding to the intermediate training model is calculated; when the loss value corresponding to the intermediate training model is less than or equal to the preset loss threshold, the intermediate training model is updated to the target model. In this way, by calculating the loss value of the intermediate training model and the preset loss threshold, the loss value of the target model obtained by training is less than the preset loss value.
[0054] It should be noted that, due to the non-transitive nature of preferences, that is, in different situations, for a certain problem, there are situations where response A>response B, response B>response C, and response C>response A. For example, when a user expresses that he or she has encountered setbacks, response A will give encouragement and empathy, making the user feel understood and supported; response B focuses on providing technical suggestions, giving specific solutions or technical guidance to help users directly deal with the problem; response C emphasizes the diversity of choices. The model will list different solutions or suggestions, allowing users to independently choose the appropriate response method.
[0055] In order to improve the response effect caused by the non-transitivity of preferences, related technologies propose to model preferences as a Nash equilibrium solution problem. Nash equilibrium describes a stable state in which no model can obtain higher returns through unilateral model adjustment. Current methods attempt to solve Nash equilibrium through self-playing.
[0056] Specifically, the response is modeled as a Nash equilibrium solved by the following matrix game:
[0057] Response A Response B Response C Response A 0,0 1 -1,1 Response B -1,1 0,0 1,-1 Response C 1,-1 -1,1 0,0
[0058] In order to find the Nash equilibrium, the commonly used method is the Mirror Descent (MD) algorithm, which obtains a new model π by iterating self-playing in each round. t , which can guarantee the regret value R T Present This means that after T rounds, the average model converges to The Nash equilibrium.
[0059] Among them, the MD algorithm is designed for table algorithms, and the response of the large model is all possible sentences, which cannot be traversed and cannot be adapted to the large model alignment problem, because the response of the large model is all possible sentences, which cannot be traversed. After relevant research, the optimal solution of the Identity Preference Optimization (IPO) method is consistent with the optimal solution in each iteration of the MD method. Due to this consistency, the IPO method can be used to approximate MD, thereby achieving Nash equilibrium more efficiently during the model update process.
[0060] The above IPO method is an alignment optimization method that focuses on optimizing the preference alignment of the model in the self-playing process of the large model. Among them, the IPO method introduces a preference adjustment mechanism in model optimization, so that the model can achieve model balance more naturally. The basic idea of IPO is to guide the model to generate more expected model behaviors by identifying and optimizing the preference features in the model, thereby achieving more efficient model alignment. The loss function of the IPO method is defined as follows:
[0061]
[0062] In the formula, Loss is the loss value, y w is the first text, y l is the second text, t is the number of iterations, is the preset language model, is the intermediate training model, η is a constant, α is a hyperparameter, π′(y) is the original model (i.e., the preset language model), π r (y′) is the new model (i.e., the intermediate training model).
[0063] Therefore, in some embodiments, the loss value corresponding to the intermediate training model in this application can be calculated by the loss function of the IPO method.
[0064] In some implementations, the loss value corresponding to the intermediate training model may also be calculated using the following loss function:
[0065]
[0066] In the formula, Loss is the loss value, y w is the first text, y l is the second text, t is the number of iterations, is the preset language model, is the intermediate training model, η is a constant, and α is a hyperparameter.
[0067] It should be noted that the model finally trained through IPO will be Instead of π t+1 , then we get the regret value lower bound R T (regret bound) is:
[0068]
[0069] Among them, if not If the restriction is made, the calculated R T The size is The sublinear growth is not satisfied, resulting in the model being unable to converge to the Nash equilibrium. In order to enable the model to converge, in some embodiments, the loss function is adjusted as follows:
[0070]
[0071] Loss in the formula new is the loss value, y w is the first text, y l is the second text, t is the number of iterations, is the preset language model, is the intermediate training model, η is a constant, and α is a hyperparameter.
[0072] in, This is the part that is added to the loss function of the present application and the loss function of the IPO method, through which the preset language model and the intermediate training model are made close, so that the model converges to the Nash equilibrium.
[0073] In one embodiment, after the preset language model is trained based on the training data set to obtain a target model, the method includes:
[0074] Obtain a test set, the test set comprising a plurality of test questions and a test response corresponding to each test question;
[0075] Testing the target model based on the test set to obtain a preference matching degree and / or output accuracy corresponding to the target model;
[0076] In the case where the preference matching degree and / or the output accuracy do not meet the preset conditions, responding to each of the multiple questions based on the target model to obtain two intermediate response texts corresponding to each question;
[0077] The two response texts corresponding to each question are divided into a third text and a fourth text based on the prediction model, and the response effect of the third text is better than the response effect of the fourth text;
[0078] Constructing an intermediate training data set based on the third text and the fourth text corresponding to each question;
[0079] The target model is trained based on the intermediate training data set to obtain an updated target model.
[0080] In an embodiment of the present application, a test set is obtained, the test set includes multiple test questions, and a test response corresponding to each test question; the target model is tested based on the test set to obtain the preference matching degree and / or output accuracy corresponding to the target model; when the preference matching degree and / or output accuracy do not meet the preset conditions, each of the multiple questions is responded to based on the target model to obtain two intermediate response texts corresponding to each question; the two response texts corresponding to each question are divided into a third text and a fourth text based on the prediction model, and the response effect of the third text is better than the response effect of the fourth text; an intermediate training data set is constructed based on the third text and the fourth text corresponding to each question; the target model is trained based on the intermediate training data set to obtain an updated target model. In this way, the trained target model is tested through the test set to obtain the preference matching degree and output accuracy, and then the preference matching degree and output accuracy are used to determine whether the target model meets the requirements. If it does not meet the requirements, the target model is iteratively updated again until the preference matching degree and output accuracy of the target model meet the preset conditions, thereby improving the output response effect of the target model.
[0081] In some embodiments, the target model is tested based on a test set to obtain a preference match corresponding to the target model. Specifically, the test set may include two responses to the same question and a preference for the responses. The target model responds to the test question in the test set to obtain two responses and a preference output result, and then the preference match is calculated using the output results of the test set and the target model.
[0082] In some embodiments, the target model is tested based on a test set to obtain the output accuracy corresponding to the target model. Specifically, the test set may include test responses to test questions, and the target model responds to the test questions in the test set to obtain response results to the questions, and then the output accuracy is calculated through the test responses and responses.
[0083] In some embodiments, the preference matching degree and / or output accuracy corresponding to the target model can be calculated by ChatGPT or other models.
[0084] In one embodiment, the preset condition includes at least one of the following:
[0085] The preference matching degree is greater than or equal to a set matching degree threshold;
[0086] The output accuracy is greater than or equal to a set accuracy threshold.
[0087] In an embodiment of the present application, whether the target model meets the requirements is determined by setting a matching threshold and a preferred matching degree, and / or setting an accuracy threshold and an output accuracy. If it does not meet the requirements, the target model is iteratively updated again to improve the output response effect of the target model.
[0088] See also Figure 2 , Figure 2 is a structural diagram of a large model alignment device based on equilibrium solution provided by an embodiment of the present invention, such as Figure 2 As shown, the large model alignment device 200 based on equilibrium solution includes:
[0089] A first acquisition module 201 is used to acquire a question data set and a prediction model, wherein the question data set includes a plurality of questions, and the prediction model is used to judge the response effect of a response in different scenarios;
[0090] A first response module 202, configured to respond to each of the multiple questions based on a preset language model, and obtain two response texts corresponding to each question;
[0091] A first division module 203, for dividing the two response texts corresponding to each question into a first text and a second text based on the prediction model, wherein the response effect of the first text is better than the response effect of the second text;
[0092] A first construction module 204, configured to construct a training data set based on the first text and the second text corresponding to each question;
[0093] The first training module 205 is used to train the preset language model based on the training data set to obtain a target model, and the target model is used to respond to the question.
[0094] In one embodiment, the first training module 205 includes:
[0095] A training unit, configured to train the preset language model based on the training data set to obtain an intermediate training model;
[0096] A calculation unit, configured to calculate a loss value corresponding to the intermediate training model based on the intermediate training model and the preset language model;
[0097] A setting unit is used to update the intermediate training model to the target model when the loss value corresponding to the intermediate training model is less than or equal to a preset loss threshold.
[0098] In one embodiment, the loss value corresponding to the intermediate training model is calculated based on the intermediate training model and the preset language model, which is expressed by the following formula:
[0099]
[0100] Loss in the formula new is the loss value, y w is the first text, y l is the second text, t is the number of iterations, is the preset language model, is the intermediate training model, η is a constant, and α is a hyperparameter.
[0101] In one embodiment, after the first training module 205, the large model alignment device 200 based on equilibrium solution includes:
[0102] A second acquisition module is used to acquire a test set, wherein the test set includes a plurality of test questions and a test response corresponding to each test question;
[0103] A testing module, used for testing the target model based on the test set to obtain a preference matching degree and / or output accuracy corresponding to the target model;
[0104] A second response module is used to respond to each of the multiple questions based on the target model to obtain two intermediate response texts corresponding to each question when the preference matching degree and / or the output accuracy do not meet the preset conditions;
[0105] A second division module is used to divide the two response texts corresponding to each question into a third text and a fourth text based on the prediction model, wherein the response effect of the third text is better than the response effect of the fourth text;
[0106] A second construction module is used to construct an intermediate training data set based on the third text and the fourth text corresponding to each question;
[0107] The second training module is used to train the target model based on the intermediate training data set to obtain an updated target model.
[0108] In one embodiment, the preset condition includes at least one of the following:
[0109] The preference matching degree is greater than or equal to a set matching degree threshold;
[0110] The output accuracy is greater than or equal to a set accuracy threshold.
[0111] The large model alignment device based on balanced solution provided in the embodiment of the present invention can realize each process of each embodiment of the above-mentioned large model alignment method based on balanced solution. The technical features correspond one to one and can achieve the same technical effect. To avoid repetition, they will not be described here.
[0112] It should be noted that the large model alignment device based on equilibrium solution in the embodiment of the present invention can be a device, or a component, integrated circuit, or chip in an electronic device.
[0113] For details, see Figure 3 The embodiment of the present application further provides an electronic device, including a bus 301, a transceiver 302, an antenna 303, a bus interface 304, a processor 305 and a memory 306.
[0114] The transceiver 302 is used to obtain a question data set and a prediction model, wherein the question data set includes a plurality of questions, and the prediction model is used to determine the response effect of a response in different scenarios;
[0115] The processor 305 is used to respond to each of the multiple questions based on a preset language model to obtain two response texts corresponding to each question;
[0116] The processor 305 is further configured to divide the two response texts corresponding to each question into a first text and a second text based on the prediction model, wherein the response effect of the first text is better than the response effect of the second text;
[0117] The processor 305 is further configured to construct a training data set based on the first text and the second text corresponding to each question;
[0118] The processor 305 is further used to train the preset language model based on the training data set to obtain a target model, and the target model is used to respond to the question.
[0119] In one embodiment, the training of the preset language model based on the training data set to obtain a target model includes:
[0120] Training the preset language model based on the training data set to obtain an intermediate training model;
[0121] Based on the intermediate training model and the preset language model, calculating a loss value corresponding to the intermediate training model;
[0122] When the loss value corresponding to the intermediate training model is less than or equal to a preset loss threshold, the intermediate training model is updated to the target model.
[0123] In one embodiment, the loss value corresponding to the intermediate training model is calculated based on the intermediate training model and the preset language model, which is expressed by the following formula:
[0124]
[0125] Loss in the formula new is the loss value, y w is the first text, y l is the second text, t is the number of iterations, is the preset language model, is the intermediate training model, η is a constant, and α is a hyperparameter.
[0126] In one embodiment, the transceiver 302 is further used to obtain a test set, wherein the test set includes a plurality of test questions and a test response corresponding to each test question;
[0127] The processor 305 is further configured to test the target model based on the test set to obtain a preference matching degree and / or output accuracy corresponding to the target model;
[0128] The processor 305 is further configured to respond to each of the multiple questions based on the target model to obtain two intermediate response texts corresponding to each question when the preference matching degree and / or the output accuracy do not meet a preset condition;
[0129] The processor 305 is further configured to divide the two response texts corresponding to each question into a third text and a fourth text based on the prediction model, wherein the response effect of the third text is better than the response effect of the fourth text;
[0130] The processor 305 is further configured to construct an intermediate training data set based on the third text and the fourth text corresponding to each question;
[0131] The processor 305 is further configured to train the target model based on the intermediate training data set to obtain an updated target model.
[0132] In one embodiment, the preset condition includes at least one of the following:
[0133] The preference matching degree is greater than or equal to a set matching degree threshold;
[0134] The output accuracy is greater than or equal to a set accuracy threshold.
[0135] exist Figure 3In the embodiment, the bus architecture (represented by bus 301) is shown, and bus 301 may include any number of interconnected buses and bridges, and bus 301 links various circuits including one or more processors represented by processor 305 and memory represented by memory 306. Bus 301 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 304 provides an interface between bus 301 and transceiver 302. Transceiver 302 may be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by processor 305 is transmitted on a wireless medium via antenna 303, and further, antenna 303 also receives data and transmits the data to processor 305.
[0136] The processor 305 is responsible for managing the bus 301 and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 306 can be used to store data used by the processor 305 when performing operations.
[0137] Optionally, the processor 305 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD).
[0138] An embodiment of the present application also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the embodiment of the large model alignment method based on equilibrium solution are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0139] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the embodiment of the large model alignment method based on equilibrium solution is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0140] An embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the embodiment of the above-mentioned large model alignment method based on equilibrium solution are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.
[0141] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.
[0143] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A large model alignment method based on equilibrium solution, characterized in that: include: Obtaining a question data set and a prediction model, wherein the question data set includes a plurality of questions, and the prediction model is used to judge the response effect of a response in different scenarios; Respond to each of the multiple questions based on a preset language model to obtain two response texts corresponding to each question; Dividing the two response texts corresponding to each question into a first text and a second text based on the prediction model, wherein the response effect of the first text is better than the response effect of the second text; Building a training data set based on the first text and the second text corresponding to each question; The preset language model is trained based on the training data set to obtain a target model, and the target model is used to respond to the question.
2. The method according to claim 1, characterized in that The step of training the preset language model based on the training data set to obtain a target model includes: Training the preset language model based on the training data set to obtain an intermediate training model; Based on the intermediate training model and the preset language model, calculating a loss value corresponding to the intermediate training model; When the loss value corresponding to the intermediate training model is less than or equal to a preset loss threshold, the intermediate training model is updated to the target model.
3. The method according to claim 2, characterized in that The loss value corresponding to the intermediate training model is calculated based on the intermediate training model and the preset language model, which is expressed by the following formula: Loss in the formula new is the loss value, y w is the first text, y l is the second text, t is the number of iterations, is the preset language model, is the intermediate training model, η is a constant, and a is a hyperparameter.
4. The method according to claim 1, characterized in that After the preset language model is trained based on the training data set to obtain a target model, the method includes: Obtain a test set, the test set comprising a plurality of test questions and a test response corresponding to each test question; Testing the target model based on the test set to obtain a preference matching degree and / or output accuracy corresponding to the target model; In the case where the preference matching degree and / or the output accuracy do not meet the preset conditions, responding to each of the multiple questions based on the target model to obtain two intermediate response texts corresponding to each question; The two response texts corresponding to each question are divided into a third text and a fourth text based on the prediction model, and the response effect of the third text is better than the response effect of the fourth text; Constructing an intermediate training data set based on the third text and the fourth text corresponding to each question; The target model is trained based on the intermediate training data set to obtain an updated target model.
5. The method according to claim 4, characterized in that The preset condition includes at least one of the following: The preference matching degree is greater than or equal to a set matching degree threshold; The output accuracy is greater than or equal to a set accuracy threshold.
6. A large model alignment device based on equilibrium solution, characterized in that: include: A first acquisition module is used to acquire a question data set and a prediction model, wherein the question data set includes multiple questions, and the prediction model is used to judge the response effect of a response in different scenarios; A first response module, used to respond to each of the multiple questions based on a preset language model to obtain two response texts corresponding to each question; A first division module, configured to divide the two response texts corresponding to each question into a first text and a second text based on the prediction model, wherein the response effect of the first text is better than the response effect of the second text; A first construction module, used to construct a training data set based on the first text and the second text corresponding to each question; The first training module is used to train the preset language model based on the training data set to obtain a target model, and the target model is used to respond to the question.
7. An electronic device, characterized in that: Including transceiver and processor, The transceiver is used to obtain a question data set and a prediction model, wherein the question data set includes a plurality of questions, and the prediction model is used to determine the response effect of a response in different scenarios; The processor is used to respond to each of the multiple questions based on a preset language model to obtain two response texts corresponding to each question; The processor is further configured to divide the two response texts corresponding to each question into a first text and a second text based on the prediction model, wherein the response effect of the first text is better than the response effect of the second text; The processor is further configured to construct a training data set based on the first text and the second text corresponding to each question; The processor is further used to train the preset language model based on the training data set to obtain a target model, and the target model is used to respond to the question.
8. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps in the large model alignment method based on equilibrium solution as described in any one of claims 1 to 5.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the large model alignment method based on equilibrium solution as described in any one of claims 1 to 5.
10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the large model alignment method based on balanced solution as described in any one of claims 1 to 5.