Language model fine tuning method and device, electronic equipment and storage medium

Abstract

The invention provides a language model fine tuning method and device, electronic equipment and a storage medium, and relates to the field of natural language model processing, and the method comprises the steps: obtaining sample data which comprises a plurality of data with different data qualities; reasoning the sample data for multiple times based on the first language model to generate multiple pieces of result information and multiple pieces of process information; determining a loss weight based on the multiple pieces of result information and the first label of the sample data; on the basis of the multiple pieces of result information, the multiple pieces of process information and a first label of the sample data, determining thinking chain data; and based on the thinking chain data and the loss weight, determining a target loss, and performing fine tuning on the to-be-fine-tuned model based on the target loss to obtain a target language model. According to the method, the quality of the sample data and the logical thinking ability of model reasoning are concerned in the optimization of the to-be-fine-tuned model through the loss weight and the thinking chain data, so that the interpretability and the mathematical ability of the language model are effectively improved, and the error rate of the model is reduced.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of natural language model processing, and more specifically, to a language model fine-tuning method, device, electronic device, and storage medium. Background Art

[0002] Recently, large-scale pre-trained language models have developed rapidly and made great progress in many fields such as translation and writing, bringing great convenience to people's daily lives. However, the existing large models still have certain deficiencies in logical reasoning ability, and it is difficult to directly obtain correct conclusions, which cannot be achieved to a satisfactory level. There is still a lot of room for development.

[0003] Among them, mathematical ability is undoubtedly a typical representative of logical reasoning ability, which can reflect the gap between different models to a certain extent. Therefore, how to optimize the language model, improve the interpretability and mathematical ability of the model, enable the model to draw conclusions after thinking, and reduce the error rate has become an urgent problem to be solved in this field. Summary of the invention

[0004] The embodiments of the present application provide a language model fine-tuning method, device, electronic device and storage medium, aiming to solve the problem of how to improve the interpretability and mathematical ability of the language model and reduce the model error rate.

[0005] A first aspect of an embodiment of the present application provides a language model fine-tuning method, the method comprising:

[0006] Acquire sample data, where the sample data includes multiple types of data with different data qualities;

[0007] Perform multiple inferences on the sample data based on the first language model to generate multiple result information and multiple process information corresponding to each result information;

[0008] Determine a loss weight based on the multiple result information and the first label of the sample data, where the loss weight is used to characterize the data quality of the sample data;

[0009] Determine thought chain data based on the plurality of result information, the plurality of process information and the first label of the sample data;

[0010] Based on the thought chain data and the loss weight, a target loss is determined, and the model to be fine-tuned is fine-tuned based on the target loss to obtain a target language model.

[0011] In an optional implementation, the performing multiple inferences on the sample data based on the first language model to generate multiple result information and multiple process information corresponding to each result information includes:

[0012] Acquire a plurality of thought chain prompt words, wherein the thought chain prompt words are used to represent an expected reasoning process of the first language model;

[0013] Add thought chain prompt words to the sample data, use the thought chain prompt words as guiding information, perform multiple inferences on the sample data based on the first language model, and generate the multiple result information and multiple process information corresponding to each result information.

[0014] In an optional implementation manner, the first label is a standard value of the result information corresponding to the sample data, and determining the loss weight based on the plurality of result information and the first label of the sample data includes:

[0015] Determine a first loss weight based on the multiple result information and the first label of the sample, where the first loss weight is used to characterize an error ratio of the multiple result information relative to the first label;

[0016] Determine a second loss weight based on the multiple result information and the first label of the sample, where the second loss weight is used to characterize the stability of multiple inferences performed on the sample data based on the first language model;

[0017] The first loss weight and the second loss weight are combined into the loss weight.

[0018] In an optional implementation, determining a first loss weight based on the plurality of result information and the first label of the sample includes:

[0019] Compare each result information with the first label to generate an error ratio corresponding to each result information;

[0020] Determining the first loss weight based on the error proportions respectively corresponding to all the result information and the amount of the result information;

[0021] If the result information is the same as the first label, the error ratio corresponding to the result information is determined to be 0; if the result information is different from the first label, the error ratio corresponding to the result information is determined to be 1.

[0022] In an optional implementation manner, determining the second loss weight based on the multiple result information and the first label of the sample includes:

[0023] Based on the multiple result information and the first label of the sample, obtaining a target number, where the target number is the number of different results in the multiple result information;

[0024] Obtaining the probability that each result information matches the first label as the target probability corresponding to each result information;

[0025] The second loss weight is determined based on the target quantity and the target probability corresponding to each result information.

[0026] In an optional implementation manner, determining the thought chain data based on the plurality of result information, the plurality of process information and the first label of the sample data includes:

[0027] Based on the first label of the sample data, the plurality of result information are screened to determine correct result information matching the first label among the plurality of result information;

[0028] The process information corresponding to the correct result information is used as the thinking chain data.

[0029] In an optional implementation manner, determining the target loss based on the thought chain data and the loss weight includes:

[0030] Acquire a second label of the sample data, where the second label is a standard value of process information corresponding to the sample data;

[0031] Based on the thought chain data and the second label, obtaining a loss value of the thought chain data;

[0032] Based on the loss weight, the loss value of the thought chain data is weighted to determine the target loss.

[0033] A second aspect of an embodiment of the present application provides a language model fine-tuning device, the device comprising:

[0034] A first acquisition module is used to acquire sample data, wherein the sample data includes multiple types of data with different data qualities;

[0035] An inference module, configured to perform multiple inferences on the sample data based on the first language model, and generate multiple result information and multiple process information corresponding to each result information;

[0036] A loss weight module, used to determine a loss weight based on the multiple result information and the first label of the sample data, wherein the loss weight is used to characterize the data quality of the sample data;

[0037] A thought chain data module, used for determining thought chain data based on the plurality of result information, the plurality of process information and the first label of the sample data;

[0038] The fine-tuning module is used to determine the target loss based on the thought chain data and the loss weight, and fine-tune the model to be fine-tuned based on the target loss to obtain a target language model.

[0039] In an optional implementation, the reasoning module includes:

[0040] A thought chain prompt word submodule, used for acquiring a plurality of thought chain prompt words, wherein the thought chain prompt words are used for representing the expected reasoning process of the first language model;

[0041] The reasoning submodule is used to add thought chain prompt words to the sample data, and use the thought chain prompt words as guiding information to perform multiple reasoning on the sample data based on the first language model to generate the multiple result information and multiple process information corresponding to each result information.

[0042] In an optional implementation, the loss weight module includes:

[0043] A first loss weight submodule, configured to determine a first loss weight based on the plurality of result information and the first label of the sample, wherein the first loss weight is used to characterize an error ratio of the plurality of result information relative to the first label;

[0044] A second loss weight submodule, configured to determine a second loss weight based on the plurality of result information and the first label of the sample, wherein the second loss weight is used to characterize the stability of multiple inferences performed on the sample data based on the first language model;

[0045] A loss weight submodule, used to combine the first loss weight and the second loss weight into the loss weight.

[0046] In an optional implementation manner, the first loss weight submodule includes:

[0047] an error ratio determination unit, configured to compare each result information with the first label to generate an error ratio corresponding to each result information;

[0048] A first loss weight unit, used to determine the first loss weight based on the error proportions respectively corresponding to all the result information and the amount of the result information;

[0049] If the result information is the same as the first label, the error ratio corresponding to the result information is determined to be 0; if the result information is different from the first label, the error ratio corresponding to the result information is determined to be 1.

[0050] In an optional implementation manner, the second loss weight submodule includes:

[0051] a target quantity determining unit, configured to obtain a target quantity based on the plurality of result information and the first label of the sample, wherein the target quantity is the number of different results in the plurality of result information;

[0052] a target probability determination unit, configured to obtain a probability that each result information matches the first label as a target probability corresponding to each result information;

[0053] The second loss weight unit is used to determine the second loss weight based on the target quantity and the target probability corresponding to each result information.

[0054] In an optional implementation manner, the thought chain data module includes:

[0055] A correct result information screening submodule, used to screen the plurality of result information based on the first label of the sample data, and determine the correct result information matching the first label among the plurality of result information;

[0056] The thought chain data determination submodule is used to use the process information corresponding to the correct result information as the thought chain data.

[0057] In an optional implementation, the fine-tuning module includes:

[0058] A second label acquisition submodule, used to acquire a second label of the sample data, where the second label is a standard value of process information corresponding to the sample data;

[0059] A loss value acquisition submodule, used for acquiring the loss value of the thought chain data based on the thought chain data and the second label;

[0060] The target loss submodule is used to perform weighted processing on the loss value of the thought chain data based on the loss weight to determine the target loss.

[0061] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the language model fine-tuning method described in any one of the first aspect of the embodiments of the present application.

[0062] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the language model fine-tuning method described in any one of the first aspect of the embodiment of the present application are implemented.

[0063] Beneficial effects:

[0064] The present application provides a language model fine-tuning method, device, electronic device and storage medium, the method comprising: obtaining sample data, the sample data comprising multiple data with different data quality; performing multiple reasoning on the sample data based on a first language model to generate multiple result information and multiple process information corresponding to each result information; determining a loss weight based on the multiple result information and a first label of the sample data, the loss weight being used to characterize the data quality of the sample data; determining thought chain data based on the multiple result information, the multiple process information and the first label of the sample data; determining a target loss based on the thought chain data and the loss weight, and fine-tuning the model to be fine-tuned based on the target loss to obtain a target language model. The present application generates loss weights and thought chain data based on the first language model, and pays attention to the quality of the sample data and the logical thinking ability of the model reasoning in the optimization of the model to be fine-tuned through the loss weight and the thought chain data, respectively, effectively improving the interpretability and mathematical ability of the language model and reducing the model error rate.

[0065] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0067] Figure 1 It is a flowchart of a language model fine-tuning method proposed in one embodiment of the present application;

[0068] Figure 2 It is a schematic diagram of the steps of a language model fine-tuning method proposed in one embodiment of the present application;

[0069] Figure 3 It is a structural schematic diagram of a language model fine-tuning device proposed in one embodiment of the present application;

[0070] Figure 4 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0072] In the drawings, the size of the constituent elements, the thickness of the layer or the area may be exaggerated for the sake of clarity. Therefore, any implementation of the present disclosure is not necessarily limited to the size shown in the drawings, and the shapes and sizes of the components in the drawings do not reflect the true proportions. In addition, the drawings schematically show ideal examples, and any implementation of the present disclosure is not limited to the shapes or values ​​shown in the drawings.

[0073] In recent years, large-scale pre-trained language models have developed rapidly and have made great progress in many fields such as translation and writing, bringing great convenience to people's daily lives. However, the existing large models still have certain deficiencies in logical reasoning ability, and it is difficult to directly obtain correct conclusions and cannot achieve a satisfactory level. There is a lot of room for development. Among them, mathematical ability is undoubtedly a typical representative of logical reasoning ability, which can reflect the gap between different models to a certain extent.

[0074] The recently emerged thinking chain technology is a good way to improve the mathematical ability of large models. Thinking chain technology requires the model to first output the intermediate thinking process and reasoning steps, and then get the final answer. When applied to the field of mathematics, the model is required to output the calculation process first, and then the result, so that developers can understand the model's thinking mode, optimize it in a targeted manner, improve the model's interpretability, and enable the model to reach a conclusion after thinking, reducing the error rate.

[0075] In view of this, the present application embodiment proposes a language model fine-tuning method. Figure 1 A flow chart of a language model fine-tuning method proposed in an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0076] S101. Acquire sample data, where the sample data includes a variety of data with different data qualities.

[0077] When the step S101 is specifically implemented, sample data is obtained, and the sample data is used as training data for fine-tuning the model to be fine-tuned, wherein the sample data includes multiple data with different data qualities. Since not all data play a positive role in the fine-tuning of the model to be fine-tuned, if there are a large amount of simple data in the sample data, the gradients generated by these simple data will dominate the optimization direction of the model to be fine-tuned, reducing the impact of more difficult and higher-quality sample data. Therefore, the embodiment of the present application sets multiple data with different data qualities in the sample data, so that during the process of training and fine-tuning the model to be fine-tuned based on the sample data, it can learn to filter out more difficult and higher-quality data from a large number of different types of sample data for training and fine-tuning, thereby improving the performance of the optimized model.

[0078] Exemplarily, the model to be fine-tuned is a language model, and the sample data includes two different types of data, one of which is a public training set used in academic research, which serves as sample data with relatively high data quality and relatively high difficulty in the model to be fine-tuned; the other type is math test questions published on the examination website, which serves as sample data with relatively low data quality and relatively low difficulty in the model to be fine-tuned. It should be noted that the above example is only an optional sample data type given to enable those skilled in the art to better understand the solution of the embodiment of the present application. The specific sample data can be determined according to actual conditions, and the embodiment of the present application does not make specific restrictions here.

[0079] In the embodiment of the present application, each of the sample data also includes corresponding label data, and the label data is used to characterize the target data of the sample data. When the model to be fine-tuned is a language model, the model to be fine-tuned is used to infer the problem information of the input sample data and generate the result corresponding to the problem information; at the same time, in order to improve the interpretability of the model to be fine-tuned and reduce the error rate, the model to be fine-tuned will also be required to output the intermediate thinking process and reasoning steps, and then generate the corresponding result based on the thinking process and reasoning process. Therefore, the label data corresponding to the sample data in the embodiment of the present application includes at least a first label and a second label, the first label is the standard value of the result information corresponding to the sample data, and the second label is the standard value of the process information corresponding to the sample data. In the embodiment of the present application, the loss corresponding to the result information and the loss corresponding to the process information are obtained respectively through the first label and the second label, and the model to be fine-tuned is fine-tuned based on these losses, which can improve the interpretability of the optimized model to be fine-tuned, and also enable the optimized model to be fine-tuned to get a conclusion after thinking, reduce the error rate, and effectively improve the model performance.

[0080] S102: Perform multiple inferences on the sample data based on the first language model to generate multiple result information and multiple process information corresponding to each result information.

[0081] When the step S102 is specifically implemented, the sample data is input into the first language model to generate multiple result information and multiple process information corresponding to each result information. The first language model is at least one model with a larger scale than the model to be fine-tuned. Since the larger first language model often has stronger logical thinking ability, the embodiment of the present application uses the first language model to infer the sample data and obtain its process information and result information, which can avoid manual labeling, effectively reduce the manpower and material resources required for fine-tuning the model to be fine-tuned, and reduce the cost of fine-tuning.

[0082] Specifically, first obtain a plurality of thought chain prompts, wherein the thought chain prompts are a plurality of process guide words preset in advance, which are used to characterize the expected reasoning process of the first language model; then, add thought chain prompts to the sample data, and use the thought chain prompts as guiding information to perform multiple reasonings on the sample data based on the first language model to generate the plurality of result information and the plurality of process information corresponding to each result information. In the embodiment of the present application, the core idea of ​​setting the thought chain prompts is to imitate the human problem-solving process, and finally get the answer through step-by-step reasoning and answering questions. By presetting a plurality of different thought chain prompts for each sample data, when the sample data is input into the first language model for reasoning, different structured paths are provided for the reasoning process of the first language model through different thought chain prompts, guiding the first language model to traverse the complexity of the problem information in the sample data based on the thought chain prompts, and obtain process information and result information that are more in line with the thought chain prompts.

[0083] Exemplarily, the question information in the sample data is "Xiao Ming bought 10 pencils and erasers, spending a total of 100 yuan. It is known that the pencil costs 6 yuan each. What is the unit price of the eraser?", and a thinking chain prompt word "first reason step by step, and then get the calculation result" is set for the question information of the sample data. The question information is input into the first language model. Under the guidance of the thinking chain prompt word, the first language model infers the question information to generate process information, and generates corresponding result information based on the reasoning process information, which depends on the reasoning of the process information. "Xiao Ming spent a total of 10*6=60 yuan on pencils, so it can be obtained that the total cost of buying erasers is 100-60=40 yuan. Xiao Ming bought 10 erasers, and the price of each eraser is 40 / 10=4 yuan, so the result is 4". It should be noted that the above example is only an optional sample data reasoning process given to enable those skilled in the art to better understand the solution of the embodiment of the present application. The specific thinking chain prompt words, sample data, process information and result information can be determined according to actual conditions, and the embodiment of the present application does not make specific restrictions here.

[0084] It is easy to understand that the multiple different thought chain prompt words preset in the embodiment of the present application can be used as guide words for the same sample data, and multiple groups of result information and process information corresponding to different thought chain prompt words can be generated based on the reasoning of the first language model.

[0085] S103: Determine a loss weight based on the multiple result information and the first label of the sample data, where the loss weight is used to characterize the data quality of the sample data.

[0086] When the step S103 is specifically implemented, since the sample data includes multiple data of different importance, on the one hand, the sample data that the model cannot infer to obtain the correct result can often have a greater impact on the improvement of the model, and the optimization performance of the model to be fine-tuned is better; on the other hand, the prediction results of the large model have a certain degree of randomness. For a large sample data, the output result information and process information are often more stable. When the inference result obtained by the model after inferring a certain sample data is more unstable, it indicates that there is more information to be learned in the sample data. Therefore, in the embodiment of the present application, after obtaining the process information and result information corresponding to the sample data, the error ratio and instability of the output prediction value (process information and result information) corresponding to the sample data are obtained through the multiple result information and the label data of the sample data to measure the quality of the sample data.

[0087] Specifically, after obtaining the multiple result information, a first loss weight is determined based on the multiple result information and the first label of the sample, and the first loss weight is used to characterize the error ratio of the multiple result information relative to the first label. Among them, for the multiple result information obtained by performing multiple inferences on the first language model, the greater the probability that the multiple result information is wrong, the greater the value of the first loss weight corresponding to the sample data; conversely, the smaller the probability that the multiple result information is wrong, the smaller the value of the corresponding first loss weight. When the multiple result information obtained by performing multiple inferences on the first language model are all correct, the first loss weight corresponding to the sample data is the smallest.

[0088] In some optional embodiments, the first loss weight is obtained as follows: each result information is compared with the first label to generate an error ratio corresponding to each result information, wherein the result information that is the same as the first label is identified as a correct result, and the result information that is different from the first label is identified as an error result; the first loss weight is determined based on the error ratios corresponding to all the result information and the number of the result information.

[0089] Optionally, for the sample data x i , the corresponding first label is y i , is the sample data x i Add thought chain prompt word p i , using the first language model f to train the sample data x i Perform k inferences to obtain multiple result information y i1 ,y i2 …y ik The first loss weight is obtained according to the following formula:

[0090]

[0091] Among them, w ie is the first loss weight; k is the weight for the sample data x i Based on the thought chain prompt word p i The number of inferences performed; j is the jth inference among the k inferences; I is the result information y i The error ratio.

[0092] According to the above formula, in the embodiment of the present application, in the process of determining the first loss weight based on the result information and the first label, if the result information is the same as the first label, the error ratio corresponding to the result information is determined to be 0; if the result information is different from the first label, the error ratio corresponding to the result information is determined to be 1. That is to say, for When ij ≠y i When y ij =y i When , the value is 0. At this time, the multiple result information y obtained by performing k inferences on the first language model ij When all are correct, the sample data x i The corresponding first loss weight w ie The minimum is

[0093] After obtaining the multiple result information and the corresponding multiple process information, a second loss weight is determined based on the multiple result information and the first label of the sample, and the second loss weight is used to characterize the stability of multiple inferences on the sample data based on the first language model.

[0094] In some optional embodiments, the second loss weight is obtained as follows: based on the multiple result information and the first label of the sample, a target number is obtained, and the target number is the number of different results in the multiple result information. For example, for the question information in the sample data, the first language model performs 5 inferences based on different thinking chain prompts to obtain 5 result information, which are 1, 1, 2, 2, and 2 respectively, including 2 different types of result information 1 and 2, and the target number is determined to be 2; then, the probability of each result information matching the first label is obtained as the target probability corresponding to each result information; based on the target number and the target probability corresponding to each result information, the second loss weight is determined, wherein when the target number of the multiple result information is smaller, it indicates that the stability of the sample data is stronger, and the second loss weight is smaller at this time; when all the result information is the same value, it indicates that the result of the sample data is the most stable, and the corresponding second loss weight is 0; on the contrary, when the target number of the multiple result information is larger, it indicates that the instability of the sample data is stronger, and the second loss weight is larger at this time; when all the result information is different, it indicates that the result of the sample data is the most unstable, and the corresponding second loss weight is the largest.

[0095] Optionally, for the sample data x i , the corresponding first label is y i , is the sample data x i Add thought chain prompt word p i , using the first language model f to train the sample data x i Perform k inferences to obtain multiple result information y i1 ,y i2 ...y ikThe first loss weight is obtained according to the following formula:

[0096]

[0097] Among them, w ii is the second loss weight; U(y) represents the set {y i1 ,y i2 …y ik} the number of different values ​​(i.e. the target number); p(y l ) is {y i1 ,y i2 …y ik}The result information is y l When all the result information is the same value, it indicates that the result is stable, corresponding to the second loss weight w ii The minimum value is 0. When all the result information is different, the second loss weight w ii The value of is the largest.

[0098] Finally, after obtaining the first loss weight and the second loss weight, the first loss weight and the second loss weight are combined into the loss weight. The loss weight simultaneously characterizes the error ratio and instability of the sample data. Optionally, the loss weight is obtained according to the following formula:

[0099] W i =W ie +W ii

[0100] Among them, w i is the loss weight; w ie is the first loss weight; w ii is the second loss weight.

[0101] In the embodiment of the present application, the importance of the sample data is measured by using the loss weight that characterizes the error ratio and instability of reasoning, thereby reducing the loss weight of simple, low-quality sample data and increasing the loss weight proportion of difficult, high-quality samples during model fine-tuning optimization. Fine-tuning the model to be fine-tuned using the loss weight as the target weight can improve the influence of the sample data on the model to be fine-tuned when the sample data is difficult and of high quality, effectively improving the effect of the target model after fine-tuning.

[0102] S104: Determine thought chain data based on the multiple result information, the multiple process information and the first label of the sample data.

[0103] When the step S104 is specifically implemented, the introduction of the thought chain data is intended to improve the thinking ability of the model to be fine-tuned. For the same problem information in the sample data, introducing multiple thinking processes into the model based on different thought chain prompts can enable the model to solve the problem from different angles, thereby further improving the reasoning ability of the model. For each sample data, different thought chain prompts are added, and the thought chain data is constructed by using multiple larger-scale first language models as teachers. Specifically, based on the first label of the sample data, the multiple result information is screened to determine the correct result information matching the first label in the multiple result information; the process information corresponding to the correct result information is used as the thought chain data. Among them, since the model will perform different reasoning processes based on different thought chain prompts during the training process, and then obtain the same or different result information, when the result information is the same as the first label, it indicates that the process information corresponding to the result information can obtain the correct result, which has greater value for optimizing the model to be fine-tuned, so the process information is used as the thought chain data for optimizing the model to be fine-tuned.

[0104] For example, the question information of the sample data is "Xiao Ming bought 10 pencils and erasers, spending a total of 100 yuan. It is known that the pencil is 6 yuan each. What is the unit price of the eraser?", and the first label is "4"; the first thought chain prompt word is set for the sample data as "first reason step by step, and then get the calculation result", and the first process information obtained by the first language model based on the first thought chain prompt word and the sample data is "Xiao Ming spent a total of 10*6=60 yuan on pencils, so it can be obtained that the total cost of buying erasers is 100-60=40 yuan, and Xiao Ming bought 10 erasers, each of which costs 40 / 10=4 yuan, so the result is 4", and the first result information is "4"; for the sample data, the second thinking chain prompt is set to "use the method of column equations to obtain the calculation result", and the second process information obtained by the first language model based on the second thinking chain prompt and the sample data is "assuming that the price of the eraser is x, according to the question, we can get: 10*(6+x)=100, and the calculation can get 6+x=10, x=4, so the result is 4", and the second result information is "4". Since the first result information and the second result information are the same as the first label, the first process information corresponding to the first result information and the second process information corresponding to the second result information are both determined as the thinking chain data. It should be noted that the above example is only an optional thinking chain data determination process given to enable those skilled in the art to better understand the solution of the embodiment of the present application. The specific thinking chain data can be determined according to actual conditions, and the embodiment of the present application does not make specific restrictions here.

[0105] S105. Determine a target loss based on the thought chain data and the loss weight, and fine-tune the model to be fine-tuned based on the target loss to obtain a target language model.

[0106] When the step S105 is specifically implemented, the target loss is determined based on the thought chain data and the loss weight, and the model to be fine-tuned is fine-tuned based on the target loss to obtain the target language model. Specifically, the second label of the sample data is first obtained, and the second label is the standard value of the process information corresponding to the sample data; based on the thought chain data and the second label, the loss value of the thought chain data is obtained, and the loss value is used to characterize the gap between the thought chain data and the second label. Although the thought chain data is the process information corresponding to the result information matching the first label, there is still a situation where the process information is wrong but the result information is correct. Therefore, by obtaining the loss value of the thought chain data, the error rate of the model to be fine-tuned in obtaining the correct process information can be reduced.

[0107] After obtaining the loss value of the thought chain data, the loss value of the thought chain data is weighted based on the loss weight to determine the target loss. Optionally, the target loss is obtained according to the following formula:

[0108]

[0109] Wherein, L is the target loss; x i is the sample data; i is the loss weight; y′ i is the second label; logP(y′ i |x i ) is the loss value of the thought chain data.

[0110] Finally, the thought chain data is used as the input data of the model to be fine-tuned, and the model to be fine-tuned is fine-tuned based on the target loss L to obtain the target model. Specifically, the thought chain data is input into the data to be fine-tuned, and appropriate training parameters for the thought chain data are set, wherein the training parameters include but are not limited to learning rate, batch size, number of training rounds, etc.; then the model to be fine-tuned is fine-tuned and optimized. During the fine-tuning process, the model to be fine-tuned will learn how to better obtain the correct process information to meet the model requirements; finally, the performance of the fine-tuned model is evaluated on the validation data set. If the performance meets the requirements, the test data set can be used for the final evaluation; if the performance is not good, it is necessary to adjust the training parameters or try other fine-tuning strategies until the target model is obtained.

[0111] In an embodiment of the present application, when obtaining the thought chain data of the sample data and the loss weight, a larger-scale first language model is used to obtain them, and the model to be fine-tuned can be exempted from manual labeling, which effectively reduces the manpower and material resources required for fine-tuning the model to be fine-tuned and reduces the cost of fine-tuning.

[0112] In order to enable those skilled in the art to understand the present application more clearly, the language model fine-tuning method described in the present application is now described in detail through the following examples.

[0113] Figure 2 FIG. 1 is a schematic diagram showing the steps of a language model fine-tuning method proposed in an embodiment of the present application. Figure 2 As shown, firstly, a large amount of public data is collected for model training, and the data sources include but are not limited to public data sets of literature research and some mathematics test papers, etc., to constitute the sample data; after adding thinking chain prompt words to each sample data, the first language model is input, and the first language model is used to perform multiple reasonings to obtain multiple result information and process information generated by each sample data based on each thinking chain prompt word.

[0114] Subsequently, a first loss weight is determined based on the plurality of result information and the first label of the sample, a second loss weight is determined based on the plurality of result information and the first label of the sample, and the first loss weight and the second loss weight are added to obtain the loss weight. Then, based on the first label of the sample data, the plurality of result information is screened to determine the correct result information that matches the first label in the plurality of result information, and the process information corresponding to the correct result information is used as the thought chain data.

[0115] Finally, the thinking chain data is input into the model to be fine-tuned, and the target loss is obtained by multiplying the loss weight of each sample data by the loss value of the corresponding thinking chain data. The model parameters and architecture of the model to be fine-tuned are updated based on the target loss to obtain the target model.

[0116] The present application provides a language model fine-tuning method, device, electronic device and storage medium, the method comprising: obtaining sample data, the sample data comprising multiple data with different data quality; performing multiple reasoning on the sample data based on a first language model to generate multiple result information and multiple process information corresponding to each result information; determining a loss weight based on the multiple result information and a first label of the sample data, the loss weight being used to characterize the data quality of the sample data; determining thought chain data based on the multiple result information, the multiple process information and the first label of the sample data; determining a target loss based on the thought chain data and the loss weight, and fine-tuning the model to be fine-tuned based on the target loss to obtain a target language model. The present application generates loss weights and thought chain data based on the first language model, and pays attention to the quality of the sample data and the logical thinking ability of the model reasoning in the optimization of the model to be fine-tuned through the loss weight and the thought chain data, respectively, effectively improving the interpretability and mathematical ability of the language model and reducing the model error rate.

[0117] Based on the same inventive concept, the present application embodiment discloses a language model fine-tuning device. Figure 3 FIG. 4 shows a schematic diagram of the structure of a language model fine-tuning device proposed in an embodiment of the present application. Figure 3 As shown, the device comprises:

[0118] A first acquisition module is used to acquire sample data, wherein the sample data includes multiple types of data with different data qualities;

[0119] An inference module, configured to perform multiple inferences on the sample data based on the first language model, and generate multiple result information and multiple process information corresponding to each result information;

[0120] A loss weight module, used to determine a loss weight based on the multiple result information and the first label of the sample data, wherein the loss weight is used to characterize the data quality of the sample data;

[0121] A thought chain data module, used for determining thought chain data based on the plurality of result information, the plurality of process information and the first label of the sample data;

[0122] The fine-tuning module is used to determine the target loss based on the thought chain data and the loss weight, and fine-tune the model to be fine-tuned based on the target loss to obtain a target language model.

[0123] In an optional implementation, the reasoning module includes:

[0124] A thought chain prompt word submodule, used for acquiring a plurality of thought chain prompt words, wherein the thought chain prompt words are used for representing the expected reasoning process of the first language model;

[0125] The reasoning submodule is used to add thought chain prompt words to the sample data, and use the thought chain prompt words as guiding information to perform multiple reasoning on the sample data based on the first language model to generate the multiple result information and multiple process information corresponding to each result information.

[0126] In an optional implementation, the loss weight module includes:

[0127] A first loss weight submodule, configured to determine a first loss weight based on the plurality of result information and the first label of the sample, wherein the first loss weight is used to characterize an error ratio of the plurality of result information relative to the first label;

[0128] A second loss weight submodule, configured to determine a second loss weight based on the plurality of result information and the first label of the sample, wherein the second loss weight is used to characterize the stability of multiple inferences performed on the sample data based on the first language model;

[0129] A loss weight submodule, used to combine the first loss weight and the second loss weight into the loss weight.

[0130] In an optional implementation manner, the first loss weight submodule includes:

[0131] an error ratio determination unit, configured to compare each result information with the first label to generate an error ratio corresponding to each result information;

[0132] A first loss weight unit, used to determine the first loss weight based on the error proportions respectively corresponding to all the result information and the amount of the result information;

[0133] If the result information is the same as the first label, the error ratio corresponding to the result information is determined to be 0; if the result information is different from the first label, the error ratio corresponding to the result information is determined to be 1.

[0134] In an optional implementation manner, the second loss weight submodule includes:

[0135] a target quantity determining unit, configured to obtain a target quantity based on the plurality of result information and the first label of the sample, wherein the target quantity is the number of different results in the plurality of result information;

[0136] a target probability determination unit, configured to obtain a probability that each result information matches the first label as a target probability corresponding to each result information;

[0137] The second loss weight unit is used to determine the second loss weight based on the target quantity and the target probability corresponding to each result information.

[0138] In an optional implementation manner, the thought chain data module includes:

[0139] A correct result information screening submodule, used to screen the plurality of result information based on the first label of the sample data, and determine the correct result information matching the first label among the plurality of result information;

[0140] The thought chain data determination submodule is used to use the process information corresponding to the correct result information as the thought chain data.

[0141] In an optional implementation, the fine-tuning module includes:

[0142] A second label acquisition submodule, used to acquire a second label of the sample data, where the second label is a standard value of process information corresponding to the sample data;

[0143] A loss value acquisition submodule, used for acquiring the loss value of the thought chain data based on the thought chain data and the second label;

[0144] The target loss submodule is used to perform weighted processing on the loss value of the thought chain data based on the loss weight to determine the target loss.

[0145] Based on the same inventive concept, the present application embodiment discloses an electronic device, Figure 4 A schematic diagram of an electronic device proposed in an embodiment of the present application is shown, such as Figure 4 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory 110 and the processor 120 are connected to each other through a bus communication, the memory 110 stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set can be loaded and run on the processor 120 to implement the steps in the language model fine-tuning method disclosed in the embodiment of the present application.

[0146] Based on the same inventive concept, an embodiment of the present application discloses a computer-readable storage medium, on which is stored at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the steps in the language model fine-tuning method disclosed in the embodiment of the present application.

[0147] Based on the same inventive concept, an embodiment of the present application discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the language model fine-tuning method disclosed in the embodiment of the present application.

[0148] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0149] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0153] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0154] The above is a detailed introduction to a language model fine-tuning method, device, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A language model fine-tuning method, characterized in that: The method comprises: Acquire sample data, where the sample data includes multiple types of data with different data qualities; Perform multiple inferences on the sample data based on the first language model to generate multiple result information and multiple process information corresponding to each result information; Determine a loss weight based on the multiple result information and the first label of the sample data, where the loss weight is used to characterize the data quality of the sample data; Determine thought chain data based on the plurality of result information, the plurality of process information and the first label of the sample data; Based on the thought chain data and the loss weight, a target loss is determined, and the model to be fine-tuned is fine-tuned based on the target loss to obtain a target language model.

2. The language model fine-tuning method according to claim 1, characterized in that: The performing multiple inferences on the sample data based on the first language model to generate multiple result information and multiple process information corresponding to each result information respectively includes: Acquire a plurality of thought chain prompt words, wherein the thought chain prompt words are used to represent an expected reasoning process of the first language model; Add thought chain prompt words to the sample data, use the thought chain prompt words as guiding information, perform multiple inferences on the sample data based on the first language model, and generate the multiple result information and multiple process information corresponding to each result information.

3. The language model fine-tuning method according to claim 1, characterized in that: The first label is a standard value of the result information corresponding to the sample data, and determining the loss weight based on the plurality of result information and the first label of the sample data includes: Determine a first loss weight based on the multiple result information and the first label of the sample, where the first loss weight is used to characterize an error ratio of the multiple result information relative to the first label; Determine a second loss weight based on the multiple result information and the first label of the sample, where the second loss weight is used to characterize the stability of multiple inferences performed on the sample data based on the first language model; The first loss weight and the second loss weight are combined into the loss weight.

4. The language model fine-tuning method according to claim 3, characterized in that: The determining a first loss weight based on the plurality of result information and the first label of the sample includes: Compare each result information with the first label to generate an error ratio corresponding to each result information; Determining the first loss weight based on the error proportions respectively corresponding to all the result information and the amount of the result information; If the result information is the same as the first label, the error ratio corresponding to the result information is determined to be 0; if the result information is different from the first label, the error ratio corresponding to the result information is determined to be 1.

5. The language model fine-tuning method according to claim 3, characterized in that: The determining a second loss weight based on the plurality of result information and the first label of the sample comprises: Based on the multiple result information and the first label of the sample, obtaining a target number, where the target number is the number of different results in the multiple result information; Obtaining the probability that each result information matches the first label as the target probability corresponding to each result information; The second loss weight is determined based on the target quantity and the target probability corresponding to each result information.

6. The language model fine-tuning method according to claim 1, characterized in that: The determining of thought chain data based on the plurality of result information, the plurality of process information and the first label of the sample data comprises: Based on the first label of the sample data, the plurality of result information are screened to determine correct result information matching the first label among the plurality of result information; The process information corresponding to the correct result information is used as the thinking chain data.

7. The language model fine-tuning method according to claim 1, characterized in that: The determining of the target loss based on the thought chain data and the loss weight comprises: Acquire a second label of the sample data, where the second label is a standard value of process information corresponding to the sample data; Based on the thought chain data and the second label, obtaining a loss value of the thought chain data; Based on the loss weight, the loss value of the thought chain data is weighted to determine the target loss.

8. A language model fine-tuning device, characterized in that: The device comprises: A first acquisition module is used to acquire sample data, wherein the sample data includes multiple types of data with different data qualities; An inference module, configured to perform multiple inferences on the sample data based on the first language model, and generate multiple result information and multiple process information corresponding to each result information; A loss weight module, used to determine a loss weight based on the multiple result information and the first label of the sample data, wherein the loss weight is used to characterize the data quality of the sample data; A thought chain data module, used for determining thought chain data based on the plurality of result information, the plurality of process information and the first label of the sample data; The fine-tuning module is used to determine the target loss based on the thought chain data and the loss weight, and fine-tune the model to be fine-tuned based on the target loss to obtain a target language model.

9. An electronic device, characterized in that: It includes a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps in the language model fine-tuning method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps in the language model fine-tuning method described in any one of claims 1 to 7 are implemented.

Images