A fine-tuning-based small-data large language model alignment optimization method and storage medium

Abstract

The application relates to a fine-tuning-based large language model alignment optimization method and a storage medium, and relates to a fine-tuning-based large language model alignment optimization method and a storage medium.The application aims to solve the problems that the existing large model fine-tuning requires a large amount of labeled data, consumes a large amount of computing resources, requires higher fine-tuning cost and development time.The process is as follows:1, a data set for a specific target task is constructed and divided into two parts, one part is a fine-tuning data set, and the other part is an alignment data set;meanwhile, a problem data set for the specific target task is constructed;2, a fine-tuning model is obtained;3, a judgment model is obtained;4, the problems in the alignment data set are input into the fine-tuning model, the fine-tuning model outputs a result, and the similarity between the fine-tuning model output result and the target result is calculated;the fine-tuning model output result is input into the judgment model, and the judgment model outputs a judgment result;5, an alignment-optimized target model is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to a large language model alignment optimization method based on small data fine-tuning and a storage medium. BACKGROUND

[0002] At present, with the rapid development of artificial intelligence technology, large language models (LLMs) such as GPT, BERT, ChatGLM, etc. have shown excellent performance in various natural language processing tasks. Deep learning-based NLP models usually have billions or even tens of billions of parameters and are suitable for various language tasks. However, building such models alone usually requires a large amount of training data and computing resources, and using commercial large language models may cause internal private data leakage. This poses a challenge to many application scenarios, especially for enterprises or individual users with limited resources or research areas.

[0003] In practical applications, large models for specific domain tasks often need to be adjusted according to task or domain requirements to achieve better performance. To meet the above needs, fine-tuning technology is widely used. Fine-tuning refers to further training a model using data specific to a task based on a pre-trained model. The fine-tuning method used in this method is the LoRA method. LoRA is a technology for fine-tuning large language models, aiming to efficiently adjust large pre-trained models to adapt to specific tasks without fully updating the entire model. The main feature of LoRA is to achieve fine-tuning by performing low-rank decomposition on model weights, thereby reducing the computational and storage overhead.

[0004] Large model alignment technology is designed to make the output of large language models more consistent with the expectations of human users, ethical standards, and actual application needs. Alignment mainly addresses the deviations and unexpected problems that may occur in the actual application of large models to ensure that the model output is both accurate and consistent with human values and norms.

[0005] Defects of prior art

[0006] 1. High data dependence: Most existing large model fine-tuning techniques that have good fine-tuning effects require a large amount of labeled data to support. In the context of private large models, due to the sensitivity and specificity of data, data collection and labeling are often limited, making it difficult to obtain a large amount of diverse labeled data for effective fine-tuning.

[0007] 2. Poor fine-tuning effect with small data: Under existing technology, fine-tuning with small data sets alone cannot guarantee the stability of the training results, and also cannot guarantee the generalization ability of the model itself. This makes the final model fine-tuning effect unpredictable, and may not perform well in specific tasks, or may cause the model to hallucinate or output that does not conform to human standards, i.e., poor alignment effect.

[0008] 3. Evaluation construction is complicated: current large model evaluation often needs to be evaluated by multiple different function models, which may require the construction of a huge dataset and consume a large amount of computing resources. For private deployment of large models, this often means higher deployment costs and development time.

[0009] 4. If better results are expected through the knowledge distillation method, single teacher model distillation cannot achieve good results, and multiple teacher models require a large amount of computing resources, which is a heavy burden for small businesses or individuals. SUMMARY

[0010] The purpose of the present application is to solve the problems of large amount of labeled data required for existing large model fine-tuning, large amount of computing resources required, higher fine-tuning cost and development time, and propose a large language model alignment optimization method based on small data amount fine-tuning and storage medium.

[0011] The specific process of a large language model alignment optimization method based on small data amount fine-tuning is as follows:

[0012] Step 1: Construct a dataset for a specific target task, and divide the dataset for the specific target task into two parts, one part is a fine-tuning dataset, and the other part is an alignment dataset;

[0013] Meanwhile, construct a question dataset for a specific target task;

[0014] Step 2: Fine-tune the large language model based on the fine-tuning dataset using LoRA to obtain a fine-tuning model;

[0015] Step 3: Obtain a judgment dataset based on the question dataset constructed in step 1 and the fine-tuning model obtained in step 2;

[0016] Fine-tune the large language model through the judgment dataset to obtain a judgment model;

[0017] Step 4: Input the questions in the alignment dataset into the fine-tuning model of step 2, and output the results of the fine-tuning model, and calculate the similarity between the output results of the fine-tuning model and the target results;

[0018] Input the output results of the fine-tuning model into the judgment model obtained in step 3, and output the judgment results of the judgment model;

[0019] Step 5: Calculate the weighted average score of the text similarity and the judgment result corresponding to each output obtained in step 4;

[0020] Determine whether the weighted average score is greater than or equal to a threshold value;

[0021] If yes, the fine-tuning model of step 2 is the aligned optimized target model;

[0022] If not, the best one is selected according to the weighted average part, a new fine-tuning dataset is constructed, and steps 2, 4 and 5 are repeatedly executed until the aligned optimized target model is obtained.

[0023] A computer storage medium, the storage medium stores at least one instruction, the at least one instruction is loaded and executed by a processor to realize the one kind of based on small data amount fine-tuning big language model alignment optimization method.

[0024] The beneficial effects of the present application are:

[0025] The present application provides a fine-tuning based on small data amount private big model alignment optimization method, which meets the demand of private big model construction in specific scenarios. Compared with the traditional big model fine-tuning, this method uses a smaller number of models for optimization and alignment. This method can balance the accuracy of the answer required by private deployment in specific scenarios and the generalization ability of big models. It can ensure that the fine-tuned big model can answer as expected in specific scenarios, while the content of the answer meets the language habits and moral standards of people. Compared with general alignment technology, the model training cost is lower, the implementation process is simple, and a large amount of labeled data is not required. Only small data sets are used for fine-tuning, which can ensure the stability of the training results, and also ensure the generalization ability of the model itself, without consuming a large amount of computing resources, and without higher fine-tuning cost and development time. It is a strong support for private deployment of big models. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The flowchart of the present application;

[0027] Figure 2 The basic flowchart of model training;

[0028] Figure 3 The fine-tuning model alignment optimization flowchart;

[0029] Figure 4 The model LoRA fine-tuning flowchart. DETAILED DESCRIPTION

[0030] Embodiment one: the specific process of the one kind of based on small data amount fine-tuning big language model alignment optimization method in the embodiment is:

[0031] Step 1: Construct a dataset for a specific target task (such as text classification or information extraction, etc.), and divide the dataset for the specific target task into two parts, one part is the fine-tuning dataset used in step 2, and the other part is the alignment dataset used in step 5;

[0032] Meanwhile, a problem data set for a specific target task is constructed;

[0033] Step 2: Fine-tuning the large language model based on the fine-tuning data set using LoRA to obtain a fine-tuned model;

[0034] Step 3: Obtain a judgment data set based on the problem data set constructed in step 1 and the fine-tuned model obtained in step 2;

[0035] Fine-tune the large language model based on the judgment data set to obtain a judgment model;

[0036] Step 4: Input the problem in the alignment data set into the fine-tuned model in step 2, and output the result of the fine-tuned model, and calculate the similarity between the output result of the fine-tuned model and the target result;

[0037] Input the output result of the fine-tuned model into the judgment model obtained in step 3, and output the judgment result of the judgment model;

[0038] Step 5: Calculate the weighted average score of the text similarity and the judgment result corresponding to each output obtained in step 4;

[0039] Determine whether the weighted average score is greater than or equal to a threshold value;

[0040] If yes, the fine-tuned model in step 2 is the aligned and optimized target model;

[0041] If not, select the best one according to the weighted average part, construct a new fine-tuning data set, and repeat steps 2, 4, and 5 until the aligned and optimized target model is obtained.

[0042] Step 6: The user inputs the text problem to be tested into the aligned and optimized target model, and the aligned and optimized target model generates text output.

[0043] Specific implementation method two: The difference between this implementation method and the specific implementation method one is that in step 1, the data set for a specific target task (such as text classification or information extraction, etc.) is constructed, and the data set for a specific target task is divided into two parts, one part is the fine-tuning data set used in step 2, and the other part is the alignment data set used in step 5;

[0044] Meanwhile, a problem data set for a specific target task is constructed;

[0045] The specific process is as follows:

[0046] Step 11: Construct a data set for a specific target task (such as text classification or information extraction, etc.); The process is as follows:

[0047] For a specific target task (such as text classification or information extraction, etc.), relevant materials are collected and sorted, the input of the large language model and the expected output of the large language model are set, a data set is constructed based on the input of the large language model and the expected output of the large language model, and the data set format is set as <input, output>;

[0048] wherein <input> represents the input data of the large language model for a specific target task, which can include text, instruction, and other types of input content; <output>represents the output of a large language model expected for a specific target task, i.e., the ideal solution or answer content of the task;

[0049] For example: <What is artificial intelligence? Artificial intelligence (AI) refers to the technology by which computer systems perform tasks that usually require human intelligence, including learning, reasoning, perception, language understanding, etc.>

[0050] Step 12: Divide the dataset constructed in step 11 into two parts, one for the fine-tuning dataset used in step 2 and the other for the alignment dataset used in step 5.

[0051] The dataset constructed in step 11 is divided into two parts in a reasonable and scattered manner, following the principles of randomness and coverage, i.e., each part of the data should contain task-critical common information and be as dispersed as possible on the dataset constructed in step 11 to avoid overfitting of the model to specific problems. At the same time, a small amount of cross-data is allowed to exist during the division process. After division, one part is the fine-tuning dataset used in step 2 and the other part is the alignment dataset used in step 5.

[0052] Step 13: Extract the key points from the dataset constructed in step 11, construct a question dataset, and the question dataset only includes input <input> .

[0053] For example:

[0054] The key points can be input that is not in the previous dataset but is related to similar problems, or questions that are deduced from the information in the input and output in the previous dataset, or questions in the <input, output> of step 11 <input> ;

[0055] For input that is not in the previous dataset but is related to similar problems: For example, related problems in the field of artificial intelligence, the dataset only has problems strongly related to the field of artificial intelligence, and the question dataset can be related problems in the field of artificial intelligence such as <input> <What is reinforcement learning?>

[0056] For reverse deduction: For example, <input, output> is <What is artificial intelligence? Artificial intelligence (AI) refers to the technology by which computer systems perform tasks that usually require human intelligence, including learning, reasoning, perception, language understanding, etc.>; can deduce the question <How does artificial intelligence achieve learning and reasoning?>

[0057] According to the dataset constructed in step 11, extract the key points in the task, and combine the answers that the fine-tuned model should be able to achieve and make, to construct a question dataset, which only includes <input> . The corresponding <output>Answer and judge the fine-tuned model in step 3.

[0058] Other steps and parameters are the same as in the first embodiment.

[0059] The third embodiment is different from the first or second embodiment in that the large language model is fine-tuned based on the fine-tuning data set in step 2 to obtain a fine-tuned model; the specific process is as follows:

[0060] For the method of fine-tuning the large model, the fine-tuning methods of the fine-tuned model in step 2 and the evaluation model in step 3 in this method are based on the LoRA method. Other fine-tuning methods can also be tried to obtain fine-tuned models and evaluation models.

[0061] Step 21: Select an open-source large language model (such as Qwen2, GLM4, etc.) as the base model, and the weight of the base model is W0;

[0062] Step 22: Freeze the weight of the base model as W0, W0 = r x i;

[0063] r x i represents the row and column of the weight W0;

[0064] Step 23: Initialize two low-rank matrices A and B, A = r x t, B = t x i;

[0065] r x t represents the row and column of matrix A respectively; t x i represents the row and column of matrix B respectively; t is much smaller than r and i;

[0066] Multiply the initialized low-rank matrices A and B to replace the weight W0 of the base model; obtain the base model with A and B introduced;

[0067] Set the initial values of A and B to random numbers close to zero, so that the model can make subtle adjustments based on the pre-training weights;

[0068] Step 24: Input the fine-tuning data set into the base model with A and B introduced, keep other parameters in the base model unchanged except A and B, only train A and B until convergence, and obtain the trained base model with A and B introduced;

[0069] A and B in the trained base model with A and B introduced are the trained A and B;

[0070] Multiply the trained A and B to obtain the new weight increment ΔW;

[0071] Step 25: Fuse the weight W0 of the base model frozen in step 22 and the new parameter weight ΔW obtained in step 24 to obtain the fused weight matrix W;

[0072] The fused weight matrix W = AW + W0 = AB + W0

[0073] Replace the weight of the base model with the fused weight matrix W to obtain a fine-tuned model.

[0074] First, select a relatively ideal open-source large language model (such as Qwen2, GLM4, etc.) as the base model, load the base model, and load its pre-training weight W0. When fine-tuning, the pre-training weight W0 is frozen to ensure that the general ability of the original model is not affected, and to avoid overfitting of the model caused by fine-tuning, which reduces the generalization ability. Then start training the new weight increment AW. The method is to initialize two low-rank matrices A and B, so that AW = AB, and set the initial value of the low-rank matrices A and B to a small random number close to zero, so that the model can make subtle adjustments based on the pre-training weight. During the fine-tuning process, only the low-rank matrices A and B are optimized, and only the A and B parameters of the low-rank matrices need to be saved after LoRA training. Fuse the original pre-training weight with the trained weight increment. Fuse the pre-training weight W0 and the new weight increment AW to obtain a new weight matrix W = AW + W0 = AB + W0. For example, when inferring for a given input x, the output after adding LoRA is h = (AW + W0)x, which ensures that it can be adjusted for specific tasks while maintaining general ability.

[0075] The other steps and parameters are the same as in embodiment one or two.

[0076] Embodiment four: The difference between this embodiment and one of embodiments one to three is that in step 3, the evaluation data set is obtained based on the question data set constructed in step 1 and the fine-tuned model obtained in step 2.

[0077] Fine-tune the large language model based on the evaluation data set to obtain an evaluation model.

[0078] The specific process is as follows:

[0079] Step 31: Based on the question data set constructed in step 13 and the fine-tuned model obtained in step 2, obtain a preliminary evaluation data set.

[0080] Step 32: Score the preliminary evaluation data set corresponding to each question obtained in step 31 to obtain a rationality score and an alignment score, and construct a list type [answer, score1, score2];

[0081] Add the rationality score and the alignment score corresponding to each question to obtain the final evaluation result; sort the final evaluation result corresponding to each question from high to low to form a complete evaluation data; until a complete evaluation data set corresponding to all questions is formed;

[0082] The complete evaluation dataset stores the list <input, output> and output = [answer n , score1 n , score2 n ];

[0083] n answers corresponding to each question, and 1 rationality score and 1 alignment score corresponding to each answer;

[0084] Add the corresponding rationality score and alignment score to obtain n final evaluation results corresponding to each question;

[0085] The preliminary evaluation dataset obtained in step 31 is submitted to the evaluators for evaluation, and the evaluators score the answers to the questions; the evaluation standard is:

[0086] Rationality score <score1>: Answer whether it is logical, whether it is reasonable to solve the problem.

[0087] alignment score <score2>: answer whether it conforms to human intention, values, whether there are harmful, inaccurate or biased results.

[0088] The judges give each answer a corresponding rationality score and alignment score based on the above criteria, build a list type answer, score1, score2], and sort from high to low according to the total score score = score1 + score2, to form a complete evaluation dataset.

[0089] Step 33: Based on the complete evaluation dataset obtained in step 32, LoRA is used to fine-tune the large language model to obtain an evaluation model.

[0090] The other steps and parameters are the same as one of the first to third embodiments.

[0091] Embodiment five: This embodiment is different from one of the first to fourth embodiments in that the preliminary evaluation dataset is obtained based on the question dataset constructed in step 13 and the fine-tuned model obtained in step 2 in step 31; the specific process is:

[0092] Step 311: input a certain question in the question dataset constructed in step 13 into the fine-tuned model obtained in step 2, and the fine-tuned model obtains an answer;

[0093] Step 312: repeat step 311 n times to obtain n different answers corresponding to the same question;

[0094] Organize the same question and n different answers to obtain a preliminary evaluation dataset <input, [asnswer1, …, answern]>;

[0095] Wherein <input> is the question, <answer n > is the n different answers generated by the fine-tuned model under the same question;

[0096] Step 313: repeat steps 311 to 312 until all questions in the question dataset constructed in step 13 are input into the fine-tuned model obtained in step 2 to obtain a preliminary evaluation dataset corresponding to all questions.

[0097] The other steps and parameters are the same as one of the first to fourth embodiments.

[0098] Embodiment six: This embodiment is different from one of the first to fifth embodiments in that the evaluation model is obtained by fine-tuning the large language model based on the complete evaluation dataset obtained in step 32 in step 33; the specific process is:

[0099] Step 331: Select an open-source large language model (such as Qwen2, GLM4, etc.) as the base model, and the weight of the base model is W0;

[0100] Step 332: Freeze the weight of the base model as W0, W0 = r x i;

[0101] r x i represents the row and column of the weight W0;

[0102] Step 333: Initialize two low-rank matrices A and B, A = r x t, B = t x i;

[0103] r x t represents the row and column of matrix A respectively; t x i represents the row and column of matrix B respectively;

[0104] Multiply the initialized low-rank matrices A and B to replace the weight W0 of the base model; obtain the base model with A and B introduced;

[0105] Set the initial values of A and B to random numbers close to zero, so that the model can make subtle adjustments based on the pre-training weights;

[0106] Step 334: Input the fine-tuning data set into the base model with A and B introduced, and keep other parameters in the base model unchanged except A and B, only train A and B until convergence, and obtain the trained base model with A and B introduced;

[0107] A and B in the trained base model with A and B introduced are the trained A and B;

[0108] Multiply the trained A and B as the new weight increment AW;

[0109] Step 335: Fuse the weight W0 of the base model frozen in step 332 and the new parameter weight AW obtained in step 334 to obtain the fused weight matrix W;

[0110] The fused weight matrix W = AW + W0 = A B + W0

[0111] Replace the weight of the base model with the fused weight matrix W to obtain the evaluation model;

[0112] Use the evaluation data set constructed in step 32 to fine-tune the base model, learn the preferences of human evaluators in this task, and train to obtain the evaluation model. After fine-tuning is completed, ensure that the fine-tuned evaluation model can complete the task in subsequent steps and can give the corresponding reasonableness score and alignment score after obtaining an input.

[0113] The other steps and parameters are the same as one of the first to fifth specific embodiments.

[0114] Specific implementation seven: different from one of the specific implementations one to six, the step 4 inputs the fine-tuning model of the step 2 with the aligned data set question, the fine-tuning model outputs the result, and the similarity between the fine-tuning model output result and the target result is calculated;

[0115] The fine-tuning model output result is input into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result;

[0116] The specific process is as follows:

[0117] Step 41: input the fine-tuning model of step 2 with the aligned data set question, the fine-tuning model outputs the result, and the similarity between the fine-tuning model output result and the target result (the output corresponding to the data set input of step 1) is calculated;

[0118] Step 42: input the text similarity corresponding to each output obtained in step 41 into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result; the evaluation result includes the rationality score and the alignment score.

[0119] The rationality and alignment of the output are evaluated by the evaluation model. Again, the multiple outputs obtained in step 41 are given to the evaluation model trained in step 3. The evaluation model gives each output a corresponding rationality score and alignment score, and these scores are sorted and summarized.

[0120] The other steps and parameters are the same as one of the specific implementations one to six.

[0121] Specific implementation eight: different from one of the specific implementations one to seven, the step 41 inputs the fine-tuning model of step 2 with the aligned data set question, the fine-tuning model outputs the result, and the similarity between the fine-tuning model output result and the target result (the output corresponding to the data set input of step 1) is calculated; the specific process is as follows:

[0122] Step 411: input the first <input> of the aligned data set obtained in step 12 into the fine-tuning model obtained in step 2, and the fine-tuning model outputs O1;

[0123] Repeat step 411m times to obtain the same <input> corresponding to the m different outputs;

[0124] Sort the same <input> and the corresponding m different outputs to obtain the same <input> corresponding to the preliminary data set m ]; O1 is the first <input1>corresponding output, O m to align the first data in the data set <input1>a corresponding mth output;

[0125] Step 412: repeat the execution of step 411 until all <input> a corresponding preliminary data set;

[0126] Step 413: calculate the same <input> a corresponding m different outputs each output and a corresponding target answer in the aligned data set <output>text similarity of each of the m different outputs (s1, L, s <input> f text similarity of each of the m different outputs (s1, L, s m ), which will be used to measure the matching degree between the model output and the expected answer;

[0127] Step 414: Repeat step 413 until all <input> text similarity of each of the m different outputs (s1, L, s <input> f text similarity of each of the m different outputs (s1, L, s m ) in the alignment data set are obtained.

[0128] The other steps and parameters are the same as one of embodiments one to seven.

[0129] Embodiment nine: The difference between this embodiment and one of embodiments one to eight is that the weighted average score of the text similarity of each output obtained in step 4 and the evaluation result is calculated in step 5.

[0130] It is determined whether the weighted average score is greater than or equal to a threshold value.

[0131] If yes, the fine-tuning model of step 2 is the alignment-optimized target model.

[0132] If not, the part after the weighted average is selected, a new fine-tuning data set is constructed, and steps 2, 4 and 5 are repeated until the alignment-optimized target model is obtained.

[0133] The specific process is as follows:

[0134] Step 51: Calculate the weighted average score of the text similarity of each output obtained in step 4 and the evaluation result and sort them; the process is as follows:

[0135] The text similarity s f , the rationality score r f and the alignment score a f of each output obtained in step 4 are normalized to obtain the weight value S(s f ), the weight value S(r f ) and the weight value S(a f ) of the text similarity s f , the rationality score r f and the alignment score a f .

[0136] The text similarity of each of the m different outputs (s1, L, s <input> f m ​​​​), each text similarity corresponds to a rationality score r f and alignment score a f , the same <input> corresponding to m different output rationality scores r f and alignment score a f ;

[0137] The normalization method is normalization by a softmax function, and the calculation formula of the softmax function is as follows:

[0138]

[0139] Wherein, xt is the value to be normalized; xi is the similarity s f , rationality score r f or alignment score a f ;

[0140] After obtaining the corresponding weight, weighted average is performed, that is, after considering the scores of the three dimensions of similarity, rationality and alignment, the final score of each answer is obtained;

[0141] The weighted average score of the text similarity and the judgment result corresponding to each output obtained in step 4 is calculated, and the expression is:

[0142] V f =s f ×S(s f )+a f ×S(a f )+r f ×S(r f )

[0143] Wherein, V f represents the weighted average score of the text similarity and the judgment result corresponding to each output obtained in step 4;

[0144] The same <input> corresponding to m different outputs corresponding to the weighted average score V f value is sorted from high to low as V';

[0145] How many <input> correspond to how many V';

[0146] Step 52: select the second highest score in each V' as the final score V2, and record all the final scores to the overall score table;

[0147] Remove the highest score and the lowest score in the overall score table, and calculate the overall score expectation E(V) of the overall score table after removing the highest score and the lowest score;

[0148] If the overall score expectation value E(V) is greater than or equal to the set expectation threshold value, the fine-tuning model of step 2 is the target model after alignment optimization;

[0149] If the overall score expectation value E(V) is less than the set expectation threshold value, the maximum value in each V' in step 51 is selected to join the fine-tuning data set in step 12 to construct a new fine-tuning data set, and steps 2, 4 and 5 are repeatedly executed based on the new fine-tuning data set until the target model after alignment optimization is obtained.

[0150] After sorting in step 51, the final score value V2 of the second highest score output of the multiple outputs of the model is selected as the final score value of the output of the current problem, and the score value is recorded in the overall score table. The overall score table records the final score values obtained after inputting the data sets in steps 4 and 5 in the alignment data set in step 12. <input> The highest score and the lowest score are removed from the overall score table, and the expectation value E(V) of the overall score is calculated. If it is greater than the set expectation threshold value, it means that the model has reached the expected alignment optimization effect, and the target model is considered to be obtained, and the fine-tuning and alignment optimization process is ended. If it is less than the expectation threshold value, the higher score value is selected as the output of the model, and a new data set is constructed according to the fine-tuning data set in step 12. The fine-tuning model is retrained, and then step 4 is repeated to obtain new model output and perform evaluation until the target model meeting the expectation is obtained.

[0151] The other steps and parameters are the same as one of the first to eighth embodiments.

[0152] Embodiment Ten: The embodiment is a computer storage medium, and the storage medium stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the method of fine-tuning a large language model based on a small amount of data.

[0153] It should be understood that the instructions include a computer program product, software or computerized method corresponding to any method described in the present application; the instructions can be used to program a computer system or other electronic device. The computer storage medium can include a readable medium having instructions stored thereon, and can include but is not limited to a magnetic storage medium, an optical storage medium; a magneto-optical storage medium including a read-only memory (ROM), a random access memory (RAM), an erasable programmable memory (such as an EPROM and an EEPROM) and a flash memory layer, or other types of media suitable for storing electronic instructions.

[0154] The present application can also have other various embodiments. Those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, but these corresponding changes and modifications should all belong to the protection scope of the claims attached to the present application.< / output> < / output> < / output>

Claims

1. A method for aligning and optimizing large language models based on fine-tuning with small amounts of data, characterized in that: The specific process of the method is as follows: Step 1: Construct a dataset for a specific target task. Divide the dataset for the specific target task into two parts: a fine-tuning dataset and an alignment dataset. Simultaneously, construct a problem dataset targeting specific objective tasks; Step 2: Fine-tune the large language model using LoRA based on the fine-tuning dataset to obtain the fine-tuned model; Step 3: Obtain the evaluation dataset based on the problem dataset constructed in Step 1 and the fine-tuned model obtained in Step 2; The evaluation model is obtained by fine-tuning the large language model using the evaluation dataset; Step 4: Input the alignment dataset into the fine-tuning model from Step 2, fine-tune the model output, and calculate the similarity between the fine-tuning model output and the target result; Input the output of the fine-tuned model into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result; Step 5: Calculate the weighted average score of the text similarity and evaluation results for each output obtained in Step 4; Determine whether the weighted average score is greater than or equal to the threshold; If so, the fine-tuning model in step 2 is the aligned and optimized target model; If not, select the best based on the weighted average and construct a new fine-tuning dataset, repeating steps 2, 4, and 5 until the aligned and optimized target model is obtained. In step 3, the evaluation dataset is obtained based on the problem dataset constructed in step 1 and the fine-tuning model obtained in step 2; The evaluation model is obtained by fine-tuning the large language model using the evaluation dataset; The specific process is as follows: Step 31: Obtain a preliminary evaluation dataset based on the problem dataset constructed in Step 13 and the fine-tuned model obtained in Step 2; Step 32: Score the preliminary evaluation dataset corresponding to each question obtained in Step 31 to obtain a reasonableness score and an alignment score, and construct a list type [answer, score1, score2]. The reasonableness score and alignment score for each question are added together to obtain the final evaluation result; the final evaluation results for each question are sorted from high to low to form a complete set of evaluation data. Until a complete evaluation dataset is formed corresponding to all the questions; The complete evaluation dataset is stored in a centralized list.<input,output> ,output = [answer] n score1 n score2 n ]; Each question has n possible answers, and each answer has one reasonableness score and one alignment score. Step 33: Based on the complete evaluation dataset obtained in Step 32, LoRA is used to fine-tune the large language model to obtain the evaluation model.

2. The large language model alignment optimization method based on small data volume fine-tuning according to claim 1, characterized in that: In step 1, a dataset for a specific target task is constructed, which is divided into two parts: a fine-tuning dataset and an alignment dataset. Simultaneously, construct a problem dataset targeting specific objective tasks; The specific process is as follows: Step 11: Construct a dataset for a specific target task; the process is as follows: For a specific target task, the input and expected output of the large language model are defined. A dataset is constructed based on the input and expected output of the large language model, and the dataset format is defined as follows.<input,output> ; in <input> This represents the input data for a large language model targeting a specific objective task; <output> The output of a large language model represents the expected output for a specific target task;< / output> Step 12: Divide the dataset constructed in Step 11 into two parts: one part is the fine-tuning dataset, and the other part is the alignment dataset; Step 13: Extract the key points from the dataset constructed in Step 11 to build a problem dataset. The problem dataset only includes the input. <input> .

3. The large language model alignment optimization method based on small data volume fine-tuning according to claim 2, characterized in that: In step 2, LoRA is used to fine-tune the large language model based on the fine-tuning dataset to obtain the fine-tuned model; the specific process is as follows: Step 21: Select the open-source large language model as the base model, and the weight of the base model is W0; Step 22: Freeze the weights of the base model as W0, W0 = r × i; r×i represents the row and column of weight W0; Step 23: Initialize two low-rank matrices A and B, where A = r × t and B = t × i; r×t represent the rows and columns of matrix A, respectively; t×i represent the rows and columns of matrix B, respectively. Multiply the initialized low-rank matrices A and B, and replace the weights W0 of the base model; thus, a base model incorporating A and B is obtained. Step 24: Input the fine-tuning dataset into the base model that incorporates A and B. Keep all other parameters in the base model unchanged except for A and B. Train only A and B until convergence to obtain the trained base model that incorporates A and B. In the trained base model that incorporates A and B, A and B are the trained A and B; Multiply the trained A and B together to obtain the new weight increment ΔW; Step 25: Combine the weights W0 of the base model frozen in Step 22 with the new parameter weights ΔW obtained in Step 24 to obtain the combined weight matrix W; The fused weight matrix W = ΔW + W0 = AB + W0 The weights of the base model are replaced by the fused weight matrix W to obtain the fine-tuned model.

4. The large language model alignment optimization method based on small data volume fine-tuning according to claim 3, characterized in that: In step 31, a preliminary evaluation dataset is obtained based on the problem dataset constructed in step 13 and the fine-tuning model obtained in step 2; the specific process is as follows: Step 311: Input a question from the question dataset constructed in Step 13 into the fine-tuning model obtained in Step 2, and the fine-tuning model will obtain the answer; Step 312: Repeat step 311n times to obtain n different answers to the same question; By organizing the same question and n different answers, a preliminary evaluation dataset is obtained. <input,[asnswer1,…,answer n >; in <input> For the question, <answer n To fine-tune the model for generating n different answers to the same question; Step 313: Repeat steps 311 to 312 until all the problem datasets constructed in step 13 are input into the fine-tuning model obtained in step 2, and obtain the preliminary evaluation datasets corresponding to all problems.

5. The large language model alignment optimization method based on small data volume fine-tuning according to claim 4, characterized in that: In step 33, the large language model is fine-tuned using LoRA based on the complete evaluation dataset obtained in step 32 to obtain the evaluation model; the specific process is as follows: Step 331: Select the open-source large language model as the base model, and the weight of the base model is W0; Step 332: Freeze the weights of the base model as W0, W0 = r × i; r×i represents the row and column of weight W0; Step 333: Initialize two low-rank matrices A and B, A = r × t, B = t × i; r×t represent the rows and columns of matrix A, respectively; t×i represent the rows and columns of matrix B, respectively. Multiply the initialized low-rank matrices A and B, and replace the weights W0 of the base model; thus, a base model incorporating A and B is obtained. Step 334: Input the fine-tuning dataset into the base model that incorporates A and B. Keep all other parameters in the base model unchanged except for A and B. Train only A and B until convergence to obtain the trained base model that incorporates A and B. In the trained base model that incorporates A and B, A and B are the trained A and B; Multiply the trained A and B together to obtain the new weight increment ΔW; Step 335: Fuse the weights W0 of the base model frozen in step 332 with the new parameter weights ΔW obtained in step 334 to obtain the fused weight matrix W; The fused weight matrix W = ΔW + W0 = AB + W0 The weights of the base model are replaced by the fused weight matrix W to obtain the evaluation model.

6. The large language model alignment optimization method based on small data volume fine-tuning according to claim 5, characterized in that: In step 4, the problem of aligning the dataset is input into the fine-tuning model of step 2, the fine-tuning model outputs the result, and the similarity between the output result of the fine-tuning model and the target result is calculated. Input the output of the fine-tuned model into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result; The specific process is as follows: Step 41: Input the alignment dataset into the fine-tuning model from Step 2, fine-tune the model output, and calculate the similarity between the fine-tuning model output and the target result; Step 42: Input the text similarity corresponding to each output obtained in Step 41 into the evaluation model obtained in Step 3. The evaluation model outputs the evaluation results, which include the reasonableness score and the alignment score.

7. The large language model alignment optimization method based on small data volume fine-tuning according to claim 6, characterized in that: In step 41, the alignment dataset is centrally input into the fine-tuning model of step 2, the fine-tuning model outputs results, and the similarity between the fine-tuning model output and the target result is calculated; the specific process is as follows: Step 411: Take the first data from the aligned dataset obtained in Step 12. <input> Input the fine-tuned model obtained in step 2; the fine-tuned model outputs O1. Repeat step 411m times to obtain the same result. <input> The corresponding m different outputs; The same <input> Organize the m different outputs to obtain the same result. <input> Corresponding preliminary dataset <input1,[O1,…,O m O1 represents the first element in the aligned dataset. <input1>The corresponding output, O m To align the first one in the dataset <input1> The corresponding m-th output; Step 412: Repeat step 411 until all data in the aligned dataset is obtained. <input> The corresponding preliminary dataset; Step 413: Calculate the same <input> Each of the m distinct outputs corresponds to the target answer in the aligned dataset. <output>Text similarity, to obtain the same <input> The text similarity (s1,…,s) of each of the m different outputs. f ,…,s m );< / output> Step 414: Repeat step 413 until all data in the aligned dataset is obtained. <input> Each <input> The text similarity (s1,…,s) corresponding to the m different outputs f ,…,s m ).

8. The large language model alignment optimization method based on small data volume fine-tuning according to claim 7, characterized in that: In step 5, the weighted average score of the text similarity and evaluation results corresponding to each output obtained in step 4 is calculated. Determine whether the weighted average score is greater than or equal to the threshold; If so, the fine-tuning model in step 2 is the aligned and optimized target model; If not, select the best based on the weighted average and construct a new fine-tuning dataset, repeating steps 2, 4, and 5 until the aligned and optimized target model is obtained. The specific process is as follows: Step 51: Calculate and sort the weighted average score of the text similarity and evaluation results for each output obtained in Step 4; the process is as follows: For each output obtained in step 4, the text similarity s is calculated. f Reasonableness score r f And aligned fraction a f Normalization is performed to obtain the text similarity s. f The weight value S(s) f Reasonableness score r f The weight value S(r) f Alignment score a) f The weight value S(a) f ); same <input> The text similarity (s1,…,s) for each of the m different outputs f ,…,s m Each text similarity corresponds to a reasonableness score r. f And aligned fraction a f ,same <input> Corresponding to m different output reasonableness scores r f And aligned fraction a f ; The normalization method is to perform normalization using the softmax function; The weighted average score of text similarity and evaluation results for each output obtained in step 4 is calculated using the following expression: V f =s f ×S(s f )+a f ×S(a f )+r f ×S(r f ) Among them, V f This represents the weighted average score of text similarity and evaluation results for each output obtained in step 4; The same <input> The weighted average score V corresponding to the m different outputs f The values ​​are sorted from highest to lowest as V′; Step 52: Select the second highest score in each V′ as the final score V2, and record all final scores in the overall score table; Remove the highest and lowest scores from the overall rating table, and calculate the expected value of the overall rating table E(V) after removing the highest and lowest scores; If the overall expected score E(V) is greater than or equal to the set expected threshold, then the fine-tuning model in step 2 is the aligned and optimized target model. If the overall expected score E(V) is less than the set expected threshold, the maximum value of each V′ in step 51 is selected and added to the fine-tuning dataset in step 12 to construct a new fine-tuning dataset. Based on the new fine-tuning dataset, steps 2, 4, and 5 are repeated until the aligned and optimized target model is obtained.

9. A computer storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement a large language model alignment optimization method based on small data volume fine-tuning as described in any one of claims 1 to 8.

Images