Large language model alignment optimization method based on small data volume fine tuning and storage medium

Abstract

The invention relates to a large language model alignment optimization method based on small data volume fine tuning and a storage medium. The invention relates to the large language model alignment optimization method based on small data volume fine tuning and the storage medium. The invention aims to solve the problems that the existing large model fine tuning needs a large amount of annotation data, needs to consume a large amount of computing resources, and needs higher fine tuning cost and development time. The method comprises the steps that 1, a data set for a specific target task is constructed and divided into two parts, one part is a fine adjustment data set, and the other part is an alignment data set; meanwhile, a problem data set for a specific target task is constructed; 2, obtaining a fine tuning model; 3, obtaining an evaluation model; 4, inputting the alignment data set problem into a fine tuning model, outputting a result by the fine tuning model, and calculating the similarity between the output result of the fine tuning model and a target result; inputting an output result of the fine-tuning model into the evaluation model, and outputting an evaluation result by the evaluation model; and 5, obtaining the target model after alignment optimization.

Description

Technical Field

[0001] The present invention relates to a large language model alignment optimization method based on small data volume fine-tuning and a storage medium. Background Art

[0002] Nowadays, with the rapid development of artificial intelligence technology, large language models (LLMs) such as GPT, BERT, ChatGLM, etc. have demonstrated excellent performance in various natural language processing tasks. NLP models based on deep learning. Usually have billions or even hundreds of billions of parameters, suitable for various language tasks. However, building such models alone usually requires a large amount of training data and computing resources, and using commercial large language models may cause internal private data leakage. This poses a challenge to many application scenarios, especially for enterprises or individual users with relatively biased research fields or limited resources.

[0003] In practical applications, large models for tasks in specific fields usually need to be adjusted according to the requirements of the task or field to achieve better performance. In response to the above needs, fine-tuning technology is widely used. Fine-tuning refers to further training the model based on the pre-trained model using data from a specific task. The fine-tuning method used in this method is the LoRA method. LoRA is a technology for fine-tuning large language models, which aims to efficiently adjust large pre-trained models to adapt to specific tasks without the need to fully update the entire model. The main feature of LoRA is to achieve fine-tuning by performing low-rank decomposition of model weights, thereby reducing computational and storage overhead.

[0004] Large model alignment technology is designed to make the output generated by large language models more consistent with human user expectations, ethical standards, and practical application needs. Alignment mainly addresses the deviations and unexpected problems that may occur in large models in practical applications, ensuring that the model output is both accurate and consistent with human values ​​and norms.

[0005] Disadvantages of the prior art

[0006] 1. High data dependence: Most existing technologies for fine-tuning large models with good fine-tuning effects require a large amount of labeled data for support. In the case of private large models, due to the sensitivity and proprietary nature of the data, data collection and labeling are often limited, making it difficult to obtain a large amount of labeled data of various types for effective fine-tuning.

[0007] 2. Poor fine-tuning effect on small data sets: Under existing technologies, it is difficult to ensure the stability of training results and the generalization ability of the model itself by relying solely on fine-tuning on small data sets. This makes it impossible to reliably predict the final model fine-tuning effect, and it may not perform well in specific tasks. It may also cause the model to hallucinate or the output does not meet human standards, that is, the alignment effect is poor.

[0008] 3. Complicated evaluation and construction: At present, large model evaluation often requires multiple evaluation models with different functions, which may require the construction of huge data sets and consume a lot of computing resources. For private deployment of large models, this often means higher deployment costs and development time.

[0009] 4. If you want to achieve better results through knowledge distillation, a single teacher model distillation cannot achieve good results, and multiple teacher models require a lot of computing resources, which is a heavy burden for small businesses or individuals. Summary of the invention

[0010] The purpose of the present invention is to solve the problem that the existing large model fine-tuning requires a large amount of labeled data, consumes a large amount of computing resources, and requires higher fine-tuning costs and development time, and to propose a large language model alignment optimization method and storage medium based on small data amount fine-tuning.

[0011] A large language model alignment optimization method based on small data volume fine-tuning. The specific process is as follows:

[0012] Step 1: Build a dataset for a specific target task and divide the dataset into two parts: one is a fine-tuning dataset and the other is an alignment dataset.

[0013] At the same time, build problem datasets for specific target tasks;

[0014] Step 2: Fine-tune the large language model using LoRA based on the fine-tuning dataset to obtain a fine-tuning model;

[0015] Step 3: Obtain the evaluation dataset based on the problem dataset constructed in step 1 and the fine-tuned model obtained in step 2;

[0016] The large language model is fine-tuned using the judgment dataset to obtain the judgment model;

[0017] Step 4: Input the problem in the aligned dataset into the fine-tuned model in step 2, the fine-tuned model outputs the result, and calculates the similarity between the output result of the fine-tuned model and the target result;

[0018] The output result of the fine-tuning model is input into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result;

[0019] Step 5: Calculate the text similarity corresponding to each output obtained in step 4 and the weighted average score of the evaluation results;

[0020] Determine whether the weighted average score is greater than or equal to a threshold;

[0021] If yes, the fine-tuned model in step 2 is the target model after alignment optimization;

[0022] If not, select the best one based on the weighted average part, build a new fine-tuning dataset, and repeat steps 2, 4, and 5 until the aligned and optimized target model is obtained.

[0023] A computer storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement a large language model alignment optimization method based on small data amount fine-tuning.

[0024] The beneficial effects of the present invention are:

[0025] The present invention provides an alignment optimization method for a private large model based on fine-tuning of a small amount of data, which meets the needs of building a private large model in a specific scenario. Compared with the traditional large model fine-tuning, this method uses a smaller number of models for optimization and alignment. This method can take into account the answer accuracy in a specific scenario required by private deployment and the generalization ability unique to the large model, while ensuring that the fine-tuned large model can answer according to the expected effect in a specific scenario, and the answer content conforms to people's language habits and moral standards. Compared with general alignment technology, the model training cost is low, the implementation process is simple, and a large amount of labeled data is not required. It only relies on a small data set for fine-tuning, and can also ensure the stability of the training results. At the same time, it can also ensure the generalization ability of the model itself, without consuming a large amount of computing resources, nor requiring higher fine-tuning costs and development time. It is a strong support for the private deployment of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of the present invention;

[0027] Figure 2 Basic flowchart for model training;

[0028] Figure 3 Optimize the flow chart for fine-tuning model alignment;

[0029] Figure 4 Flowchart for fine-tuning the LoRA model. DETAILED DESCRIPTION

[0030] Specific implementation method 1: This implementation method is a large language model alignment optimization method based on small data volume fine-tuning. The specific process is:

[0031] Step 1: Build a dataset for a specific target task (such as text classification or information extraction), and divide the dataset for the specific target task into two parts, one is the fine-tuning dataset used in step 2, and the other is the alignment dataset used in step 5;

[0032] At the same time, build problem datasets for specific target tasks;

[0033] Step 2: Fine-tune the large language model using LoRA based on the fine-tuning dataset to obtain a fine-tuning model;

[0034] Step 3: Obtain the evaluation dataset based on the problem dataset constructed in step 1 and the fine-tuned model obtained in step 2;

[0035] The large language model is fine-tuned using the judgment dataset to obtain the judgment model;

[0036] Step 4: Input the problem in the aligned dataset into the fine-tuned model in step 2, the fine-tuned model outputs the result, and calculates the similarity between the output result of the fine-tuned model and the target result;

[0037] The output result of the fine-tuning model is input into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result;

[0038] Step 5: Calculate the text similarity corresponding to each output obtained in step 4 and the weighted average score of the evaluation results;

[0039] Determine whether the weighted average score is greater than or equal to a threshold;

[0040] If yes, the fine-tuned model in step 2 is the target model after alignment optimization;

[0041] If not, select the best one based on the weighted average part, build a new fine-tuning dataset, and repeat steps 2, 4, and 5 until the aligned and optimized target model is obtained.

[0042] Step 6: The user inputs the text question to be tested into the aligned and optimized target model, and the aligned and optimized target model generates text output.

[0043] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that in step 1, a data set for a specific target task (such as text classification or information extraction, etc.) is constructed, and the data set for the specific target task is divided into two parts, one part is the fine-tuning data set used in step 2, and the other part is the alignment data set used in step 5;

[0044] At the same time, build problem datasets for specific target tasks;

[0045] The specific process is:

[0046] Step 11: Build a dataset for a specific target task (such as text classification or information extraction); the process is:

[0047] For specific target tasks (such as text classification or information extraction, etc.), collect and organize relevant information, set the input of the large language model and the expected output of the large language model, build a data set based on the input of the large language model and the expected output of the large language model, and set the data set format to<input,output> ;

[0048] in <input> Represents the input data of a large language model for a specific target task, which may include input content such as text and instructions; <output>Represents the desired output of the large language model for a specific target task, i.e., the ideal solution or answer to the task;

[0049] For example: <What is artificial intelligence? , Artificial intelligence (AI) refers to the technology that enables computer systems to perform tasks that usually require human intelligence, including learning, reasoning, perception, language understanding, etc. >

[0050] Step 12: Divide the dataset constructed in step 11 into two parts, one is the fine-tuning dataset used in step 2, and the other is the alignment dataset used in step 5;

[0051] The dataset constructed in step 11 is reasonably and relatively dispersedly divided into two parts, following the principles of randomness and coverage, that is, each part of the data should contain the key common tasks, and be as dispersed as possible on the dataset constructed in step 11 to avoid overfitting of the model to specific problems. At the same time, a small amount of cross data is allowed to exist during the division process. After the division is completed, one part is the fine-tuning dataset used in step 2, and the other part is the alignment dataset used in step 5.

[0052] Step 13: Extract the key points from the dataset constructed in step 11 and construct a problem dataset, which only includes the input <input> .

[0053] Example:

[0054] The key points can be inputs that are not in the previous dataset but are related to similar problems, or problems that are inferred based on the input and output information in the previous dataset, or the problems in step 11.<input,output> Problems in <input> ;

[0055] For inputs that are not in the previous dataset but are related to similar problems: For example, related problems in the field of artificial intelligence, the dataset only contains problems that are strongly related to the field of artificial intelligence. The problem dataset can be related problems in the field of artificial intelligence, such as <input> What is reinforcement learning?

[0056] For reverse deduction: for example<input,output> Yes <What is artificial intelligence? Artificial intelligence (AI) refers to the technology of computer systems performing tasks that usually require human intelligence, including learning, reasoning, perception, language understanding, etc. >; you can combine this information to infer the question <How does artificial intelligence achieve learning and reasoning? >

[0057] Based on the data set constructed in step 11, extract the key points of the task, and construct a question data set based on the answers that the model can and should make after fine-tuning. <input> . Corresponding <output>Provides answers and judges’ scores for the fine-tuned model in step 3.

[0058] The other steps and parameters are the same as those in the first embodiment.

[0059] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that in step 2, LoRA is used to fine-tune the large language model based on the fine-tuning data set to obtain a fine-tuning model; the specific process is:

[0060] As for the method of fine-tuning the large model, the fine-tuning methods for the fine-tuning model in step 2 and the evaluation model in step 3 in this method are both based on the LoRA method. Other fine-tuning methods can also be tried to obtain the fine-tuning model and the evaluation model.

[0061] Step 21: Select an open source large language model (such as Qwen2, GLM4, etc.) as the base model, and the weight of the base model is W0;

[0062] Step 22: Freeze the weight of the base model to W0, W0 = r × i;

[0063] r×i represents the rows and columns of weight W0;

[0064] Step 23: Initialize two low-rank matrices A and B, A = r × t, B = t × i;

[0065] r×t represents the rows and columns of matrix A respectively; t×i represents the rows and columns of matrix B respectively; t is much smaller than r and i;

[0066] Multiply the initialized low-rank matrices A and B to replace the weight W0 of the base model; obtain the base model that introduces A and B;

[0067] Set the initial values ​​of A and B to random numbers close to zero so that the model can make subtle adjustments based on the pre-trained weights;

[0068] Step 24: Input the fine-tuning data set into the base model that introduces A and B. The other parameters in the base model except A and B remain unchanged. Only A and B are trained until convergence, and the trained base model that introduces A and B is obtained.

[0069] The A and B in the trained base model that introduces A and B are the trained A and B;

[0070] Multiply the trained A and B as the new weight increment ΔW;

[0071] Step 25: Fuse the weight W0 of the base model frozen in step 22 and the new parameter weight ΔW obtained in step 24 to obtain a fused weight matrix W;

[0072] The fused weight matrix W = ΔW + W0 = AB + W0

[0073] The fused weight matrix W is used to replace the weights of the base model to obtain a fine-tuning model;

[0074] First, select a relatively ideal open source large language model (such as Qwen2, GLM4, etc.) as the base model, load the base model, and load its pre-trained weight W0. When fine-tuning, freeze the pre-trained weight W0 to ensure that the general ability of the original model is not affected, avoid overfitting of the model due to fine-tuning, and reduce its generalization ability. Then start training the new weight increment ΔW. The method is to initialize two low-rank matrices A and B so that ΔW = AB, and set the initial values ​​of the low-rank matrices A and B to small random numbers close to zero so that the model can be fine-tuned based on the pre-trained weights. During the fine-tuning training process, only the low-rank matrices A and B are optimized. After LoRA training, only the A and B parameters of the low-rank matrices need to be saved. Fusion of the original pre-trained weights and the trained weight increments. Fusion of the pre-trained weights W0 and the new weight increment ΔW obtains a new weight matrix W = ΔW + W0 = AB + W0. For example, during inference, for a given input x, the output after adding LoRA is h = (ΔW + W0)x, ensuring that it can be adjusted for specific tasks while maintaining general capabilities.

[0075] The other steps and parameters are the same as those in the first or second embodiment.

[0076] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that, in step 3, the evaluation data set is obtained based on the problem data set constructed in step 1 and the fine-tuning model obtained in step 2;

[0077] The large language model is fine-tuned using the judgment dataset to obtain the judgment model;

[0078] The specific process is:

[0079] Step 31: Obtain a preliminary evaluation dataset based on the problem dataset constructed in step 13 and the fine-tuning model obtained in step 2;

[0080] Step 32: Score the preliminary judgment data set corresponding to each question obtained in step 31, obtain the rationality score and alignment score, and construct a list type [answer, score1, score2];

[0081] Add the rationality score and alignment score corresponding to each question as the final evaluation result; sort the final evaluation results corresponding to each question from high to low to form a complete evaluation data set; until a complete evaluation data set corresponding to all questions is formed;

[0082] The complete evaluation data set is stored in a list<input,output> ,output=[answer n ,score1 n ,score2 n ];

[0083] Each question has n answers, each answer has 1 reasonableness score and 1 alignment score;

[0084] Add the corresponding rationality scores and alignment scores to obtain n final evaluation results for each question;

[0085] The preliminary evaluation data set obtained in step 31 is submitted to the judges for evaluation, and the answers are scored according to the questions; the evaluation criteria are:

[0086] Reasonableness score <score1>: Whether the answer is logical and whether it is a reasonable solution to the problem.

[0087] Alignment score <score2>: Whether the answers are consistent with human intentions and values, and whether there are harmful, inaccurate or biased results.

[0088] The judges give each answer a corresponding rationality score and alignment score based on the above criteria, construct a list type [answer, score1, score2], and sort them from high to low according to the total score score = score1 + score2 to form a complete evaluation data set.

[0089] Step 33: Based on the complete judgment data set obtained in step 32, LoRA is used to fine-tune the large language model to obtain a judgment model.

[0090] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0091] Specific implementation method 5: This implementation method is different from any one of specific implementation methods 1 to 4 in that in step 31, a preliminary evaluation data set is obtained based on the problem data set constructed in step 13 and the fine-tuning model obtained in step 2; the specific process is:

[0092] Step 311: input a question in the question data set constructed in step 13 into the fine-tuning model obtained in step 2, and the fine-tuning model obtains an answer;

[0093] Step 312: Repeat step 311n times to obtain n different answers corresponding to the same question;

[0094] The same question and n different answers are sorted to obtain a preliminary evaluation data set<input,[asnswer1,…,answern]> ;

[0095] in <input> For the problem, <answer n >n different answers generated by the fine-tuned model for the same question;

[0096] Step 313: Repeat steps 311 to 312 until all the question data sets constructed in step 13 are input into the fine-tuning model obtained in step 2 to obtain a preliminary evaluation data set corresponding to all questions.

[0097] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0098] Specific implementation method 6: This implementation method is different from any one of specific implementation methods 1 to 5 in that in step 33, LoRA is used to fine-tune the large language model based on the complete evaluation data set obtained in step 32 to obtain the evaluation model; the specific process is:

[0099] Step 331: Select an open source large language model (such as Qwen2, GLM4, etc.) as the base model, and the weight of the base model is W0;

[0100] Step 332: Freeze the weight of the base model to W0, W0=r×i;

[0101] r×i represents the rows and columns of weight W0;

[0102] Step 333: Initialize two low-rank matrices A and B, A = r×t, B = t×i;

[0103] r×t represents the rows and columns of matrix A respectively; t×i represents the rows and columns of matrix B respectively;

[0104] Multiply the initialized low-rank matrices A and B to replace the weight W0 of the base model; obtain the base model that introduces A and B;

[0105] Set the initial values ​​of A and B to random numbers close to zero so that the model can make subtle adjustments based on the pre-trained weights;

[0106] Step 334: Input the fine-tuning data set into the base model that introduces A and B. The other parameters in the base model except A and B remain unchanged. Only A and B are trained until convergence, so as to obtain a trained base model that introduces A and B.

[0107] The A and B in the trained base model that introduces A and B are the trained A and B;

[0108] Multiply the trained A and B as the new weight increment ΔW;

[0109] Step 335: Fusing the weight W0 of the base model frozen in step 332 and the new parameter weight ΔW obtained in step 334 to obtain a fused weight matrix W;

[0110] The fused weight matrix W = ΔW + W0 = AB + W0

[0111] The fused weight matrix W is used to replace the weight of the base model to obtain the evaluation model;

[0112] Fine-tune the base model using the judgment dataset constructed in step 32, learn the preferences of human judges in this task, and train the judgment model. After fine-tuning, ensure that the fine-tuned judgment model can complete the task in subsequent steps and give the corresponding rationality score and alignment score after obtaining an input.

[0113] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0114] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that in step 4, the problem in the alignment data set is input into the fine-tuning model of step 2, the fine-tuning model outputs a result, and the similarity between the output result of the fine-tuning model and the target result is calculated;

[0115] The output result of the fine-tuning model is input into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result;

[0116] The specific process is:

[0117] Step 41: Input the problem in the aligned dataset into the fine-tuning model in step 2, fine-tune the model output, and calculate the similarity between the output of the fine-tuning model and the target result (the output corresponding to the input in the dataset in step 1);

[0118] Step 42: Input the text similarity corresponding to each output obtained in step 41 into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result; the evaluation result includes a rationality score and an alignment score.

[0119] The rationality and alignment of the output are evaluated by the judgment model. The multiple outputs obtained in step 41 are again given to the judgment model trained in step 3. The judgment model gives a corresponding rationality score and alignment score for each output, and these scores are sorted and summarized.

[0120] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0121] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that, in step 41, the problem in the alignment data set is input into the fine-tuning model of step 2, the fine-tuning model outputs the result, and the similarity between the fine-tuning model output result and the target result (the output corresponding to the input in the data set of step 1) is calculated; the specific process is:

[0122] Step 411: The first one of the aligned data sets obtained in step 12 <input> Input the fine-tuned model obtained in step 2, and the fine-tuned model outputs O1;

[0123] Repeat step 411m times to obtain the same <input> The corresponding m different outputs;

[0124] The same <input> And the corresponding m different outputs are sorted to get the same <input> Corresponding preliminary dataset <input1,[O1,L,O m ]>; O1 is the first one in the alignment data set <input1>The corresponding output, O m To align the first <input1>The corresponding mth output;

[0125] Step 412: Repeat step 411 until all the aligned data sets are obtained. <input> The corresponding preliminary dataset;

[0126] Step 413: Calculate the same <input> Each of the corresponding m different outputs corresponds to the corresponding target answer in the alignment dataset <output>The text similarity of <input> The text similarity (s1, L, s f ,L,s m ), these scores will be used to measure how well the model output matches the expected answer;

[0127] Step 414: Repeat step 413 until all the aligned data sets are obtained. <input> Each <input> The text similarity corresponding to the m different outputs (s1, L, s f ,L,s m ).

[0128] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0129] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that the text similarity corresponding to each output obtained in step 4 and the weighted average score of the evaluation result are calculated in step 5;

[0130] Determine whether the weighted average score is greater than or equal to a threshold;

[0131] If yes, the fine-tuned model in step 2 is the target model after alignment optimization;

[0132] If not, select the best one based on the weighted average part, build a new fine-tuning dataset, and repeat steps 2, 4, and 5 until the aligned and optimized target model is obtained;

[0133] The specific process is:

[0134] Step 51: Calculate the text similarity and weighted average score of the evaluation results corresponding to each output obtained in step 4 and sort them; the process is:

[0135] The text similarity s of each output obtained in step 4 is f , rationality score r f and the alignment score a f Normalize and get the text similarity s f The weight value S(s f ), rationality score r f The weight value S(r f ), alignment score a f The weight value S(a f );

[0136] same <input> The text similarity (s1, L, s f ,L,s m ), each text similarity corresponds to a rationality score r f and the alignment score a f ,same <input> Corresponding to m different output rationality scores r f and the alignment score a f ;

[0137] The normalization method is to perform normalization through the softmax function, and the calculation formula of the softmax function is as follows:

[0138]

[0139] Among them, xt is the value to be normalized; xi is the similarity s f , rationality score r f or alignment score a f ;

[0140] After obtaining the corresponding weights, a weighted average is performed, that is, the scores of the three dimensions of similarity, rationality and alignment are comprehensively considered to obtain the final score of each answer;

[0141] Calculate the text similarity and weighted average score of the evaluation results corresponding to each output obtained in step 4. The expression is:

[0142] V f =s f ×S(s f )+a f ×S(a f )+r f ×S(r f )

[0143] Among them, V f Represents the weighted average score of the text similarity and evaluation results corresponding to each output obtained in step 4;

[0144] The same <input> The corresponding weighted average score V for the m different outputs f The values ​​are sorted from high to low as V′;

[0145] How many <input> How many V′ does it correspond to?

[0146] Step 52: Select the second highest score in each V′ as the final score V2, and record all final scores in the overall score table;

[0147] Remove the highest score and the lowest score from the overall score table, and calculate the expected value E(V) of the overall score after removing the highest score and the lowest score;

[0148] If the expected value of the overall score E(V) is greater than or equal to the set expected threshold, the fine-tuning model in step 2 is the target model after alignment optimization;

[0149] If the expected value of the overall score E(V) is less than the set expected threshold, the maximum value of each V′ in step 51 is selected and added to the fine-tuning dataset in step 12 to construct a new fine-tuning dataset. Based on the new fine-tuning dataset, steps 2, 4, and 5 are repeated until the aligned and optimized target model is obtained.

[0150] After the sorting in step 51, the final score V2 of the second highest score output of the model is selected as the final score value output for this problem, and the score is recorded in the overall score table. The overall score table records the scores of the multiple alignments in step 4. <input> The final score obtained after input. Remove the highest and lowest scores from the overall score table, and calculate the expected value E(V) of the overall score. If it is greater than the set expected threshold, it means that the model has achieved the expected alignment optimization effect, and it is considered that the target model is obtained, and the entire fine-tuning and alignment optimization process is completed. If it is less than the expected threshold, select the best one, select the one with the higher score, and construct a new dataset based on the fine-tuning dataset in step 12, retrain the fine-tuning model, and then repeat step 4 to obtain the new model output and evaluate it until the target model that meets the expectations is obtained.

[0151] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0152] Specific implementation method ten: This implementation method is a computer storage medium, which stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the large language model alignment optimization method based on small data volume fine-tuning.

[0153] It should be understood that the instructions include computer program products, software or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system, or other electronic device. Computer storage media may include readable media on which instructions are stored, which may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions.

[0154] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.< / output> < / output> < / output>

Claims

1. A large language model alignment optimization method based on small data volume fine-tuning, characterized by: The specific process of the method is: Step 1: Build a dataset for a specific target task and divide the dataset into two parts: one is a fine-tuning dataset and the other is an alignment dataset. At the same time, build problem datasets for specific target tasks; Step 2: Fine-tune the large language model using LoRA based on the fine-tuning dataset to obtain a fine-tuning model; Step 3: Obtain the evaluation dataset based on the problem dataset constructed in step 1 and the fine-tuned model obtained in step 2; The large language model is fine-tuned using the judgment dataset to obtain the judgment model; Step 4: Input the problem in the aligned dataset into the fine-tuned model in step 2, the fine-tuned model outputs the result, and calculates the similarity between the output result of the fine-tuned model and the target result; The output result of the fine-tuning model is input into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result; Step 5: Calculate the text similarity corresponding to each output obtained in step 4 and the weighted average score of the evaluation results; Determine whether the weighted average score is greater than or equal to a threshold; If yes, the fine-tuned model in step 2 is the target model after alignment optimization; If not, select the best one based on the weighted average part, build a new fine-tuning dataset, and repeat steps 2, 4, and 5 until the aligned and optimized target model is obtained.

2. According to claim 1, a large language model alignment optimization method based on small data volume fine-tuning is characterized by: In step 1, a data set for a specific target task is constructed, and the data set for the specific target task is divided into two parts, one part is a fine-tuning data set, and the other part is an alignment data set; At the same time, build problem datasets for specific target tasks; The specific process is: Step 11: Build a dataset for a specific target task; the process is: For a specific target task, set the input of the large language model and the expected output of the large language model, build a dataset based on the input of the large language model and the expected output of the large language model, and set the dataset format to<input,output> ; in <input> Represents the input data of a large language model for a specific target task; <output> Represents the desired output of a large language model for a specific target task;< / output> Step 12: Divide the dataset constructed in step 11 into two parts, one is the fine-tuning dataset and the other is the alignment dataset; Step 13: Extract the key points from the dataset constructed in step 11 and construct a problem dataset, which only includes the input <input> .

3. The large language model alignment optimization method based on small data volume fine-tuning according to claim 2 is characterized by: In step 2, LoRA is used to fine-tune the large language model based on the fine-tuning dataset to obtain a fine-tuning model; the specific process is: Step 21: Select the open source large language model as the base model, and the weight of the base model is W0; Step 22: Freeze the weight of the base model to W0, W0 = r × i; r×i represents the rows and columns of weight W0; Step 23: Initialize two low-rank matrices A and B, A = r × t, B = t × i; r×t represents the rows and columns of matrix A respectively; t×i represents the rows and columns of matrix B respectively; Multiply the initialized low-rank matrices A and B to replace the weight W0 of the base model; obtain the base model that introduces A and B; Step 24: Input the fine-tuning data set into the base model that introduces A and B. The other parameters in the base model except A and B remain unchanged. Only A and B are trained until convergence, and the trained base model that introduces A and B is obtained. The A and B in the trained base model that introduces A and B are the trained A and B; Multiply the trained A and B as the new weight increment ΔW; Step 25: Fuse the weight W0 of the base model frozen in step 22 and the new parameter weight ΔW obtained in step 24 to obtain a fused weight matrix W; The fused weight matrix W = ΔW + W0 = AB + W0 The fused weight matrix W is used to replace the weights of the base model to obtain the fine-tuning model.

4. The large language model alignment optimization method based on small data volume fine-tuning according to claim 3 is characterized by: In the step 3, a judgment dataset is obtained based on the problem dataset constructed in step 1 and the fine-tuning model obtained in step 2; The large language model is fine-tuned using the judgment dataset to obtain the judgment model; The specific process is: Step 31: Obtain a preliminary evaluation dataset based on the problem dataset constructed in step 13 and the fine-tuning model obtained in step 2; Step 32: Score the preliminary judgment data set corresponding to each question obtained in step 31, obtain the rationality score and alignment score, and construct a list type [answer, score1, score2]; Add the rationality score and alignment score corresponding to each question as the final evaluation result; sort the final evaluation results corresponding to each question from high to low to form a complete evaluation data; Until a complete evaluation data set corresponding to all questions is formed; The complete evaluation data set is stored in a list<input,output> ,output=[answer n ,score1 n ,score2 n ]; Each question has n answers, each answer has 1 reasonableness score and 1 alignment score; Step 33: Based on the complete judgment data set obtained in step 32, LoRA is used to fine-tune the large language model to obtain a judgment model.

5. The large language model alignment optimization method based on small data volume fine-tuning according to claim 4, characterized in that: In step 31, a preliminary evaluation data set is obtained based on the problem data set constructed in step 13 and the fine-tuning model obtained in step 2. The specific process is as follows: Step 311: input a question in the question data set constructed in step 13 into the fine-tuning model obtained in step 2, and the fine-tuning model obtains an answer; Step 312: Repeat step 311n times to obtain n different answers corresponding to the same question; The same question and n different answers are sorted to obtain a preliminary evaluation data set <input,[asnswer1,…,answer n ]>; in <input> For the problem, <answer n >n different answers generated by the fine-tuned model for the same question; Step 313: Repeat steps 311 to 312 until all the question data sets constructed in step 13 are input into the fine-tuning model obtained in step 2 to obtain a preliminary evaluation data set corresponding to all questions.

6. The large language model alignment optimization method based on small data volume fine-tuning according to claim 5, characterized in that: In step 33, LoRA is used to fine-tune the large language model based on the complete evaluation data set obtained in step 32 to obtain the evaluation model; the specific process is: Step 331: Select an open source large language model (such as Qwen2, GLM4, etc.) as the base model, and the weight of the base model is W0; Step 332: Freeze the weight of the base model to W0, W0=r×i; r×i represents the rows and columns of weight W0; Step 333: Initialize two low-rank matrices A and B, A = r×t, B = t×i; r×t represents the rows and columns of matrix A respectively; t×i represents the rows and columns of matrix B respectively; Multiply the initialized low-rank matrices A and B to replace the weight W0 of the base model; obtain the base model that introduces A and B; Step 334: Input the fine-tuning data set into the base model that introduces A and B. The other parameters in the base model except A and B remain unchanged. Only A and B are trained until convergence, so as to obtain a trained base model that introduces A and B. The A and B in the trained base model that introduces A and B are the trained A and B; Multiply the trained A and B as the new weight increment ΔW; Step 335: Fusing the weight W0 of the base model frozen in step 332 and the new parameter weight ΔW obtained in step 334 to obtain a fused weight matrix W; The fused weight matrix W = ΔW + W0 = AB + W0 The fused weight matrix W is used to replace the weights of the base model to obtain the judgment model.

7. The large language model alignment optimization method based on small data volume fine-tuning according to claim 6, characterized in that: In step 4, the problem in the alignment data set is input into the fine-tuning model in step 2, the fine-tuning model outputs a result, and the similarity between the output result of the fine-tuning model and the target result is calculated; The output result of the fine-tuning model is input into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result; The specific process is: Step 41: Input the problem in the alignment data set into the fine-tuning model in step 2, the fine-tuning model outputs the result, and calculates the similarity between the output result of the fine-tuning model and the target result; Step 42: Input the text similarity corresponding to each output obtained in step 41 into the evaluation model obtained in step 3, and the evaluation model outputs the evaluation result; the evaluation result includes a rationality score and an alignment score.

8. The large language model alignment optimization method based on small data volume fine-tuning according to claim 7, characterized in that: In step 41, the problem in the alignment data set is input into the fine-tuning model of step 2, the fine-tuning model outputs a result, and the similarity between the output result of the fine-tuning model and the target result is calculated; the specific process is: Step 411: The first one of the aligned data sets obtained in step 12 <input> Input the fine-tuned model obtained in step 2, and the fine-tuned model outputs O1; Repeat step 411m times to obtain the same <input> The corresponding m different outputs; The same <input> And the corresponding m different outputs are sorted to get the same <input> Corresponding preliminary dataset <input1,[O1,L,O m ]>; O1 is the first one in the alignment data set <input1>The corresponding output, O m To align the first <input1> The corresponding mth output; Step 412: Repeat step 411 until all the aligned data sets are obtained. <input> The corresponding preliminary dataset; Step 413: Calculate the same <input> Each of the corresponding m different outputs corresponds to the corresponding target answer in the alignment dataset <output>The text similarity of <input> The text similarity (s1, L, s f ,L,s m );< / output> Step 414: Repeat step 413 until all the aligned data sets are obtained. <input> Each <input> The text similarity corresponding to the m different outputs (s1, L, s f ,L,s m ).

9. The large language model alignment optimization method based on small data volume fine-tuning according to claim 8, characterized in that: In step 5, the text similarity corresponding to each output obtained in step 4 and the weighted average score of the evaluation result are calculated; Determine whether the weighted average score is greater than or equal to a threshold; If yes, the fine-tuned model in step 2 is the target model after alignment optimization; If not, select the best one based on the weighted average part, build a new fine-tuning dataset, and repeat steps 2, 4, and 5 until the aligned and optimized target model is obtained; The specific process is: Step 51: Calculate the text similarity and weighted average score of the evaluation results corresponding to each output obtained in step 4 and sort them; the process is: The text similarity s of each output obtained in step 4 is f , rationality score r f and the alignment score a f Normalize and get the text similarity s f The weight value S(s f ), rationality score r f The weight value S(r f ), alignment score a f The weight value S(a f ); same <input> The text similarity (s1, L, s f ,L,s m ), each text similarity corresponds to a rationality score r f and the alignment score a f ,same <input> Corresponding to m different output rationality scores r f and the alignment score a f ; The normalization method is to perform normalization through a softmax function; Calculate the text similarity and weighted average score of the evaluation results corresponding to each output obtained in step 4. The expression is: V f =s f ×S(s f )+a f ×S(a f )+r f ×S(r f ) Among them, V f Represents the weighted average score of the text similarity and evaluation results corresponding to each output obtained in step 4; The same <input> The corresponding weighted average score V for the m different outputs f The values ​​are sorted from high to low as V′; Step 52: Select the second highest score in each V′ as the final score V2, and record all final scores in the overall score table; Remove the highest score and the lowest score from the overall score table, and calculate the expected value E(V) of the overall score after removing the highest score and the lowest score; If the expected value of the overall score E(V) is greater than or equal to the set expected threshold, the fine-tuning model in step 2 is the target model after alignment optimization; If the expected value of the overall score E(V) is less than the set expected threshold, the maximum value of each V′ in step 51 is selected and added to the fine-tuning dataset in step 12 to construct a new fine-tuning dataset. Based on the new fine-tuning dataset, steps 2, 4, and 5 are repeated until the aligned and optimized target model is obtained.

10. A computer storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement a large language model alignment optimization method based on small data volume fine-tuning as described in any one of claims 1 to 9.

Images