Low-computing-power high-performance large language model field fine tuning method

Abstract

The invention provides a low-computing-power high-performance large language model field fine tuning method, which relates to the technical field of large language models, and comprises the following steps: configuring output data corresponding to input data; output data are obtained through the large language model, whether the output data meet expectations or not is judged, and conditional probabilities of the input data under different conditions are determined; classifying the conditional probability of the input data, distributing marks, distributing a scale value for each token according to the distributed marks, and constructing a token-level loss function; and adjusting the large language model through forward propagation and back propagation calculation according to the constructed token-level loss function. According to the method, the token conditional probabilities of the natural languages under different conditions are compared, the natural languages are directly used as supervision signals, the significance marks in the natural languages are positioned, scale assignment is carried out, and compared with other algorithms, the method has lower time complexity and better effects.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models. Specifically, it particularly relates to a fine-tuning method for large language models with low computing power and high performance in the field. Background Art

[0002] Artificial intelligence has swept across the globe and is profoundly changing the production and lifestyle of human society at an unprecedented speed, breadth, and depth, having a profound impact on global economic development and the progress of human civilization. In recent years, there have been continuous breakthrough innovations in the field of artificial intelligence such as large language models. Large models possess powerful semantic understanding, logical reasoning, scenario learning, and other powerful capabilities, and are regarded as an important path towards general artificial intelligence (AGI). AI large models will become the infrastructure in the new generation of information industrial revolution and the new quality productivity in the digital economy era, empowering the intelligent construction of various industries and fields, with broad prospects.

[0003] When large language models are applied to fine-tuning tasks in specific fields, due to the lack of knowledge data in specific fields, existing large model fine-tuning technologies face unprecedented challenges. Large model fine-tuning usually relies on a large amount of labeled data and requires external guidance and feedback, such as human alignment, scalar rewards, label rewards, and demonstrations. However, when faced with insufficient data on specific field professional issues, these methods often seem inadequate.

[0004] In addition, during the process of large model implementation and application, in order to fully exert the upper limit of the large model's capabilities, the challenge of data shortage in specific fields needs to be faced. Large models have obtained general capabilities through pre-training on massive data, but their initial performance in specific fields is often unsatisfactory. Therefore, a certain amount of field data needs to be provided as a corpus to fine-tune the large model and conduct "special topic training" on the large model. The fine-tuning technology of large language models usually relies on a large amount of labeled data, external guidance, and feedback, such as human alignment, scalar rewards, and demonstrations. However, in practical applications, the scarcity and high cost of field data pose unprecedented challenges to existing fine-tuning technologies.

[0005] Regarding the problems in the related technology, no effective solution has been proposed yet. Summary of the Invention

[0006] In view of this, the present invention provides a fine-tuning method for large language models with low computing power and high performance in the field to solve the above-mentioned problems.

[0007] To solve the above problems, the specific technical solution adopted by the present invention is as follows:

[0008] A fine-tuning method for large language models with low computing power and high performance in the field, comprising:

[0009] S1. Obtain the input data to be input into the large language model and configure the output data corresponding to the input data;

[0010] S2. Input the input data into the large language model, obtain the output data through the large language model, determine whether the output data meets the expectation, and determine the conditional probability of the input data under different conditions;

[0011] S3. Classify and assign labels to the conditional probabilities of the input data, and assign scale values to each token according to the assigned labels to construct a token-level loss function;

[0012] S4. Adjust the large language model by calculating forward propagation and backpropagation according to the constructed token-level loss function.

[0013] Preferably, the input data includes questions, questions plus judgments, and questions plus standard answers;

[0014] The output data corresponding to the input data includes the standard answer to the question and the evaluation of the wrong answer output by the large language model.

[0015] Preferably, the step of inputting the input data into the large language model, obtaining the output data through the large language model, determining whether the output data meets the expectation, and determining the conditional probability of the input data under different conditions includes:

[0016] S21. After the input data is input into the large language model, the large language model generates tokens based on the input data to form an inference result, and uses the inference result as the output data;

[0017] S22. Compare the inference result with the expected result to determine whether the inference result meets the preset expectation. If it does not meet, it means the output data is incorrect; if it meets, it means the output data is correct;

[0018] S23. According to the judgment result of the inference result, determine the type of the input data and determine the conditional probability of the inference result.

[0019] Preferably, the step of determining the type of the input data and determining the conditional probability of the inference result according to the judgment result of the inference result includes:

[0020] When the inference result is that the output data is correct, the types of the input data include basic questions and standard questions, and the conditional probabilities of the inference result include basic conditional probability and standard conditional probability;

[0021] When the inference result is that the output data is incorrect, the types of the input data include basic questions, evaluation questions, and standard questions, and the conditional probabilities of the inference result include basic conditional probability, evaluation conditional probability, and standard conditional probability.

[0022] Preferably, the method of classifying the conditional probability of input data, assigning labels, and assigning scale values to each token according to the assigned labels to construct a token-level loss function includes:

[0023] S31. When the output data is correct, use the preset conditional probability, and combine with the semantic clustering method to classify the conditional probability of the input data, and assign correct identifications to the tokens according to the classification results, and construct a first scale function according to the correct identification assignment results;

[0024] S32. When the output data is incorrect, calculate the conditional ratio, and combine with the semantic clustering method to classify the conditional probability of the input data and assign incorrect identifications, and construct a second scale function according to the incorrect identification assignment results;

[0025] S33. Construct a token-level loss function according to the first scale function and the second scale function.

[0026] Preferably, the step of, when the output data is correct, using the preset threshold conditional probability and combining with the semantic clustering method to classify the conditional probability of the input data and assign correct identifications includes:

[0027] Compare the conditional probability when the output data is correct with the preset threshold conditional probability. When the conditional probability when the output data is correct is greater than the threshold conditional probability, assign the token as a significant label;

[0028] Perform semantic clustering processing on the significant labels, identify the related phrases of the significant labels and assign the tokens as sub-significant labels;

[0029] For the remaining unlabeled tokens, assign them as irrelevant labels.

[0030] Preferably, the step of, when the output data is incorrect, calculating the conditional ratio and combining with the semantic clustering method to classify the conditional probability of the input data and assign incorrect identifications;

[0031] Compare the conditional probability when the output data is correct with the preset threshold conditional probability. When the conditional probability when the output data is correct is greater than the threshold conditional probability, assign the token as a significant label;

[0032] Perform semantic clustering processing on the significant labels, identify the related phrases of the significant labels and assign them as sub-significant labels;

[0033] For the remaining unlabeled tokens, assign them as irrelevant labels.

[0034] Preferably, the expression of the first scale function is:

[0035]

[0036] In the formula, S 1 (y t ) represents the first scaling function, and P(y t |X standard ,y t- ) represents the standard conditional probability, represents the threshold conditional probability, y t represents the output data, Y saliency represents the significance marker, Y sub-saliency represents the sub-significance marker, Y irrelevant represents the irrelevant marker, c 1 、c 2 、c 3 represent hyperparameters.

[0037] Preferably, the expression of the second scaling function is:

[0038]

[0039] In the formula, S 2 (y t ) represents the second scaling function, r 1 、r 2 represent the conditional ratios, P(y t |X standard ,y t- ) represents the standard conditional probability, P(y t |X base ,y t- ) represents the base conditional probability, P(y t |X judge ,y t- ) represents the evaluation conditional probability.

[0040] Preferably, the expression of the token-level loss function is:

[0041]

[0042] In the formula, L represents the token-level loss function, N represents the total number of tokens, S 1 (y t ) represents the first scaling function, S 2 (y t ) represents the first scaling function, y t represents the output data, P(y t |X base ,y t- ) represents the base conditional probability, y t ∈Y incorrect represents that the output data is correct, y t ∈Yincorrect Indicates that the output data is an error.

[0043] The beneficial effects of the present invention are as follows:

[0044] 1. By comparing the conditional probabilities of each natural language token under different conditions, the present invention directly uses natural language as a supervision signal to locate the significant markers therein and assign scale values. Compared with other algorithms, it has a lower time complexity and better effects, and is applicable to all scenarios where the output can be generated through natural language chain reasoning and has labeled data, such as programming, medical diagnosis, natural language reasoning, complex question-and-answer systems, etc. By comparing the generated output with the label, the annotation of significant tokens can be realized, and then the NLFT algorithm can be applied to achieve token-level fine-tuning.

[0045] 2. The present invention uses natural language for fine-tuning. By utilizing the powerful language understanding ability of the target language model, the guiding information of natural language is attached to the token-level output. This technology has the significant advantages of fine-grained and lightweight fine-tuning, can quickly improve the fine-tuning effect relying on a small amount of data, reduce the entry threshold of data and computing power, and help the application of large models move towards popularization. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0047] Figure 1 is a flowchart of a fine-tuning method in the field of large language models with low computing power and high performance according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of the performance of the complete data set in a fine-tuning method in the field of large language models with low computing power and high performance according to an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of the performance of different data volumes at 200 steps in a fine-tuning method in the field of large language models with low computing power and high performance according to an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of the model performance at different epochs in a fine-tuning method in the field of large language models with low computing power and high performance according to an embodiment of the present invention;

[0051] Figure 5Schematic diagram of model performance at different epochs in a fine-tuning method for low-computing-power and high-performance large language models according to an embodiment of the present invention;

[0052] Figure 6 Schematic diagram of time consumption performance in a fine-tuning method for low-computing-power and high-performance large language models according to an embodiment of the present invention;

[0053] Figure 7 Schematic diagram of GPU consumption performance in a fine-tuning method for low-computing-power and high-performance large language models according to an embodiment of the present invention. Detailed implementation manners

[0054] In order to enable those skilled in the art of the present technology to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0055] According to an embodiment of the present invention, a fine-tuning method for low-computing-power and high-performance large language models is provided.

[0056] Now, the present invention will be further described in conjunction with the accompanying drawings and specific implementation manners. As Figure 1 shown, the fine-tuning method for low-computing-power and high-performance large language models according to an embodiment of the present invention includes:

[0057] S1. Obtain input data for input to the large language model and configure output data corresponding to the input data;

[0058] S2. Input the input data into the large language model, obtain the output data through the large language model, judge whether the output data meets the expectation, and determine the conditional probability of the input data under different conditions;

[0059] S3. Classify and assign labels to the conditional probabilities of the input data, and assign scale values to each token according to the assigned labels to construct a token-level loss function;

[0060] S4. Adjust the large language model by calculating forward propagation and backward propagation according to the constructed token-level loss function.

[0061] As a preferred implementation manner, the input data includes questions, questions plus judgments, and questions plus standard answers;

[0062] The output data corresponding to the input data includes the standard answers to the questions and the evaluations of the wrong answers output by the large language model.

[0063] As a preferred embodiment, the steps of inputting input data into a large language model, obtaining output data through the large language model, determining whether the output data meets expectations, and determining the conditional probability of the input data under different conditions include:

[0064] S21. After the input data is input into the large language model, the large language model generates tokens based on the input data to form an inference result, and uses the inference result as the output data;

[0065] S22. Compare the inference result with the expected result to determine whether the inference result meets the preset expectation. If it does not meet, it means the output data is incorrect; if it meets, it means the output data is correct;

[0066] S23. According to the judgment result of the inference result, determine the type of the input data and determine the conditional probability of the inference result.

[0067] As a preferred embodiment, the steps of determining the type of the input data and determining the conditional probability of the inference result according to the judgment result of the inference result include:

[0068] When the inference result is that the output data is correct, the types of the input data include basic questions and standard questions, and the conditional probability of the inference result includes basic conditional probability and standard conditional probability;

[0069] When the inference result is that the output data is incorrect, the types of the input data include basic questions, evaluation questions and standard questions, and the conditional probability of the inference result includes basic conditional probability, evaluation conditional probability and standard conditional probability.

[0070] As a preferred embodiment, the steps of classifying and assigning labels to the conditional probability of the input data, and constructing a token-level loss function according to the assigned labels by assigning scale values to each token include:

[0071] S31. When the output data is correct, use the preset conditional probability, and combine with the semantic clustering method to classify the conditional probability of the input data, assign correct identifications to the tokens according to the classification results, and construct a first scale function according to the correct identification assignment results;

[0072] S32. When the output data is incorrect, calculate the conditional ratio, and combine with the semantic clustering method to classify the conditional probability of the input data and assign incorrect identifications, and construct a second scale function according to the incorrect identification assignment results;

[0073] S33. Construct a token-level loss function according to the first scale function and the second scale function.

[0074] As a preferred embodiment, when the output data is correct, using the pre-set threshold conditional probability and combining with the semantic clustering method to classify the conditional probability of the input data and assign correct labels includes:

[0075] Compare the conditional probability when the output data is correct with the pre-set threshold conditional probability. When the conditional probability when the output data is correct is greater than the threshold conditional probability, then assign the token as a significant marker;

[0076] Perform semantic clustering processing on the significant markers, identify the related phrases of the significant markers and assign the token as a sub-significant marker;

[0077] For the remaining unmarked tokens, assign them as irrelevant markers.

[0078] As a preferred embodiment, when the output data is incorrect, by calculating the conditional ratio and combining with the semantic clustering method to classify the conditional probability of the input data and assign incorrect labels;

[0079] Compare the conditional probability when the output data is correct with the pre-set threshold conditional probability. When the conditional probability when the output data is correct is greater than the threshold conditional probability, then assign the token as a significant marker;

[0080] Perform semantic clustering processing on the significant markers, identify the related phrases of the significant markers and assign it as a sub-significant marker;

[0081] For the remaining unmarked tokens, assign them as irrelevant markers.

[0082] As a preferred embodiment, the expression of the first scale function is:

[0083]

[0084] In the formula, S 1 (y t ) represents the first scale function, P(y t |X standard ,y t- ) represents the standard conditional probability, represents the threshold conditional probability, y t represents the output data, Y saliency represents the significant marker, Y sub-saliency represents the sub-significant marker, Y irrelevant represents the irrelevant marker, c 1 、c 2 、c 3 represent hyperparameters.

[0085] As a preferred embodiment, the expression of the second scale function is:

[0086]

[0087] Wherein, S 2 (y t ) represents the second scaling function, r 1 , r 2 represents the conditional ratio, P(y t |X standard , y t- ) represents the standard conditional probability, P(y t |X base , y t- ) represents the base conditional probability, P(y t |X judge , y t- ) represents the evaluation conditional probability.

[0088] As a preferred embodiment, the expression of the token-level loss function is:

[0089]

[0090] Wherein, L represents the token-level loss function, N represents the total number of tokens (i.e., the total number of tokens), S 1 (y t ) represents the first scaling function, S 2 (y t ) represents the first scaling function, y t represents the output data, P(y t |X base , y t- ) represents the base conditional probability, y t ∈ Y incorrect indicates that the output data is correct, y t ∈ Y incorrect indicates that the output data is incorrect.

[0091] To facilitate understanding of the above technical solution of the present invention, the above technical solution of the present invention will be further described from the perspectives of architecture and principle as follows:

[0092] In the present invention, through the token-based fine-tuning algorithm NLFT, it focuses on natural language chain-of-thought (CoT) reasoning and its results. Assume that the input of the large language model (LLM) is X, and the reasoning result is Y = {y 1 , y 2 ,..., y n, by executing different input prompts X, the conditional probability of each token in the same inference output under different input conditions can be obtained. Then, by comparing these conditional probabilities, significance markers are assigned, and marker-level loss calculation is performed to achieve fine-grained adjustment of the LLM. In summary, the NLFT algorithm is significantly superior to the SFT algorithm in terms of time and space complexity.

[0093] Specifically, the following hypothesis is proposed through extensive experimental observations: Represent different prompt inputs as X, and the natural language chain reasoning result as Y = {y 1 , y 2 ,..., y n}, and the conditional probability of each term under different inputs is represented as P(y t |X, y t- ). Assume:

[0094] 1. When the output Y is consistent with the expectation (i.e., the output is correct), the conditional probability P(y t |X, y t |X, y t- ) of the key scoring marker (significance marker) y standard = <question, standard answer> compared to X base <question>Significantly higher.

[0095] 2. When the output Y meets the expectation (i.e., the output is incorrect), the key scoring marker (significance marker) y t The conditional probability P(y t |X, y t- ) shows great variations under three hint inputs: X base = <question>, X judge = <question, judgment> and X standard = <question, standard answer>.

[0096] Based on the above assumptions, the NLFT algorithm is proposed. First, according to whether the output meets the expectation, the conditional probability of each term under different prompt inputs is obtained; then, significant markers are located through contrastive learning, and phrases containing these markers are located using semantic similarity clustering. Finally, a token-level loss function is constructed to achieve fine-grained fine-tuning.

[0097] Among them, formally, the process of generating results through natural language CoT can be decomposed into a series of next-token prediction actions. Specifically, the CoT inference result is taken as Y = {y 1 , y 2 ,..., y n}, where each <y t > is a token inferred based on the given input X and the previous output y t- = {y 1 , y 2 ,..., y t-1}. In addition, each y t has its own conditional probability function P(y t |X, y t- ). As mentioned above, the conditional probability of significant markers changes significantly with different inputs X. Based on this observation, the inference result Y is divided into correct and incorrect results, and then conditional probability lookup and collection are selectively performed for each category.

[0098] Correct output. When the CoT inference output Y is correct, the input X is divided into X base = <question>and X standard = <question, standardanswer> two categories. Then, for each token y of the output Y t , there will be two conditional probabilities P(y t |X base , y t- ) and P(y t |X standard , y t- ). In the next section, significance labels will be assigned based on these conditional probabilities.

[0099] Output error. When the inference output Y is incorrect, the input X is divided into X base = <question>, X judge = <question, judgement>, X standard = <question, standardanswer> three categories.

[0100] Then, for each y of the output Y t , there will be three conditional probabilities P(y t |X base , y t- ), P(y t |X judge , y t- ), and P(y t |X standard , y t- ). In the next section, significance labels will be assigned based on these conditional probabilities.

[0101] In addition, probability-driven significance token assignment includes:

[0102] After obtaining the conditional probabilities, continue to classify them and assign significance labels. Similarly, the assignment strategies are also divided into two categories: correct results and incorrect results.

[0103] Correct output. When the CoT inference output Y is correct, set the threshold conditional probability When the conditional probability , it is considered that this token is more likely to be adopted under the condition of X standard than under the threshold conditional probability. This means that there is a greater correlation between the label and the condition of X standard , so it is assigned as a significance label. Semantic clustering is performed around these significance labels to identify their related phrases and designate them as sub-significance labels. Finally, the remaining tokens are classified as irrelevant tokens.

[0104] Incorrect output. For an incorrect CoT inference output Y, the conditional probability threshold cannot be simply set because, in this case, the conditional probabilities of all tokens are usually low. Therefore, calculate the following two conditional ratios:

[0105]

[0106] where r 1 , r 2 represent the conditional ratios, P(y t |X standard , y t- ) represents the standard conditional probability, P(y t |X base , y t- ) represents the base conditional probability, P(y t |X judge ,y t- ) represents the evaluation conditional probability.

[0107] If a token is a significant token, then its conditional probability under the condition of X judge should be much higher than its conditional probability under the conditions of X base and X standard . Therefore, the corresponding ratios r 1 and r 2 will also be higher. Thus, if the corresponding ratios r 1 and r 2 of the token both exceed the preset value r 0 , and its own conditional probability exceeds the threshold , then the token will be assigned as a significant token, denoted as the significant marker Y saliecvy . Semantic clustering is performed around these significant markers to identify relevant phrases. Specifically, according to the vector representation of the token, the cosine similarity with the semantic vectors of its surrounding tokens is calculated, and the tokens with high similarity are included in the sub-significant marker Y sub-saliency . The remaining tokens are denoted as irrelevant markers Y irrelevant .

[0108] Given that these markers are close to significant markers in incorrect cases, they are directly classified as irrelevant markers and assigned a scale of 0 in the subsequent contrast learning process.

[0109] Again, scale values are assigned according to the previously assigned significant markers, and then contrast learning is performed. Similarly, the scale strategies are divided into two categories: correct results and incorrect results.

[0110] Correct output: As shown before, in the correct CoT inference output, tokens are divided into three types: significant tokens, sub-significant tokens, and irrelevant tokens. For each symbol, there are different scales:

[0111]

[0112] where c 1 , c 2 , c 3 are hyperparameters, and c 2 < c 3 is a hyperparameter. It can be observed that in this configuration, all three scales increase as the conditional probability P(y t |X standard ,y t- ) increases. The scale of the significant marker is the largest and always greater than 1, and the scale of the sub-significant marker is always greater than that of the irrelevant marker.

[0113] Output error: In the incorrect CoT inference output, the tokens are classified into two categories: significant tokens and irrelevant tokens. For irrelevant tokens, the scale is set to 0 because in the incorrect output, only significant tokens are expected to be considered. For significant tokens, there is a scale:

[0114]

[0115] The scale is greater than 0 and less than 1, and these scales also increase as the conditional probability P(y t |X judge ,y t- ) increases.

[0116] After obtaining the scale for each token, the final loss function is obtained:

[0117]

[0118] In addition, based on the final loss function, after the large language model calculates the loss during forward propagation, the Autograd (an automatic gradient calculation system) mechanism of PyTorch (an open-source deep learning library) will automatically perform backpropagation loss.backward() according to the computational graph to calculate the gradients of each parameter. Then, the optimizer uses these gradients to update the parameters optimizer.step(). After each training step, it is necessary to call optimizer.zero_grad() to clear the gradients of the previous round to prevent cumulative errors. This process ensures that the parameters are optimized in the direction of minimizing the loss.

[0119] Experimental results:

[0120] A preliminary verification of NLFT was carried out on the mathematical problem dataset GSM8K. The GSM8K dataset provides problems in natural language form, standard problem-solving processes, and numerical form of the correct answers. The GSM8K training set has a total of 7473 data, and the test set has 1319 data.

[0121] When the accuracy of the base model to be fine-tuned is low, other models with better performance are selected to generate answers. This process is called "teaching". And when the model is capable of generating a certain proportion of correct responses, the trained model is continued to learn from its own answers and produce results. This process is called self-study. The model was compared with SFT (Supervised Fine-Tuning) and ReFT (Reinforcement Fine-Tuning).

[0122] First, the performance of the model on the complete dataset was compared. As Figure 2 As shown (using 25%, 50%, 75%, 100% of the GSM8K training set, corresponding to 1,868, 3,737, 5,605, 7,473 samples respectively, comparing the accuracy of SFT and NLFT, with 0 representing the basic model before fine-tuning, Proportion of GSM8K training dataset used representing the proportion of the GSM8K training dataset used, Avg Math Reasoning Acc representing the average math reasoning accuracy, and Math Accon GSM8K representing the math accuracy on GSM8K), it can be observed that NLFT always achieves a higher accuracy than SFT. NLFT achieves an accuracy of over 70% on all four percentage datasets, while SFT is between 44% and 46%. Compared to SFT, the accuracy of NLFT has increased by over 25%. In addition, the change in the proportion of the training dataset has little impact on the accuracy of NLFT and SFT, indicating that the boundary effect of improving accuracy by expanding the data scale will gradually decrease. Therefore, the focus is shifted to smaller-scale training datasets to fill the gap in the accuracy details from 0 to 25%.

[0123] To study the performance of NLFT when using the smallest dataset samples as the training set, data sizes from 50 to 400 were adopted and divided every 50 data points. In Figure 3 (for using the smallest dataset samples of GSM8K as the training set, including the accuracy comparison of NLFT trained with 200 steps, 1 time, and 2 times respectively. To better illustrate the increase in accuracy from data size 50 to 100, a graph with a data size of 75 is additionally provided. Under the settings of 1-epoch and 2-epoch training, Training Dataset Size represents the training dataset size, and with 200steps represents having 200 steps), experiments with a fixed training step of 200 were conducted at different data sizes. It was observed that NLFT had already achieved an accuracy of 62.93% at 50 data points, which is close to the accuracy at 400 data points. Figure 4 Accuracy comparison between NLFT and SFT for the 50 smallest dataset samples in the 1 - 16 epochs. Training Epoch(s) represents the number of training rounds, and Data Size represents the data size. The accuracy of NLFT and SFT was compared within 16 consecutive epochs (1 epoch means training once using all samples in the training set. Generally speaking, the value of epoch is the number of times the entire data is cycled). It was observed that the accuracy was 11.30% at the 1st epoch. By the 2nd epoch, it sharply rose to 30.8%, which was the accuracy that SFT could reach only at the fifth epoch. Subsequently, the model accuracy continued to increase, reaching 60.1% at the 4th epoch and remaining above 60% thereafter. Meanwhile, SFT rose rapidly from the 1st epoch to the 5th epoch, reaching the highest accuracy of 34.4% at the 8th epoch and then gradually decreasing. These results indicate that NLFT demonstrates breakthrough learning potential on limited datasets, which is not possessed by fine-tuning algorithms such as SFT.

[0124] To verify the performance of different fine-tuning algorithms under the same data scale and the same training degree, experiments were also carried out on the first 800 random data. The accuracies of NLFT, SFT, and ReFT trained for 30 epochs were obtained through training and testing, as Figure 5 shown (Accuracy comparison of NLFT, SFT, and ReFT algorithms when the data scale is 800. Both experiments of ReFT were pre-warmed with SFT. The initial accuracy was epoch 0. with pre-warmed LM 1st try represents the first try of the pre-warmed LM, and with pre-warmed LM 2nd try represents the second try of the pre-warmed LM). In the experiment, due to the instability of fine-tuning algorithms based on reinforcement learning such as ReFT and their high requirements for the amount of data, ReFT degraded to only repeating the instruction. While SFT could learn normally under the configuration of 800 data, and reached an accuracy of 39.88% at epoch 5, and then the accuracy decreased by 30%. In contrast, NLFT not only could be trained stably, but also reached an accuracy of 71.65% and remained stable after the highest accuracy. This shows the universality of NLFT for data scale and the high efficiency of training effect

[0125] In terms of time consumption and video memory consumption, as Figure 6 (Compare the average time overhead of each epoch for the NLFT, SFT, and ReFT algorithms. The data volume for each experiment is fixed at 800. Compared with NLFT and SFT, the time overhead of ReFT is significantly higher. The average time consumption of NLFT is about 3 times that of SFT. Time Consumption represents the time consumption in seconds) and Figure 7 as shown (Comparison of GPU Memory Utilization of SFT, ReFT, and NLFT. In ReFT, the batch size is set to 2. The runtime batch sizes of SFT and NLFT are the same, both 2. The GPU memory usage of NLFT is as lightweight as that of SFT and more than 10 times lower than that of ReFT. Comparison of GPU Memory Utilization represents the comparison of GPU memory utilization, and GPU Memory Consumption represents the GPU memory consumption in GB). When training for 1 epoch in a 2-GPU configuration, SFT on average takes 7.77 min and occupies a total video memory of 44.55 GB; NLFT on average takes 26.1 min and occupies a total video memory of 46.87 GB; while ReFT cannot run in a 2-GPU configuration. In the analysis of algorithm complexity, it is known that NLFT has 3 times more forward inference processes than SFT. Therefore, the constant term of its time complexity is at least 3, which can be verified from the ratio of the time consumption of the two. Even with the increased constant term, NLFT still belongs to a lightweight fine-tuning algorithm with linear time complexity.

[0126] In summary, with the above technical solutions of the present invention, by comparing the conditional probabilities of each natural language token under different conditions, the present invention directly uses natural language as a supervision signal to locate the significant markers therein and assign scales. Compared with other algorithms, it has a lower time complexity and better effects, and is applicable to all scenarios where the output can be generated through natural language chain reasoning and has labeled data, such as programming, medical diagnosis, natural language inference, complex question-answering systems, etc. By comparing the generated output with the labels, the annotation of significant tokens can be achieved, and thus the NLFT algorithm can be applied to achieve token-level fine-tuning. The present invention uses natural language for fine-tuning. By utilizing the powerful language understanding ability of the target language model, the guiding information of natural language is attached to the token-level output. This technology has the significant advantages of fine-grained and lightweight fine-tuning, can quickly improve the fine-tuning effect relying on a small amount of data, lower the entry threshold of data and computing power, and help the application of large models move towards popularization.

[0127] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system or a computer program product. Therefore, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-usable program code.

[0128] The specific embodiments described above further elaborate on the purpose, technical solution and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.< / question> < / question> < / question> < / question>

Claims

1. A low-computing-power, high-performance large language model domain fine-tuning method, characterized in that: include: S1. Obtain input data for inputting into a large language model, and configure output data corresponding to the input data; S2. Input the input data into the large language model, obtain the output data through the large language model, and judge whether the output data meets the expectations, and determine the conditional probability of the input data under different conditions; S3. Classify and assign tags to the conditional probabilities of the input data, and assign scale values ​​to each token based on the assigned tags to construct a token-level loss function; S4. According to the constructed token-level loss function, the large language model is adjusted through forward propagation and back-propagation calculations.

2. According to claim 1, a low-computing-power, high-performance large language model domain fine-tuning method is characterized in that: The input data includes questions, questions plus judgments, and questions plus standard answers; The output data corresponding to the input data includes a standard answer to the question and an evaluation of an incorrect answer output by the large language model.

3. According to claim 1, a low-computing-power, high-performance large language model domain fine-tuning method is characterized in that: The step of inputting the input data into the large language model, obtaining the output data through the large language model, judging whether the output data meets the expectation, and determining the conditional probability of the input data under different conditions includes: S21, after the input data is input into the large language model, the large language model generates tokens according to the input data, forms an inference result, and uses the inference result as output data; S22, comparing the inference result with the expected result to determine whether the inference result meets the preset expectation. If not, it indicates that the output data is wrong. If it is satisfied, it indicates that the output data is correct. S23. Determine the type of input data based on the judgment result of the inference result, and determine the conditional probability of the inference result.

4. The low-computing-power, high-performance large language model domain fine-tuning method according to claim 3, characterized in that: Determining the type of input data based on the judgment result of the inference result and determining the conditional probability of the inference result includes: When the inference result is that the output data is correct, the types of input data include basic questions and standard questions, and the conditional probability of the inference result includes basic conditional probability and standard conditional probability; When the inference result is an output data error, the types of input data include basic questions, evaluation questions and standard questions, and the conditional probability of the inference result includes basic conditional probability, evaluation conditional probability and standard conditional probability.

5. The low-computing-power, high-performance large language model domain fine-tuning method according to claim 3, characterized in that: The conditional probability of the input data is classified and assigned tags, and a scale value is assigned to each token according to the assigned tags, and the token-level loss function is constructed, including: S31. When the output data is correct, the conditional probability of the input data is classified using the pre-set conditional probability and the semantic clustering method, and the tokens are correctly identified and assigned according to the classification results, and the first scaling function is constructed according to the correct identification assignment results; S32, when the output data is wrong, by calculating the conditional ratio, and combining the semantic clustering method to classify the conditional probability of the input data and assign error labels, a second scaling function is constructed according to the error label assignment result; S33, constructing a token-level loss function according to the first scaling function and the second scaling function.

6. A low-computing-power, high-performance large language model domain fine-tuning method according to claim 5, characterized in that: When the output data is correct, the conditional probability of the input data is classified and processed using the preset threshold conditional probability and combined with the semantic clustering method, and the correct identification assignment includes: The conditional probability when the output data is correct is compared with the preset threshold conditional probability. When the conditional probability when the output data is correct is greater than the threshold conditional probability, the token is assigned as a significant mark; Perform semantic clustering on the salient tokens, identify the related phrases of the salient tokens and assign the tokens as less salient tokens; For the remaining unmarked tokens, they are assigned irrelevant tags.

7. The low-computing-power, high-performance large language model domain fine-tuning method according to claim 5, characterized in that: When the output data is erroneous, the conditional probability of the input data is classified and assigned an error mark by calculating the conditional ratio and combining the semantic clustering method; The conditional probability when the output data is correct is compared with the preset threshold conditional probability. When the conditional probability when the output data is correct is greater than the threshold conditional probability, the token is assigned as a significant mark; Perform semantic clustering on the significant tags, identify the related phrases of the significant tags and assign them as sub-salient tags; For the remaining unmarked tokens, they are assigned irrelevant tags.

8. The low-computing-power, high-performance large language model domain fine-tuning method according to claim 5, characterized in that: The expression of the first scaling function is: In the formula, S1(y t ) represents the first scaling function, P(y t |X standard ,y t- ) represents the standard conditional probability, represents the threshold conditional probability, y t Represents output data, Y saliency Indicates a significant mark, Y sub-saliency Indicates a secondary significant marker, Y irrelevant represents an irrelevant label, and c1, c2, and c3 represent hyperparameters.

9. The low-computing-power, high-performance large language model domain fine-tuning method according to claim 5, characterized in that: The expression of the second scaling function is: In the formula, S2(y t ) represents the second scaling function, r1 and r2 represent conditional ratios, and P(y t |X standard ,y t- ) represents the standard conditional probability, P(y t |X base ,y t- ) represents the basic conditional probability, P(y t |X judge ,y t- ) represents the evaluation conditional probability.

10. The low-computing-power, high-performance large language model domain fine-tuning method according to claim 1, characterized in that: The expression of the token-level loss function is: Where L represents the token-level loss function, N represents the total number of tokens, and S1(y t ) represents the first scale function, S2(y t ) represents the first scale function, y t Represents the output data, P(y t |X base ,y t- ) represents the basic conditional probability, y t ∈Y incorrect Indicates that the output data is correct, y t ∈Y incorrect Indicates that the output data is an error.

Images