A large language model proxy tuning method with adaptive conditions

Abstract

The present invention relates to a large language model proxy tuning method with adaptive conditions, and belongs to the field of natural language processing technology in the field of artificial intelligence. The present invention first calculates the predicted logits of a basic model, an expert model, and an anti-expert model; through a conditional judgment mechanism, respectively calculates the index of the consistent prediction between the basic model and the expert model, the index of the consistent prediction between the expert model and the anti-expert model and the inconsistent prediction between the basic model and the expert model, and the index of other cases; according to the conditional judgment result, the logits in different cases are selectively updated; when the prediction of the basic model and the expert model is consistent, the logits of the expert model are selected for updating; after completing the selective update of the logits, the adjusted logits are converted into a probability distribution through a softmax function, and the final predicted text is generated according to the probability distribution; the present invention enhances the prediction accuracy and generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to a large language model agent tuning method with adaptive conditions, which belongs to the field of natural language processing technology in the field of artificial intelligence and can be widely used in the fields of intelligent customer service, content generation, education and training, medical treatment, legal compliance, financial services, human resources management, etc. Background Art

[0002] In recent years, large-scale pretrained language models (LLMs) have made significant progress in the field of natural language processing (NLP). These models have demonstrated strong general capabilities by pre-training on massive text data and are able to achieve excellent performance in multiple downstream tasks. However, despite the broad versatility of LLMs, they still need further optimization to better achieve the desired behavior of specific tasks. For example, the model may need to be adjusted for specific domains, specific tasks, or specific instructions.

[0003] At present, the methods for optimizing large-scale language models mainly include the following: (1) Full Fine-tuning, which further adjusts all parameters of the pre-trained model to better adapt to specific tasks or fields. Its advantage is that it can achieve the best performance on a specific task because all model parameters have been adjusted. However, its computational cost is extremely high and requires a lot of computing resources and time. In addition, for some private models, their internal parameters cannot be accessed, so fine-tuning is impossible. (2) Parameter-Efficient Fine-tuning: A method that only adjusts some parameters of the model to save computing resources. Common methods include Low-Rank Adaptation (LoRA) and Prompt Tuning. These methods introduce a small number of trainable parameters (such as low-rank matrices or prompt word vectors) to perform fine-tuning for specific tasks while keeping most model parameters unchanged. However, in some complex tasks, the performance of parameter-efficient fine-tuning may not be as good as full parameter fine-tuning. (3) Decoding-Time Adjustment: A method that adjusts the predicted distribution of a model during the process of generating output without changing the internal parameters of the model. Proxy tuning belongs to this category of methods. It introduces additional guidance models (such as expert models and anti-expert models) to modify the output of the large model during the reasoning process to achieve a specific generation effect. The expert model and anti-expert model with small parameters enhance the base model with large parameters to provide more accurate output during the reasoning process. The base model refers to a language model that has been pre-trained on a large scale, has wide versatility, and can perform well in a variety of tasks. The expert model is a fine-tuned version of the base model on a specific task or field, which can provide more professional and accurate predictions. The anti-expert model is an unfine-tuned version of the expert model, which is used to provide comparative information.

[0004] Although the existing proxy tuning method effectively adjusts the output of large models during the inference process, its unified parameter adjustment and lack of conditional judgment strategies limit the performance optimization of the model under different input conditions. In addition, the method fails to fully handle the special relationship between the base model and the expert model and anti-expert model, which may introduce bias.

[0005] To this end, the present invention proposes an adaptive conditional large language model proxy tuning method (Adaptive Conditional Proxy-Tuning, ACPT). Summary of the invention

[0006] The problem solved by the present invention is: the present invention provides a large language model proxy tuning method with adaptive conditions, which is used to solve the problem that the existing methods cannot be adjusted dynamically according to different inputs, resulting in insufficient or excessive adjustment in some cases; the existing methods adopt a unified adjustment strategy for all tokens, fail to distinguish the specific needs in different situations, and fail to fully reflect the complex prediction relationship; the existing methods fail to effectively handle the special relationship between the basic model and the expert model and the anti-expert model, which may introduce deviations and affect the prediction accuracy; the present invention introduces conditional judgment and selective update mechanisms to more finely adjust logits, thereby dynamically adapting to different situations and enhancing the prediction accuracy and generalization ability of the model. This improvement not only retains the advantages of the original method, but also significantly improves the adaptability and performance of the model in multi-task scenarios.

[0007] The technical solution of the present invention is: a large language model proxy tuning method with adaptive conditions, the method comprising:

[0008] Step 1, Initialization: First, calculate the predicted logits of the base model, expert model, and anti-expert model; logits are the unnormalized scores generated by the model at the output layer, which are used to represent the relative likelihood of each possible output; logits also usually refer to the original scores of the neural network output layer before applying an activation function (such as softmax). They represent the unnormalized predicted values ​​of the model for each category.

[0009] Step 2, conditional judgment: through the conditional judgment mechanism, calculate the index where the predictions of the basic model and the expert model are consistent, the index where the predictions of the expert model and the anti-expert model are consistent and the predictions of the basic model and the expert model are inconsistent, and the indexes in other cases respectively;

[0010] Step 3, Selective update: According to the conditional judgment results, the logits in different situations are selectively updated; when the predictions of the basic model and the expert model are consistent, the logits of the expert model are selected for update; when the predictions of the expert model and the anti-expert model are consistent and the predictions of the basic model and the expert model are inconsistent, the logits of the basic model are selected for update; in other cases, the linear combination of the basic model logits plus the expert model logits minus the anti-expert model logits is used for update;

[0011] Step 4. Generate predicted text: After completing the selective update of logits, the adjusted logits are converted into probability distribution through the softmax function, and the final predicted text is generated based on the probability distribution.

[0012] Furthermore, the Step 1 includes:

[0013] During the initialization process, the predicted logits of the base model, expert model, and anti-expert model are calculated. Logits are the unnormalized scores generated by the model at the output layer, which are used to represent the relative likelihood of each possible output. Let the predicted logits of the base model be logits base , the predicted logits of the expert model are logits expert , the predicted logits of the anti-expert model are logits antiexpert ;

[0014] The mathematical expression is as follows:

[0015] logits base =f base (x 1:i )

[0016] logits expert =f expert (x 1:i )

[0017] logits antiexpert =f antiexpert (x 1:i )

[0018] Among them, f base 、f expert and f antiexpert Respectively represent the prediction functions of the basic model, expert model and anti-expert model, x 1:i Represents an input text sequence.

[0019] Furthermore, in Step 2, calculating the index of the consistency between the prediction of the basic model and the expert model includes:

[0020] When the prediction results of the basic model and the expert model for the same input are consistent, it means that the prediction of the basic model on the input has a certain degree of credibility. The prediction results of the expert model are directly adopted. The prediction results of the expert model are directly adopted. The logist matrix (logits) used to generate the predicted text is used in the end. next In the logits table, the rows that satisfy the condition that the prediction results of the base model and the expert model for the same input are consistent are represented by expert The corresponding row vector replacement is calculated as follows:

[0021] match base_expert =(argmax(logits base )==argmax(logits expert ))

[0022] Among them, argmax is a commonly used operator, which is used to find the index of the maximum value in a vector or array. Suppose a vector v = [v1, v2, ..., v n ], the mathematical definition of the argmax operator is as follows:

[0023]

[0024] In other words, argmax returns the value that makes v i The index i, v where the maximum value is reached i Indicates the maximum value of the elements in vector v, match base_expert It is a matrix with b rows and 1 column, which indicates the row index value that satisfies the condition that "the prediction results of the basic model and the expert model for the same input are consistent". "==" indicates that the "equal" relationship operation is performed on the elements at the corresponding positions of the matrix. If the two operands are equal in size, the operation result is 1, otherwise it is 0.

[0025] Furthermore, in Step 2, the indexes for which the predictions of the expert model and the counter-expert model are consistent and the predictions of the basic model and the expert model are inconsistent are calculated as follows:

[0026] When the prediction results of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent, it means that there may be problems with the prediction of the expert model on the input. At this time, the prediction results of the basic model are adjusted. The adjustment refers to the logit matrix used to generate the predicted text, i.e., logits. next The row vector that satisfies the condition "when the prediction results of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent" is replaced by logits base The corresponding row in the table; the calculation formula is as follows:

[0027] match expert_antiexpert =(argmax(logits expert )==argmax(logits antiexpert ))

[0028] not_match base_expert =(argmax(logits base )≠argmax(logits expert ))

[0029] match expert_antiexpert_base_not =match expert_antiexpert ∧not_match base_expert

[0030] Among them, match expert_antiexpertIndicates the row index value of "the expert model and the anti-expert model prediction are consistent", not_match base_expert Indicates the row index value of "the predictions of the basic model and the expert model are inconsistent", match expert_antiexpert_base_not Indicates the row index where the expert model and the anti-expert model predict the same thing but the basic model and the expert model predict the same thing. expert_antiexpert Indicates the row index that the expert model and the anti-expert model predict the same, not_match base_expert Indicates the row index where the predictions of the base model and the expert model are inconsistent; the index calculation for other cases is as follows:

[0031] For all other cases, that is, when the predictions of the basic model and the expert model are inconsistent and the predictions of the expert model and the anti-expert model are inconsistent, the prediction results of the basic model and the expert model are used for comprehensive adjustment. The comprehensive adjustment of the prediction results of the basic model and the expert model specifically refers to the logits that meet the conditions next The update or replacement of the corresponding row vector in the matrix is ​​calculated as follows:

[0032]

[0033] Furthermore, the Step 3 includes:

[0034] (1) When the predictions of the basic model and the expert model are consistent:

[0035] When the prediction results of the basic model and the expert model are consistent, the logits of the expert model are selected for update; at this time, the prediction results of the expert model are more reliable and are directly used to generate the final output; the update formula is as follows:

[0036] logits next [match base_expert ] = logits expert [match base_expert ]

[0037] (2) When the predictions of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent:

[0038] When the prediction results of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent, it means that the prediction of the expert model may be biased. At this time, the logits of the basic model are selected for update. The update formula is as follows:

[0039] logits next [match expert_antiexpert_base_not ]

[0040] =logits base [matchexpert_antiexpert_base_not_ ]

[0041] (3) Other situations:

[0042] For all other cases, the linear combination of the base model logits plus the expert model logits minus the anti-expert model logits is used to update the prediction results of each model. The update formula is as follows:

[0043] logits next [other cases ]

[0044] =logits base [other cases ]+logits expert [other cases ]-logits antiexpert [other cases ]

[0045] logits next Represents the logist matrix and logits that are ultimately used to generate text predictions expert Represents the logist matrix of the expert model, other cases Indicates the row index that satisfies the "other conditions";

[0046] Furthermore, the Step 4 includes:

[0047] After completing the dynamic adjustment of logits, the last step is to convert the adjusted logits into probability distribution through the softmax function and generate the final predicted text based on the probability distribution; the softmax function is used to convert the unnormalized logits into probability values, and the formula is as follows:

[0048]

[0049] Among them, P(y i ) represents the probability of predicting the next word to be the i-th word in the vocabulary, N is the total number of words in the vocabulary, logits next,i Represents the adjusted logits of the i-th word;

[0050] In this way, the probability distribution of each possible output is obtained, and then the final predicted text is generated based on these probabilities; a greedy algorithm or a beam search algorithm is used to select the most likely token from the probability distribution and gradually generate the complete text.

[0051] The beneficial effects of the present invention are:

[0052] (1) Dynamically adjust parameters: By introducing a conditional judgment mechanism, the logits update strategy can be dynamically adjusted according to different input and prediction situations, which enhances the flexibility and adaptability of the method.

[0053] (2) Refined processing: When the predictions of the basic model and the expert model are consistent, the logits of the expert model are directly used; when the expert model and the anti-expert model are consistent and the basic model and the expert model are inconsistent, the logits of the basic model are used; in other cases, the linear combination method is still used. This refined processing method can more accurately reflect the complex relationship between model predictions.

[0054] (3) Improving prediction accuracy: Through more complex logical judgment and selective update mechanisms, the output information of expert models and anti-expert models can be more effectively utilized to improve the prediction accuracy and generation effect of large models.

[0055] (4) Enhanced adaptability: The improved proxy tuning method retains the advantages of the original method and further enhances the adaptability and performance of the model in various tasks and scenarios through refined processing and dynamic adjustment.

[0056] (5) The present invention can be widely used in the fields of intelligent customer service, content generation, education and training, medical treatment, legal compliance, financial services, and human resource management. Through the method of the present invention, the model output can be dynamically adjusted according to different inputs, the response accuracy and user satisfaction can be improved, high-quality text content can be generated, teaching and medical diagnosis can be assisted, legal documents and financial reports can be generated, etc., thereby significantly improving the work efficiency and effect in various fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the process flow of the large language model proxy tuning method with adaptive conditions provided by the present invention;

[0058] Figure 2 It is a schematic diagram of condition judgment in the present invention;

[0059] Figure 3 It is a schematic diagram of selective updating in the present invention. DETAILED DESCRIPTION

[0060] Example 1: Figure 1-Figure 3 As shown, a large language model proxy tuning method with adaptive conditions, the method comprising:

[0061] Step 1, Initialization: First, calculate the predicted logits of the base model, expert model, and anti-expert model; logits are the unnormalized scores generated by the model at the output layer, which are used to represent the relative likelihood of each possible output;

[0062] The base model is a large-scale pre-trained language model with broad versatility. Specifically, the base model has learned the basic structure and common sense of the language by training on massive text data, and can perform well in a variety of tasks. However, for specific tasks or fields, the performance of the base model may not be ideal.

[0063] The expert model is a fine-tuned version of the basic model for a specific task or field, which can provide more professional and accurate predictions. Fine-tuning means further training the pre-trained model using data from a specific task to make the model perform better on the task. The anti-expert model is an unfine-tuned version of the expert model, which is used to provide comparative information.

[0064] During the initialization process, the predicted logits of the base model, expert model, and anti-expert model are calculated. Logits are the unnormalized scores generated by the model at the output layer, which are used to represent the relative likelihood of each possible output. Let the predicted logits of the base model be logits base , the predicted logits of the expert model are logits expert , the predicted logits of the anti-expert model are logits antiexpert ;

[0065] The mathematical expression is as follows:

[0066] logits base =f base (x 1:i )

[0067] logits expert =f expert (x 1:i )

[0068] logits antiexpert =f antiexpert (x 1:i )

[0069] Among them, f base 、f expert and f antiexpert Respectively represent the prediction functions of the basic model, expert model and anti-expert model, x 1:i Represents an input text sequence.

[0070] Step 2, conditional judgment: through the conditional judgment mechanism, calculate the index where the predictions of the basic model and the expert model are consistent, the index where the predictions of the expert model and the anti-expert model are consistent and the predictions of the basic model and the expert model are inconsistent, and the index in other cases respectively; the purpose of this step is to dynamically adjust the output of the model according to different prediction situations. Figure 2 Shown is a schematic diagram of the conditional judgment mechanism in the present invention.

[0071] In Step 2, the indexes for calculating the consistency between the basic model and the expert model predictions include:

[0072] When the prediction results of the basic model and the expert model for the same input are consistent, it means that the prediction of the basic model on the input has a certain degree of credibility. The prediction results of the expert model are directly adopted. The prediction results of the expert model are directly adopted. The logist matrix (logits) used to generate the predicted text is used in the end. next In the logits table, the rows that satisfy the condition that the prediction results of the base model and the expert model for the same input are consistent are represented by expert The corresponding row vector replacement is calculated as follows:

[0073] match base_expert =(argmax(logits base )==argmax(logits expert ))

[0074] Among them, argmax is a commonly used operator, which is used to find the index of the maximum value in a vector or array. Suppose a vector v = [v1, v2, ..., v n ], the mathematical definition of the argmax operator is as follows:

[0075]

[0076] In other words, argmax returns the value that makes v i The index io,v at which the maximum value is reached i Indicates the maximum value of the elements in vector v, match base_expert It is a matrix with b rows and 1 column, which indicates the row index value that satisfies the condition that "the prediction results of the basic model and the expert model for the same input are consistent". "==" indicates that the "equal" relationship operation is performed on the elements at the corresponding positions of the matrix. If the two operands are equal in size, the operation result is 1, otherwise it is 0.

[0077] Furthermore, in Step 2, the indexes for which the predictions of the expert model and the counter-expert model are consistent and the predictions of the basic model and the expert model are inconsistent are calculated as follows:

[0078] When the prediction results of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent, it means that there may be problems with the prediction of the expert model on the input. At this time, the prediction results of the basic model are adjusted. The adjustment refers to the logit matrix used to generate the predicted text, i.e., logits. next The row vector that satisfies the condition "when the prediction results of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent" is replaced by logits base The corresponding row in the table; the calculation formula is as follows:

[0079] match expert_antiexpert =(argmax(logits expert )==argmax(logits antiexpert ))

[0080] not_match base_expert =(argmax(logits base )≠argmax(logits expert ))

[0081] match expert_antiexpert_base_not =match expert_antiexpert ∧not_match base_expert

[0082] Among them, match expert_antiexpert Indicates the row index value of "the expert model and the anti-expert model prediction are consistent", not_match base_expert Indicates the row index value of "the predictions of the basic model and the expert model are inconsistent", match expert_antiexpert_base_not Indicates the row index where the expert model and the anti-expert model predict the same thing but the basic model and the expert model predict the same thing. expert_antiexpert Indicates the row index that the expert model and the anti-expert model predict the same, not_match base_expert Indicates the row index where the predictions of the base model and the expert model are inconsistent; the index calculation for other cases is as follows:

[0083] For all other cases, that is, when the predictions of the basic model and the expert model are inconsistent and the predictions of the expert model and the anti-expert model are inconsistent, the prediction results of the basic model and the expert model are used for comprehensive adjustment. The comprehensive adjustment of the prediction results of the basic model and the expert model specifically refers to the logits that meet the conditions next The update or replacement of the corresponding row vector in the matrix is ​​calculated as follows:

[0084]

[0085] Specific:

[0086] logits base logits expert logits antiexpert is a matrix with b rows and v columns, where b refers to the number of input sentences and v refers to the dimension of the vocabulary. The conditional judgment in step 2 refers to calculating the logical relationship between the column indexes of the maximum values ​​of each row of the three matrices to obtain the row indexes in the three cases.

[0087] set up

[0088] Use argmax to calculate the column index of the maximum value in each row.

[0089] (1) Calculate the row index that the basic model and the expert model predict the same, argmax(logits base ) and argmax(logits expert ) Perform relational operations on the elements at the corresponding positions to obtain the row index that meets this condition 1 indicates a row that satisfies the condition, and 0 indicates a row that does not satisfy the condition.

[0090] (2) The expert model and the anti-expert model predict the same thing, but the basic model and the expert model predict the same thing. First calculate the row index value of "the expert model and the anti-expert model predict the same thing"

[0091]

[0092] Then calculate the row index value of "the predictions of the basic model and the expert model are inconsistent" Here, “!=” indicates that the “not equal” relationship operation is performed on the elements at the corresponding positions of the matrix. If the two operands are not equal in size, the result of the operation is 1, otherwise it is 0.

[0093] Finally, we can get the row indexes where the predictions of the expert model and the anti-expert model are consistent and the predictions of the basic model and the expert model are inconsistent.

[0094]

[0095] Where ∧ represents the logical "and" operation of the elements at the corresponding positions of the matrix. If the two operands are the same, the result of the operation is 1, otherwise it is 0.

[0096] (3) Other cases refer to cases other than the above two cases. The row index value that meets this condition is

[0097]

[0098] Where V represents a logical NOT operation and V represents a logical OR operation.

[0099] Step 3, Selective Update: Based on the conditional judgment results, selectively update the logits in different situations; this step further improves the prediction accuracy and flexibility of the model by distinguishing different situations. Figure 3 It is a schematic diagram of the selective update strategy in the present invention.

[0100] When the predictions of the basic model and the expert model are consistent, the logits of the expert model are selected for update; when the predictions of the expert model and the anti-expert model are consistent and the predictions of the basic model and the expert model are inconsistent, the logits of the basic model are selected for update; in other cases, the linear combination of the basic model logits plus the expert model logits minus the anti-expert model logits is used for update;

[0101] The Step 3 includes:

[0102] (1) When the predictions of the basic model and the expert model are consistent:

[0103] When the prediction results of the basic model and the expert model are consistent, the logits of the expert model are selected for update; at this time, the prediction results of the expert model are more reliable and are directly used to generate the final output; the update formula is as follows:

[0104] logits next [match base_expert ] = logits expert [match base_expert ]

[0105] (2) When the predictions of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent:

[0106] When the prediction results of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent, it means that the prediction of the expert model may be biased. At this time, the logits of the basic model are selected for update. The update formula is as follows:

[0107] logits next [match expert_antiexpert_base_not ]

[0108] =logits base [match expert_antiexpert_base_not_ ]

[0109] (3) Other situations:

[0110] For all other cases, the linear combination of the base model logits plus the expert model logits minus the anti-expert model logits is used to update the prediction results of each model. The update formula is as follows:

[0111] logits next [other cases ]

[0112] =logits base [other cases ]+logits expert [other cases ]-logits antiexpert [other cases ]

[0113] logits next Represents the logist matrix and logits that are ultimately used to generate text predictions expert Represents the logist matrix of the expert model, other cases Indicates the row index that satisfies the "other conditions";

[0114] Step 4. Generate predicted text: After completing the selective update of logits, the adjusted logits are converted into probability distribution through the softmax function, and the final predicted text is generated based on the probability distribution.

[0115] Through the above selective update mechanism, dynamic adjustment of logits under different input conditions is achieved, enhancing the flexibility and adaptability of the model. This dynamic adjustment can not only improve the prediction accuracy of the model, but also improve its generalization ability in multi-task scenarios.

[0116] After completing the dynamic adjustment of logits, the last step is to convert the adjusted logits into probability distribution through the softmax function and generate the final predicted text based on the probability distribution; the softmax function is used to convert the unnormalized logits into probability values, and the formula is as follows:

[0117]

[0118] Among them, P(y i ) represents the probability of predicting the next word to be the i-th word in the vocabulary, N is the total number of words in the vocabulary, logits next,i Represents the adjusted logits of the i-th word;

[0119] In this way, the probability distribution of each possible output is obtained, and then the final predicted text is generated based on these probabilities; a greedy algorithm (greedy decoding) or a beam search algorithm (beam search) is used to select the most likely token from the probability distribution and gradually generate the complete text.

[0120] Theoretically, the method of the present invention can better utilize the output information of the expert model and the anti-expert model by dynamically adjusting logits, thereby achieving refined control of the output of the large model. Through conditional judgment and selective update mechanism, the limitations brought by uniform adjustment of parameters can be effectively avoided, ensuring that the model can provide accurate and efficient predictions in different situations.

[0121] Specifically, the dynamic adjustment mechanism utilizes the idea of ​​ensemble learning in machine learning. Ensemble learning can usually achieve better performance than a single model by combining the prediction results of multiple models. Our selective update strategy is essentially a weighted integration method that dynamically adjusts weights according to different situations to improve the prediction ability of the model. In addition, the dynamic adjustment mechanism also draws on the idea of ​​contrastive learning to optimize the final output by comparing the prediction results of different models. This method can effectively improve the generalization ability of the model, especially when dealing with diverse inputs and complex tasks. The present invention has the following key functions:

[0122] (1) Conditional judgment mechanism: The present invention introduces a conditional judgment mechanism to dynamically adjust the logits update strategy according to different input and prediction situations. This mechanism enables the model to respond flexibly in different situations and avoids the limitations brought about by uniformly adjusting parameters.

[0123] (2) Selective update strategy: The present invention proposes a selective update strategy, which selectively updates the logits in different situations according to the conditional judgment results. Specifically, when the predictions of the basic model and the expert model are consistent, the logits of the expert model are selected; when the predictions of the expert model and the anti-expert model are consistent and the predictions of the basic model and the expert model are inconsistent, the logits of the basic model are selected; in other cases, a linear combination of the basic model logits plus the expert model logits minus the anti-expert model logits is used.

[0124] (3) Dynamically adjust logits: Through the above-mentioned conditional judgment mechanism and selective update mechanism, the present invention realizes dynamic adjustment of logits under different input conditions, enhancing the flexibility and adaptability of the model. This dynamic adjustment not only improves the prediction accuracy of the model, but also improves its generalization ability in multi-task scenarios;

[0125] In order to verify the effectiveness of the adaptive conditional large language model proxy tuning method (ACPT) of the present invention, experiments were conducted in several important application scenarios: code generation, mathematical calculations, and authenticity evaluation.

[0126] Experimental setup:

[0127] The 13B and 70B models in the LLaMA-2 model family were used as the base models, the LLaMA-2-7B model was used as the anti-expert model, CODELLaMA-7B-PYTHON was used as the expert model when evaluating code generation, and LLaMA-2-7B-CHAT was used as the expert model for the remaining tasks. The 13B and 70B models were adjusted through the proxy tuning method, hoping that their performance in the instruction tuning task can be close to or even exceed the effect of direct tuning.

[0128] Dataset:

[0129] Three evaluation datasets were used to test the performance of the model in different tasks, including:

[0130] CodexEval: An evaluation dataset that requires generating Python functions.

[0131] GSM: A dataset containing arithmetic word problems.

[0132] TruthfulQA: A dataset containing misleading questions for evaluating the truthfulness and informativeness of models. Experimental results:

[0133] The experimental results are shown in Table 1. The present invention significantly improves the performance of the basic model on multiple tasks and outperforms the original proxy tuning method on most tasks.

[0134] Table 1

[0135] Model CodexEval Accuracy GSM Accuracy TruthfulQA Accuracy Expert Model 68.04％ 23.2％ 55.91％ Base Model-13B 34.67％ 6.67％ 38.64％ Base Model - 13B + Original Agent Tuning 64.51％ 24.26％ 56.06％ Basic Model-13B+Invention 66.60％ 24.94％ 55.34％ Base Model - 70B 61.88％ 9.63％ 42.35％ Base model - 70B + original proxy tuning 72.42％ 33.28％ 57.65％ Basic model - 70B + invention 73.45％ 39.42％ 59.67％

[0136] The method of the present invention not only retains the advantages of the original proxy tuning method, but also significantly improves the performance of large language models in multiple tasks and scenarios by introducing more complex logical judgment and dynamic adjustment mechanisms. It has important application value and promotion prospects.

[0137] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A large language model proxy tuning method with adaptive conditions, characterized by: The method comprises: Step 1, Initialization: First, calculate the predicted logits of the base model, expert model, and anti-expert model; logits are the unnormalized scores generated by the model at the output layer, which are used to represent the relative likelihood of each possible output; Step 2, conditional judgment: through the conditional judgment mechanism, calculate the index where the predictions of the basic model and the expert model are consistent, the index where the predictions of the expert model and the anti-expert model are consistent and the predictions of the basic model and the expert model are inconsistent, and the indexes in other cases respectively; Step 3, Selective update: According to the conditional judgment results, the logits in different situations are selectively updated; when the predictions of the basic model and the expert model are consistent, the logits of the expert model are selected for update; when the predictions of the expert model and the anti-expert model are consistent and the predictions of the basic model and the expert model are inconsistent, the logits of the basic model are selected for update; in other cases, the linear combination of the basic model logits plus the expert model logits minus the anti-expert model logits is used for update; Step 4, Generate predicted text: After completing the selective update of logits, convert the adjusted logits into probability distribution through the softmax function, and generate the final predicted text based on the probability distribution; The Step 1 includes: During the initialization process, the predicted logits of the base model, expert model, and anti-expert model are calculated. Logits are the unnormalized scores generated by the model at the output layer, which are used to represent the relative likelihood of each possible output. Let the predicted logits of the base model be logits base , the predicted logits of the expert model are logits expert , the predicted logits of the anti-expert model are logits antiexpert ; The mathematical expression is as follows: logits base =f base (x 1:i ) logits expert =f expert (x 1:i ) logits antiexpert =f antiexpert (x 1:i ) Among them, f base 、f expert and f antiexpert Respectively represent the prediction functions of the basic model, expert model and anti-expert model, x 1:i Represents an input text sequence; In Step 2, the indexes for calculating the consistency between the basic model and the expert model predictions include: When the prediction results of the basic model and the expert model for the same input are consistent, the prediction results of the expert model are directly used. The prediction results of the expert model are directly used in the logist matrix (logits) used to generate the predicted text. next In the logits table, we use the logits function to represent the rows that satisfy the condition that the prediction results of the base model and the expert model for the same input are consistent. expert The corresponding row vector replacement is calculated as follows: match base_expert =(argmax(logits base )==argmax(logits expert )) Argmax is a commonly used operator, which is used to find the index of the maximum value in a vector or array. Suppose a vector v = [v1, v2, ..., v n ], the mathematical definition of the argmax operator is as follows: In other words, argmax returns the value that makes v i The index i, v where the maximum value is reached i Indicates the maximum value of the elements in vector v, match base_expert It is a matrix with b rows and 1 column, which represents the row index value that satisfies the condition that "the prediction results of the basic model and the expert model for the same input are consistent". "==" means that the "equal" relationship operation is performed on the elements at the corresponding positions of the matrix. If the two operands are equal in size, the operation result is 1, otherwise it is 0.

2. The method for optimizing a large language model agent with adaptive conditions according to claim 1, characterized in that: In Step 2, the indexes for which the predictions of the expert model and the counter-expert model are consistent and the predictions of the basic model and the expert model are inconsistent are calculated as follows: When the prediction results of the expert model and the anti-expert model are consistent, but the prediction results of the basic model and the expert model for the input are inconsistent, the prediction results of the basic model are adjusted. The adjustment refers to the logits matrix used to generate the predicted text. next The row vector that satisfies the condition "when the prediction results of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent" is replaced by logits base The corresponding row in the table; the calculation formula is as follows: match expert_antiexpert =(argmax(logits expert )==argmax(logits antiexpert ))not_match base_expert =(argmax(logits base )≠argmax(logits expert )) match expert_antiexpert_base_not =match expert_antiexpert ∧not_match base_expert Among them, match expert_antiexpert Indicates the row index value of "the expert model and the anti-expert model prediction are consistent", not_match base_expert Indicates the row index value of "the predictions of the basic model and the expert model are inconsistent", match expert_antiexpert_base_not Indicates the row index where the expert model and the anti-expert model predict the same thing but the basic model and the expert model predict the same thing. expert_antiexpert Indicates the row index that the expert model and the anti-expert model predict the same, not_match base_expert Indicates the row index where the predictions of the base model and the expert model are inconsistent; The index calculation for other cases is as follows: For all other cases, that is, when the predictions of the basic model and the expert model are inconsistent and the predictions of the expert model and the anti-expert model are inconsistent, the prediction results of the basic model and the expert model are used for comprehensive adjustment. The comprehensive adjustment of the prediction results of the basic model and the expert model specifically refers to the logits that meet the conditions next The update or replacement of the corresponding row vector in the matrix is ​​calculated as follows:

3. The method for optimizing a large language model agent with adaptive conditions according to claim 1, characterized in that: The Step 3 includes: (1) When the predictions of the basic model and the expert model are consistent: When the prediction results of the basic model and the expert model are consistent, the logits of the expert model are selected for update; at this time, the prediction results of the expert model are more reliable and are directly used to generate the final output; the update formula is as follows: logits next [match base_expert ]=logits expert [match base_expert ] (2) When the predictions of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent: When the prediction results of the expert model and the anti-expert model are consistent, but the predictions of the basic model and the expert model are inconsistent, it means that the prediction of the expert model may be biased. At this time, the logits of the basic model are selected for update. The update formula is as follows: logits next [match expert_antiexpert_base_not ] =logits base [match expert_antiexpert_base_not_ ] (3) Other situations: For all other cases, the linear combination of the base model logits plus the expert model logits minus the anti-expert model logits is used to update the prediction results of each model. The update formula is as follows: logits next [other cases ] =logits base [other cases ]+logits expert [other cases ] -logits antiexpert [other cases ] logits next Represents the logist matrix and logits that are ultimately used to generate text predictions expert Represents the logist matrix of the expert model, other cases Indicates the row index that satisfies the "other condition".

4. The method for optimizing a large language model agent with adaptive conditions according to claim 1, characterized in that: The Step 4 includes: After completing the dynamic adjustment of logits, the last step is to convert the adjusted logits into probability distribution through the softmax function and generate the final predicted text based on the probability distribution; the softmax function is used to convert the unnormalized logits into probability values, and the formula is as follows: Among them, P(y i ) represents the probability of predicting the next word to be the i-th word in the vocabulary, N is the total number of words in the vocabulary, logits next,i Represents the adjusted logits of the i-th word; In this way, the probability distribution of each possible output is obtained, and then the final predicted text is generated based on these probabilities; a greedy algorithm or a beam search algorithm is used to select the most likely token from the probability distribution and gradually generate the complete text.

Images