Model adjustment method and device, medium and program product

Abstract

The invention discloses a model adjustment method and device, a medium and a program product, and relates to the technical field of data processing, and the method comprises the steps: determining target reasoning data and at least one prefix data corresponding to each group of training data through a teacher model, and carrying out the fine adjustment of a student model based on the prefix data and the target reasoning data, no annotation needs to be added in the process, so that a large number of computing resources do not need to be consumed, the technical problem that related fine adjustment methods depend on manual annotation is solved, and the technical effect of improving the model fine adjustment efficiency is achieved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a model adjustment method, device, medium, and program product. Background Art

[0002] Knowledge Distillation refers to a model compression technology by which users can use a large "teacher" model to train and fine-tune a small "student" model.

[0003] Related fine-tuning methods include supervised fine-tuning (SFT), rejection sampling fine-tuning (RFT), self-learning thruster (STaR), etc. The above methods usually rely on manually labeled training data or a large amount of sampled data, consuming a large amount of computing resources and having a high cost, resulting in low efficiency of model fine-tuning. Summary of the Invention

[0004] This application provides a model adjustment method, device, medium, and program product to at least solve the problem of low adjustment efficiency of related inference models in related technologies.

[0005] This application provides a model adjustment method, including:

[0006] Input at least one set of training data into the teacher model to obtain target inference data corresponding to each training data and at least one prefix data; wherein, the training data includes question text data and answer text data; the target inference data includes first inference data generated by the teacher model based on a preset inference strategy; the prefix data includes multiple inference tokens in multiple second inference data randomly generated by the teacher model;

[0007] Fine-tune the student model based on each prefix data and each target inference data to obtain a target student model; wherein, the number of first parameters in the student model is less than the number of second parameters in the teacher model.

[0008] This application also provides a model adjustment device, including:

[0009] A first input module, configured to input at least one set of training data into the teacher model to obtain target inference data corresponding to each training data and at least one prefix data; wherein, the training data includes question text data and answer text data; the target inference data includes first inference data generated by the teacher model based on a preset inference strategy; the prefix data includes multiple inference tokens in multiple second inference data randomly generated by the teacher model;

[0010] A fine-tuning module, configured to fine-tune a student model based on each prefix data and each target inference data to obtain a target student model; wherein, the number of first parameters in the student model is less than the number of second parameters in the teacher model.

[0011] The present application further provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any one of the above model adjustment methods when executing the computer program.

[0012] The present application further provides a computer-readable storage medium, in which a computer program is stored, and wherein the computer program implements the steps of any one of the above model adjustment methods when executed by a processor.

[0013] The present application further provides a computer program product, including a computer program, and the computer program implements the steps of any one of the above model adjustment methods when executed by a processor.

[0014] Through the present application, the prefix data corresponding to each group of training data determined by the teacher model can be used, without adding annotations to the training data, and the student model can be fine-tuned based on the prefix data and the target inference data. Therefore, the technical problem that related fine-tuning methods rely on manual annotation can be solved, and the technical effect of improving the efficiency of model fine-tuning can be achieved. Description of the Drawings

[0015] To more clearly illustrate the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 It is a schematic application diagram of a model adjustment method provided by an embodiment of the present application;

[0017] Figure 2 It is a first schematic flowchart of a model adjustment method provided by an embodiment of the present application;

[0018] Figure 3 It is a second schematic flowchart of a model adjustment method provided by an embodiment of the present application;

[0019] Figure 4 It is a third schematic flowchart of a model adjustment method provided by an embodiment of the present application;

[0020] Figure 5 It is a schematic structural diagram of a model adjustment device provided by an embodiment of the present application;

[0021] Figure 6A schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0022] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0023] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0024] Chain of Thought (CoT) reasoning is a strategy used in Natural Language Processing (NLP) to improve the reasoning ability of models, especially in complex question-and-answer tasks. Chain of Thought reasoning requires the model to generate a series of intermediate reasoning steps before giving the final answer. The reasoning steps detail how the model starts from the question and arrives at the answer through logical reasoning, which helps the model gradually simplify the question until the answer is obtained. In addition, by showing the reasoning process, the interpretability of the model is enhanced, enabling users to understand how the model arrives at a specific answer. It is widely applied to various NLP tasks, such as mathematical problem-solving, common sense reasoning, program generation, etc. Usually, in these tasks, the model needs to demonstrate its understanding of the question and provide a detailed reasoning process.

[0025] The steps of reasoning in the Chain of Thought reasoning method mainly include: (1) Understanding the question: Input the question to the model, and the model understands the posed question; (2) Generating intermediate steps: The model generates a series of intermediate reasoning steps, each step being a logically coherent statement that together constitutes the complete reasoning process; (3) Obtaining the answer: When all necessary intermediate reasoning is completed, the model finally gives the answer to the question.

[0026] Pre-training is a strategy for training deep learning models. Its core lies in using a large-scale dataset to preliminarily train the model so that the model learns general feature representations. This process is similar to the basic learning stage of humans before learning new knowledge, where they accumulate experience through extensive reading and observation.

[0027] Fine-tuning refers to further training on a dataset for a specific task based on a pre-trained model to adjust the model parameters so that it can better adapt to the target task. During fine-tuning, most layers of the pre-trained model are usually frozen, and only the newly added layers or a small number of key layers are trained. This can not only retain the features learned by the pre-trained model but also quickly adapt to the specific requirements of the new task. In addition, choosing appropriate learning rates and the number of training epochs is also the key to successful fine-tuning.

[0028] Pre-trained language model: Generally, it refers to designing a language model training task based on a large-scale corpus (including language training materials such as sentences, paragraphs, etc.), training a large-scale neural network algorithm structure to learn and implement. The final large-scale neural network algorithm structure and parameters are the pre-trained language model. For subsequent other tasks, feature extraction or task fine-tuning can be performed on the basis of this model to achieve specific task purposes. The idea of pre-training is to first train a task to obtain a set of model parameters, then use this set of model parameters to initialize the network model parameters, and then use the initialized network model to train other tasks to obtain a model adapted to other tasks. By pre-training on a large-scale corpus, the neural language representation model can learn powerful language representation capabilities and extract rich syntactic and semantic information from the text. The pre-trained language model can provide word tokens (tokens) containing rich semantic information and sentence-level features for downstream tasks, or directly perform fine-tuning for downstream tasks on the pre-trained model, conveniently and quickly obtaining a downstream-specific model.

[0029] The neural network algorithm structure for pre-training the language model can be CNN, RNN, LSTM, etc., or a model constructed with an attention network, such as transformer, bert, GPT, Clip, etc. This application does not make any limitations here. The attention network refers to a network model trained using the attention mechanism. This model assigns different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence and enabling the model to finally obtain a more accurate output.

[0030] Knowledge Distillation refers to a model compression technique. Users can perform knowledge distillation on a large model to obtain a compact student model that can achieve similar accuracy on specific downstream tasks. This includes: creating or selecting a student model with a significantly smaller number of pre-trained parameters than the given teacher model, preparing a dataset related to the downstream task (such as the GSM8K dataset), using the teacher model to infer the problems in the dataset and output soft labels (usually the probability distribution of the output layer), and using the data and soft labels in the dataset to fine-tune the student model while optimizing both the distillation loss and the standard fine-tuning loss.

[0031] Related fine-tuning methods include Supervised Fine-Tuning (SFT), Rejection Sampling Fine-Tuning (RFT), Self-Taught Reasoner (STaR), etc. Among them, the Supervised Fine-Tuning (SFT) method can use manually annotated inference process data to fine-tune large language models (LLMs) and improve their inference capabilities. However, this method relies on a large amount of high-quality annotated data and has a high cost.

[0032] It can be understood that in some specific fields or tasks, it may be difficult to find sufficient annotated data to support model fine-tuning. Especially in the absence of annotation resources, it is relatively difficult to perform supervised fine-tuning.

[0033] Moreover, in some complex inference tasks (such as mathematical reasoning, code generation, etc.), it may take professionals several hours or even longer to annotate a complete inference process. In addition, if the annotated data contains biased or noisy data, it may lead to overfitting or poor performance of the model during the fine-tuning process.

[0034] Rejection Sampling Fine-Tuning (RFT) can fine-tune by generating multiple inference trajectories and selecting the correct ones. Although it does not require annotated data, it requires a large amount of sampling and computational resources. Self-Taught Reasoner (STaR) refers to an iterative self-learning method that can generate solutions through the model for self-verification and improvement, and also requires a large amount of computational resources and verification mechanisms. This limits the feasibility of these methods in practical applications, especially for users or institutions with limited resources.

[0035] That is to say, the above methods usually have problems such as relying on manually annotated training data or a large amount of sampled data, consuming a large amount of computational resources and having a high cost, resulting in low efficiency of model fine-tuning.

[0036] To solve the above problems, the embodiments of the present application provide a model adjustment method, device, medium, and program product. By using a teacher model, target inference data and at least one prefix data corresponding to each group of training data are determined, and the student model is fine-tuned based on each prefix data and each target inference data. During the process, no annotations need to be added, so a large amount of computing resources do not need to be consumed, the fine-tuning cost is reduced, and thus the efficiency of model fine-tuning is improved.

[0037] The model adjustment method provided by the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, personal computers (PCs), etc., and no specific limitations are made here.

[0038] To enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Combined with the specific application environment architecture or specific hardware architecture on which the execution of the model adjustment method depends, the specific application environment architecture or specific hardware architecture is described here. Figure 1 It is an application schematic diagram of a model adjustment method provided by the embodiments of the present application.

[0040] See Figure 1 , the electronic device includes a teacher model and a student model. Among them, the teacher model is a first large language model that has been pre-trained and is used to perform answer inference on problem text data (which can also be called the first pre-trained language model for question answering), and the student model is a second large language model that is used to perform answer inference on problem text data (which can also be called the second pre-trained language model for question answering), and the number of first parameters in the student model is less than the number of second parameters in the teacher model. After the user inputs at least one group of training data (the training data includes problem text data and answer text data) into the electronic device, the teacher model processes the training data to obtain the corresponding prefix data and target inference data, and inputs the problem text data, the corresponding answer text data, the prefix data, and the target inference data into the student model to fine-tune the student model to obtain the target student model. In this way, the target student model can generate corresponding model inference data based on the input problem data and answer data. The model inference data refers to the inference process information output by the model and based on the problem data to obtain the answer data.

[0041] Figure 2 It is one of the flow schematic diagrams of the model adjustment method provided by the embodiments of the present application. As Figure 2 shown, the embodiments of the present application provide a model adjustment method, and the method is described in detail as follows:

[0042] S201: Input at least one set of training data into the teacher model to obtain the target inference data corresponding to each training data and at least one prefix data; wherein, the training data includes question text data and answer text data; the target inference data includes the first inference data generated by the teacher model based on a preset inference strategy; the prefix data includes multiple inference tokens in multiple second inference data randomly generated by the teacher model.

[0043] Optionally, each set of training data includes, but is not limited to, question text data and answer text data. Among them, the question text data is the text data used to describe question information, for example, mathematical question information, or code generation question information; the answer text data is the text data used to describe the answer information corresponding to the question information, for example, the mathematical answer information corresponding to the mathematical question information, or the code answer information corresponding to the code generation question information.

[0044] Optionally, input at least one set of training data into the teacher model, and use the teacher model to generate a text sequence containing multiple-step reasoning explanations based on each question text data and the corresponding answer text data in at least one set of training data. Each sequence includes the question text data, reasoning steps (which can also be called inference data), and answer text data. Based on the above reasoning steps, determine the target inference data and at least one prefix data corresponding to each training data. The target inference data refers to the first inference data composed of multiple first inference tokens generated by the teacher model based on a preset inference strategy and corresponding to each set of training data. The prefix data refers to the text data composed of the second inference tokens in the prefix part of each second inference data randomly generated by the teacher model and corresponding to each set of training data.

[0045] Exemplarily, the teacher model can be the large language model Llama3 model or the mathematical question model Qwen-Math. The student model can be the large language model Llam3-70b or the large language model Llam3-8b.

[0046] S202: Fine-tune the student model based on each prefix data and each target inference data to obtain the target student model; wherein, the number of first parameters in the student model is less than the number of second parameters in the teacher model.

[0047] Optionally, input each question text data, the corresponding answer text data, prefix data, and target inference data into the student model. Based on each question text data, the corresponding answer text data, and prefix data, perform a preliminary fine-tuning on the student model. Then, based on each question text data, the corresponding answer text data, and target inference data, perform a secondary fine-tuning on the student model, that is, use the intermediate inference steps generated by the teacher model corresponding to the question text data and answer text data for the fine-tuning of the student model, so that the student model can learn the core inference ability of the teacher model to obtain the target student model, enabling the target student model to output the corresponding model inference data based on the input question data and answer data.

[0048] Optionally, knowledge distillation refers to training a small "student" model to imitate the behavior of a large "teacher" model, which can reduce the computational and storage requirements of the model while maintaining performance and is more easily integrated into practical applications. The model adjustment method provided in the embodiments of this application can adopt the Chain of Thought (CoT) distillation method.

[0049] Exemplarily, when the training data belongs to datasets with inference steps such as the GSM8K dataset, the few-shot machine learning method can be used to generate inference steps to increase the accuracy of the inference steps generated by the model.

[0050] In the embodiments of this application, the teacher model outputs the prefix text data of the inference content to generate a prefix dataset with low acquisition difficulty and no need for additional annotation or calibration. Then, a small number of complete target inference data are generated through the Chain of Thought (CoT) inference. By combining these two types of data, a double fine-tuning is performed on the student model to obtain a target student model with a smaller number of parameters than the teacher model and can achieve a performance similar to that of the teacher model in logical reasoning tasks, which is suitable for downstream tasks that require theoretical inference or logical calculation. Moreover, due to the small number of parameters of the target student model, it is applicable to various actual application scenarios.

[0051] Figure 3 This is the second flowchart of the model adjustment method provided in the embodiments of this application. As Figure 3 shown, step S201 includes:

[0052] S2011: Input at least one set of training data into the teacher model to obtain the first inference data corresponding to each training data generated by the teacher model based on a preset inference strategy;

[0053] S2012: For each training data, determine the corresponding first inference data as the target inference data;

[0054] S2013: Input at least one set of training data into the teacher model to obtain at least one second inference data corresponding to each training data output by the teacher model;

[0055] S2014: Generate corresponding prefix data for each training data based on each second inference data.

[0056] Optionally, input each group of training data into the teacher model to obtain the first inference tokens corresponding to each training data generated by the teacher model based on a preset inference strategy. Use each training data and its corresponding first inference tokens as new training data, and repeatedly execute the above step of the teacher model inferring new first inference tokens based on the input training data, so as to obtain the first inference data corresponding to each training data and determine it as the target inference data.

[0057] Optionally, input at least one group of training data into the teacher model at least once to obtain at least one second inference data corresponding to each group of training data output by the teacher model. For each group of training data, generate prefix data corresponding to each of the at least one second inference data in the corresponding at least one second inference data.

[0058] Optionally, each second inference data includes at least one second inference token (the second inference token can also be referred to as a character, or token).

[0059] It can be understood that the process of the teacher model generating multiple second inference data for the same group of question text data and answer text data has a certain degree of randomness. That is to say, the at least one second inference data corresponding to the same group of question text data and answer text data by the teacher model has a certain similarity. The prefix data generated based on the above second inference data can be used as training data to fine-tune the student model, so that the student model can learn a process of the teacher model performing multiple inferences on the same group of question text data and answer text data. That is to say, the student model can not only learn the knowledge of prefix inference but also retain the original inference structure, thus ensuring the overall performance. And the teacher model does not need to verify the accuracy when generating prefix data, nor does it need to perform additional annotation, which can reduce the computing power cost required for manual annotation.

[0060] Optionally, the training data includes at least one token, which can also be referred to as a training token, a character, or a token. The teacher model includes multiple candidate tokens, and the candidate tokens can indicate tokens (or characters) corresponding to the language type of the training data.

[0061] Figure 4 This is the third flow diagram of the model adjustment method provided by the embodiments of the present application. As Figure 4 shown, step S2011 includes:

[0062] S20111: Input each training data into the teacher model to determine the corresponding target log difference for each training data; the target log difference is used to indicate the log probability difference between the training data and each candidate token.

[0063] S20112: For each training data, determine the candidate token corresponding to the largest log difference as the first inference token.

[0064] S20113: For each training data, update the training data based on the first inference token, and return to execute the steps of inputting each training data into the teacher model to determine the corresponding target log difference for each training data and subsequent steps until a stop character is detected, and obtain the first inference data corresponding to each training data.

[0065] Optionally, input each training data into the teacher model to determine the corresponding target log difference for each training data. Among them, the target log difference is used to indicate the log probability difference between the training data and each candidate token.

[0066] Taking the language type of Chinese as the training data as an example, assuming that the number of all tokens (or characters) in Chinese is 30,000, the target log difference can be used to indicate the log probability difference between the training data and each candidate token among the 30,000 candidate tokens.

[0067] Optionally, for each training data, compare according to the numerical value of the target log difference, and determine the candidate token corresponding to the largest target log difference as the first inference token corresponding to the training data.

[0068] Optionally, for each training data, update the training data based on each training data and the corresponding first inference token (that is, add the corresponding first inference token to the training data) to obtain new training data, and input the new training data into the teacher model to obtain a new first inference token (that is, return to execute the steps of inputting each training data into the teacher model to determine the corresponding target log difference for each training data and subsequent steps), until a stop character is detected, and based on each first inference token corresponding to each training data, obtain the first inference data corresponding to each training data, and determine the first inference data as the target inference data.

[0069] Optionally, the stop character is used to indicate the token in the candidate tokens that represents the end information.

[0070] Exemplarily, the stop character can be a "period". Another exemplarily, the stop character can be an empty character.

[0071] It can be understood that the target inference data includes the first inference token corresponding to the training data output by the teacher model each time.

[0072] For example, the tokens in training data A include a1, a2, and a3. Among them, the candidate token corresponding to the target log difference with the largest value in training data A is a4, then a4 is determined as the first inference token corresponding to training data A. Update a1, a2, a3, and a4 to new training data A and input it into the teacher model, determine the new target log difference of training data A, and determine the candidate token corresponding to the new target log difference with the largest value as a5 as the new first inference token. Update a1, a2, a3, a4, and a5 to another new training data A and input it into the teacher model to obtain another new first inference token, and loop until a stop character is detected. Determine each first inference token (such as a4, a5, etc.) as the target inference data corresponding to training data A (i.e., a1, a2, a3) (i.e., including a4, a5...).

[0073] Optionally, input each training data into the teacher model, and determine the target log difference corresponding to each training data, including:

[0074] Input each training data into the teacher model, and determine the first probability based on the question text data and answer text data of each training data;

[0075] Determine the second probability based on the question text data of each training data;

[0076] For each training data, determine the log difference between the corresponding first probability and the second probability as the target log difference.

[0077] Optionally, input each training data into the teacher model, and determine the first probability of each training data and each candidate token based on the question text data and answer text data of each training data (including each token in the question text data and answer text data pool and the first inference token corresponding to the training data).

[0078] Optionally, based on the question text data in each training data (including each token in the question text data and the first inference token corresponding to the training data), determine the second probability of each training data and each candidate token, calculate the log difference between the first probability and the second probability of each training data, and determine it as the target log difference corresponding to each training data.

[0079] By way of example and not limitation, when updating the training data based on the first inference tokens corresponding to each training data to obtain new training data and inputting the new training data into the teacher model, the target log difference is used to indicate the log probability difference between each of the first inference tokens corresponding to the training data and each candidate token. That is, when the input data is the new training data, the first probability is the probability determined based on the question text data, the answer text data of each training data, and each of the first inference tokens corresponding to each training data, and the second probability is the probability determined based on the question text data of each training data and each of the first inference tokens corresponding to each training data.

[0080] Exemplarily, the target log difference can be calculated by the following formula:

[0081] ;

[0082] where, represents the first probability, represents the second probability, q represents the question text data, represents the answer text data corresponding to the question text data q, represents that the answer text data is empty. When the data input into the teacher model is the original training data, represents the inference token corresponding to the original training data (this inference token can also be called a candidate token and is any one of the candidate tokens), represents each token included in the question text data and the answer text data in the original training data, represents each token included in the question text data in the original training data.

[0083] And, when the training data input into the teacher model is the new training data, represents the i-th inference token corresponding to the training data (this inference token can also be called a candidate token and is any one of the candidate tokens), 、 both represent each of the first inference tokens before the i-th inference token. In this way, when the question text data is q and the answer text data is , the first probability of generating (which can also be called determining among the candidate tokens) the corresponding new inference token can be calculated respectively. And when the input question text data is q and the input empty answer text data is , the second probability of generating (which can also be called determining among the candidate tokens) the corresponding new inference token is calculated, and so on until the stop character is detected, thereby determining the target log difference between the training data (including the question text data q, the answer text data ) and each candidate token.

[0084] It can be understood that by calculating the logarithmic difference between the first probability and the second probability , it can be obtained that: the inference token corresponding to the training data input to the teacher model , the growth rate of confidence when inferring the correct answer compared to inferring an empty character. The larger the growth rate of confidence, the stronger the correlation between the inference token and the correct answer. That is to say, the inference data generated by the model more strongly supports the correct answer. The reason for setting the comparison answer as an empty character is to penalize the tokens when not considering the correct answer , and the above tokens may not be specific to the correct answer.

[0085] Therefore, among the target logarithmic differences between the training data and each candidate token, the candidate token corresponding to the largest target logarithmic difference can be determined as the first inference token, as shown in the following formula:

[0086] ;

[0087] wherein, represents the i-th inference token corresponding to the training data (this inference token can also be called a candidate token, and is any one of the candidate tokens), represents the i-th first inference token output by the teacher model.

[0088] Still taking the above training data A as an example, when the training data input to the teacher model is the original training data A (including the token a1, a2, a3 of the question text data and the answer text data), represents a1, a2, a3, represents, among the candidate tokens, the inference token that can be inferred based on a1, a2, a3. When determining the first inference token corresponding to a1, a2, a3 based on the logarithmic difference is a4, the data input to the teacher model is the new training data A (including the token a1, a2, a3 of the question text data and the answer text data and the corresponding first inference token a4). At this time represents a4, represents, among the candidate tokens, the inference token that can be inferred based on a4. And, when determining the first inference token corresponding to a1, a2, a3, a4 based on the logarithmic difference is a5, the data input to the teacher model is the new training data A (including the token a1, a2, a3 of the question text data and the answer text data and the corresponding first inference token a4, a5). At this time represents a4 and a5, It represents the inferred token that can be inferred based on a4 and a5 among each candidate token. By analogy, multiple first inferred tokens output by the teacher model are obtained, constituting a complete inference process including a small number of inference steps.

[0089] Optionally, the second inference data includes multiple second inferred tokens. For each training data, corresponding prefix data is generated based on each second inference data, including:

[0090] For each training data, determine the number of second inferred tokens in each second inference data and the order of each second inferred token.

[0091] Based on the number of second inferred tokens in each second inference data, determine the first quantity, where the first quantity is the number of second inferred tokens in the prefix data corresponding to each second inference data.

[0092] For each second inference data, based on the order of each second inferred token, sequentially select the first quantity of second inferred tokens from each second inferred token and determine them as the prefix data.

[0093] Optionally, for each group of training data, determine the number of second inferred tokens in each second inference data corresponding to each training data, and the order of each second inferred token in the corresponding second inference data. Based on the number of second inferred tokens in each second inference data, determine the number of second inferred tokens in the corresponding prefix data (i.e., the first quantity). For each second inference data, based on the order of each second inferred token in the corresponding second inference data, the first quantity of second inferred tokens can be sequentially selected from each second inferred token, and the above first quantity of second inferred tokens is determined as the prefix data corresponding to the second inference data.

[0094] Suppose the teacher model generates m pieces of second inference data for each question q , where m can be set according to the scale of the dataset and there is no need to verify whether the final result is correct. For each second inference data c, take the first k second inferred tokens as the prefix data, where k is the first quantity. For example, when the first quantity k = 8, take the first 8 second inferred tokens of the second inference data as the prefix data. The first quantity k can also be called a hyperparameter, and an appropriate value of k can be selected through experiments.

[0095] It can be understood that if the number of parameters of the teacher model is small (such as Llama), then k can be set to a small value. If the professional inference ability of the teacher model is strong and the number of parameters is large (such as the math problem model Qwen - Math), then k can be set to a large value. Among them, the number of prefix datasets should be greater than the training dataset.

[0096] It can be understood that truncating the second inference data and only obtaining the previous second inference tokens in the second inference data as prefix data, without the need to concern whether the second inference data and the inference result are correct, enables the generated prefix data to be random. Different inference steps can be generated by adjusting the second inference parameters of the teacher model, thereby generating different prefix data. For example, by increasing the model parameter Temperature, or increasing the value of the model parameter top-k, etc.

[0097] It can be understood that the initial inference tokens in the inference data corresponding to different inference structures (for example, the first 10 - 15 second inference tokens in the second inference data) are generally consistent. That is to say, even if the subsequent second inference tokens in the second inference data may be different, the initial second inference tokens are usually similar, and logical reasoning errors generally occur in the latter half of the inference tokens in the second inference data. Therefore, only intercepting the initial second inference tokens of the second inference data as prefix data to fine-tune the student model can guide the inference process of the student model, enabling it to learn the key feature information of the problem and make correct inferences and judgments at the initial stage, thereby improving the overall inference accuracy of the learning.

[0098] Among them, the first quantity can be specifically set according to the actual situation. Exemplarily, the first quantity is a preset proportion of the number of second inference tokens, and the preset proportion can be 40%, 45% or 50%. For example, if the number of second inference tokens in a second inference data is 30 and the preset proportion is 40%, the first 12 second inference tokens corresponding to this second inference data are the prefix data of this second inference data. Another example, if the number of tokens output by a second inference data is 32 and the preset proportion is 45%, the first 15 second inference tokens corresponding to this second inference data are the prefix data of this second inference data.

[0099] Optionally, fine-tuning the student model based on each prefix data and each target inference data to obtain a target student model, including:

[0100] Fine-tuning the student model based on each prefix data corresponding to each training data using a first loss function to obtain a first student model;

[0101] Fine-tuning the first student model based on the target inference data corresponding to each training data using a second loss function to obtain a target student model.

[0102] Optionally, the problem text data in each training data, the respective inference data corresponding to each training data, and each prefix data are input into the student model, and the student model is fine-tuned using the first loss function, enabling the student model to learn the inference information features of each prefix data in the teacher model during the process of inferring the answer data for the problem text data, thereby obtaining the first student model. The problem text data, the answer text data, the respective inference data corresponding to each training data, and each target inference data in each training data are input into the first student model, and the first student model is fine-tuned using the second loss function, enabling the student model to learn the inference information features in the target inference data during the process of inferring the answer data for the problem text data in the teacher model, thereby obtaining the target student model.

[0103] In the embodiment of the present application, by determining the inference data output by the teacher model, prefix data that does not require annotation and a small amount of complete target inference data are determined to construct a fine-tuning data set, and the student model is fine-tuned in two stages, enabling the student model to not only learn the knowledge of prefix inference in the teacher model but also retain the original inference structure, thereby ensuring the overall performance. Accordingly, the step-by-step inference ability of the teacher model is possessed, and the performance similar to that of the teacher model can be achieved in a specific task, while being applicable to multiple task scenarios.

[0104] Optionally, in order to avoid the student model losing the inference structure due to fine-tuning the student model only using the prefix data, and at the same time to enable the student model to have a complete logical inference ability similar to that of the teacher model, a second fine-tuning is required on a data set including complete inference steps (such as including each problem text data, each answer text data, the respective corresponding inference data, and the target inference data, etc.).

[0105] Optionally, the first loss function is determined based on the third probability of each problem text data; the third probability is the ratio between each prefix data corresponding to each problem text data and the subsequent inference token; the subsequent inference token is used to indicate the second inference token in the second inference data that is located after the prefix data.

[0106] Optionally, the expression of the first loss function is:

[0107] ;

[0108] where q represents the problem text data, represents the l-th prefix data corresponding to the problem text data q; represents the subsequent inference token of the l-th prefix data corresponding to the problem text data q, represents the third probability of the problem text data q.

[0109] Optionally, the first loss function is determined based on the third probability of each problem text data; the third probability is the ratio between each prefix data corresponding to each problem text data and the corresponding subsequent inference token.

[0110] Among them, the subsequent inference token is used to indicate the inference token in the inference data that is after the prefix data.

[0111] For example, if the prefix data is the first 10 second inference tokens in the second inference data corresponding to the problem text data, then the subsequent inference token is the 11th second inference token in the second inference data.

[0112] It can be understood that the smaller the value of the third probability, the more convergent the first loss function is. When the first loss function converges, the first student model is obtained.

[0113] Optionally, the second loss function is determined based on the fourth probability of each problem text data. The fourth probability is the ratio between each first inference token in the target inference data corresponding to each problem text data and the corresponding prior inference token. The prior inference token is used to indicate each first inference token before the first inference token.

[0114] Optionally, the expression of the second loss function is:

[0115] ;

[0116] Among them, q represents the problem text data, represents the i-th first inference token in the target inference data corresponding to the problem text data q, represents the prior inference token corresponding to the above i-th first inference token, represents the fourth probability of the problem text data q.

[0117] Optionally, the prior inference token can be used to indicate each first inference token in the target inference data before the current first inference token, corresponding to can also be referred to as each first inference token in the target inference data before the above i-th first inference token.

[0118] Still taking the above training data A as an example, the training data A includes the problem text data q. When the training data A input into the first student model includes a1, a2, a3, and the target inference data includes a4, a5, then represents the 1st first inference token a4 in the target inference data corresponding to the problem text data q, represents the prior inference token (empty) corresponding to the above 1st first inference token a4, and Denote the second first inference token a5 in the target inference data corresponding to the problem text data q, Denote the prior inference token a4 corresponding to the second first inference token a5 above, and so on.

[0119] It can be understood that the smaller the fourth probability value, the more convergent the second loss function. When the second loss function converges, the target student model is obtained.

[0120] Optionally, after fine-tuning the student model based on each prefix data and each target inference data to obtain the target student model, it includes:

[0121] Obtain at least one set of target data; each target data includes target problem text data, target answer text data, and target inference text data;

[0122] Input each target problem text data and the corresponding target answer text data into the target student model to obtain the model inference data output by the target student model corresponding to each target problem text data;

[0123] Determine the first similarity between each model inference data and the corresponding target inference text data;

[0124] When each first similarity is greater than or equal to the first threshold, determine that the adjustment of the target student model is completed.

[0125] Optionally, obtain at least one set of target data, and each set of target data includes a target problem text data, target answer text data, and target inference text data. Input each target problem text data and the corresponding target answer text data into the target student model, so that the target student model reasons about the target problem text data to obtain the inference information of the target answer text data, thereby obtaining the model inference data output by the target student model corresponding to each target problem text data. Calculate the first similarity between each model inference data and the target inference text data corresponding to each model inference data. When the first similarity corresponding to each target data is greater than or equal to the first threshold, determine that the adjustment of the target student model is completed, so as to deploy the adjusted target student model in the actual application scenario.

[0126] Among them, the first threshold can be determined according to the actual situation. For example, the first threshold is 85%, or the first threshold is 90%.

[0127] Optionally, the first similarity can be the character similarity between the model inference data and the corresponding target inference text data.

[0128] Optionally, determining the first similarity between each model inference data and the corresponding target inference text data includes:

[0129] Determine at least one first token in each model inference data and at least one second token in the corresponding target inference text data;

[0130] Determine the matching degree between at least one first token and at least one second token, and determine the matching degree as the first similarity.

[0131] Optionally, determine at least one first token in each model inference data and at least one second token in the target inference text data, and match each first token with each second token to determine the similarity between at least one first token in each model inference data and at least one second token in the target inference text data as the first similarity of the corresponding target data.

[0132] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation manner.

[0133] Figure 5 It is a schematic structural diagram of a model adjustment device provided by an embodiment of the present application. As Figure 5 shown, an embodiment of the present application also provides a model adjustment device, including:

[0134] A first input module 501, configured to input at least one set of training data into a teacher model to obtain target inference data corresponding to each training data and at least one prefix data; wherein, the training data includes question text data and answer text data; the target inference data includes first inference data generated by the teacher model based on a preset inference strategy; the prefix data includes multiple inference tokens in multiple second inference data randomly generated by the teacher model;

[0135] A fine-tuning module 502, configured to fine-tune a student model based on each prefix data and each target inference data to obtain a target student model; wherein, the number of first parameters in the student model is less than the number of second parameters in the teacher model.

[0136] Optionally, the first input module includes:

[0137] A first input unit, configured to input at least one set of training data into a teacher model to obtain first inference data generated by the teacher model based on a preset inference strategy and corresponding to each training data;

[0138] A determination unit, configured to, for each training data, determine the corresponding first inference data as the target inference data;

[0139] A second input unit for inputting at least one set of training data into the teacher model to obtain at least one second inference data corresponding to each training data output by the teacher model;

[0140] A generation unit for generating corresponding prefix data for each training data based on each second inference data.

[0141] Optionally, the teacher model includes multiple candidate tokens; the first input unit includes:

[0142] A first determination subunit for inputting each training data into the teacher model to determine the target log difference corresponding to each training data; the target log difference is used to indicate the log difference in probability between the training data and each candidate token;

[0143] A second determination subunit for, for each training data, determining the candidate token corresponding to the largest log difference as the first inference token;

[0144] A third determination subunit for, for each training data, updating the training data based on the first inference token and returning to execute the steps of inputting each training data into the teacher model to determine the target log difference corresponding to each training data and subsequent steps until a stop character is detected, to obtain the first inference data corresponding to each training data.

[0145] Optionally, the first determination subunit is specifically configured to:

[0146] Input each training data into the teacher model, and determine the first probability based on the question text data and answer text data of each training data;

[0147] Determine the second probability based on the question text data of each training data;

[0148] For each training data, determine the log difference between the corresponding first probability and the second probability as the target log difference.

[0149] Optionally, the second inference data includes multiple second inference tokens, and the generation unit is specifically configured to:

[0150] For each training data, determine the number of second inference tokens of each second inference data and the order of each second inference token;

[0151] Based on the number of second inference tokens of each second inference data, determine a first number, where the first number is the number of second inference tokens in the prefix data corresponding to each second inference data;

[0152] For each second inference data, based on the order of each second inference token, sequentially select the first number of second inference tokens from each second inference token and determine them as the prefix data.

[0153] Optionally, the fine-tuning module includes:

[0154] A first fine-tuning unit, configured to fine-tune the student model based on each prefix data corresponding to each training data by using a first loss function to obtain a first student model;

[0155] A second fine-tuning unit, configured to fine-tune the first student model based on the target inference data corresponding to each training data by using a second loss function to obtain a target student model.

[0156] Optionally, the first loss function is determined based on a third probability of each question text data; the third probability is a ratio between each prefix data corresponding to each question text data and the corresponding subsequent inference token; the subsequent inference token is used to indicate a second inference token in the second inference data that is located after the prefix data.

[0157] Optionally, the expression of the first loss function is:

[0158] ;

[0159] where q represents the question text data, represents the l-th prefix data corresponding to the question text data q; represents the subsequent inference token of the l-th prefix data corresponding to the question text data q, represents the third probability of the question text data q.

[0160] Optionally, the second loss function is determined based on a fourth probability of each question text data, the fourth probability is a ratio between each first inference token in the target inference data corresponding to each question text data and the corresponding prior inference token, and the prior inference token is used to indicate each first inference token located before the first inference token.

[0161] Optionally, the expression of the second loss function is:

[0162] ;

[0163] where q represents the question text data, represents the i-th first inference token in the target inference data corresponding to the question text data q, represents the prior inference token corresponding to the i-th first inference token, represents the fourth probability of the question text data q.

[0164] Optionally, the model adjustment device includes:

[0165] An acquisition module, configured to acquire at least one group of target data; each target data includes target question text data, target answer text data, and target inference text data;

[0166] A second input module, configured to input each piece of target problem text data and the corresponding target answer text data into a target student model, so as to obtain model inference data output by the target student model and corresponding to each piece of target problem text data;

[0167] A first determination module, configured to determine a first similarity between each piece of model inference data and the corresponding target inference text data;

[0168] A second determination module, configured to determine that the adjustment of the target student model is completed when each first similarity is greater than or equal to a first threshold.

[0169] Optionally, the second determination module is specifically configured to:

[0170] Determine at least one first token in each piece of model inference data and at least one second token in the corresponding target inference text data;

[0171] Determine the matching degree between at least one first token and at least one second token, and determine the matching degree as the first similarity.

[0172] For the description of the features in the corresponding embodiment of the model adjustment device, reference may be made to the relevant description of the corresponding embodiment of the model adjustment method, which will not be elaborated here one by one.

[0173] Figure 6 The structural schematic diagram of the electronic device provided by this application is shown as Figure 6 shown. The electronic device provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the electronic device further includes a communication component 603. Among them, the processor 601, the memory 602, and the communication component 603 are connected through a bus.

[0174] In a specific implementation process, at least one processor 601 executes computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above-mentioned model adjustment method embodiment.

[0175] For the specific implementation process of the processor 601, reference may be made to the above-mentioned method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0176] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly implemented by the execution of the hardware processor, or implemented by the combination of the hardware and software modules in the processor.

[0177] The memory may include random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk memory.

[0178] The bus may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0179] The embodiments of the present application also provide a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above-described model adjustment method embodiments when running.

[0180] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs and other media that can store computer programs.

[0181] The embodiments of the present application also provide a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-described model adjustment method embodiments.

[0182] Embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-described model adjustment method embodiments are implemented.

[0183] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0184] The above has introduced in detail a model adjustment method, device, medium, and program product provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A model adjustment method, characterized in that, Including: Input at least one set of training data into the teacher model to obtain target inference data corresponding to each piece of the training data and at least one prefix data; wherein, the training data includes question text data and answer text data; the target inference data includes first inference data generated by the teacher model based on a preset inference strategy; the prefix data includes multiple inference tokens among multiple second inference data randomly generated by the teacher model. Fine-tune the student model based on each piece of the prefix data and each piece of the target inference data to obtain a target student model; wherein, the number of first parameters in the student model is less than the number of second parameters in the teacher model.

2. The model adjustment method according to claim 1, wherein The step of inputting at least one set of training data into the teacher model to obtain target inference data corresponding to each piece of the training data and at least one prefix data includes: Input the at least one set of training data into the teacher model to obtain first inference data corresponding to each piece of the training data, which is generated by the teacher model based on the preset inference strategy. For each piece of the training data, determine the corresponding first inference data as the target inference data. Input the at least one set of training data into the teacher model to obtain at least one second inference data corresponding to each piece of the training data output by the teacher model. For each piece of the training data, generate corresponding prefix data based on each piece of the second inference data.

3. The model adjustment method according to claim 2, wherein The teacher model includes multiple candidate tokens; the step of inputting the at least one set of training data into the teacher model to obtain first inference data corresponding to each piece of the training data, which is generated by the teacher model based on the preset inference strategy, includes: Input each piece of the training data into the teacher model to determine the target log difference corresponding to each piece of the training data; the target log difference is used to indicate the log difference in probability between the training data and each candidate token. For each piece of the training data, determine the candidate token corresponding to the largest log difference as the first inference token. For each piece of the training data, update the training data based on the first inference token, and return to execute the steps of inputting each piece of the training data into the teacher model to determine the target log difference corresponding to each piece of the training data and subsequent steps until a stop character is detected, to obtain first inference data corresponding to each piece of the training data.

4. The model adjustment method according to claim 3, characterized in that The step of inputting each piece of the training data into the teacher model to determine the target log difference corresponding to each piece of the training data includes: Input each piece of the training data into the teacher model, and determine a first probability based on the question text data and answer text data of each piece of the training data. Determine a second probability based on the question text data of each piece of the training data. For each piece of the training data, determine the log difference between the corresponding first probability and the second probability as the target log difference.

5. The model adjustment method according to claim 2, wherein The second inference data includes multiple second inference tokens; the step of generating corresponding prefix data based on each piece of the second inference data for each piece of the training data includes: For each piece of the training data, determine the number of second inference tokens of each piece of the second inference data and the order of each second inference token. Determine a first quantity based on the second inference token quantity of each of the second inference data, where the first quantity is the quantity of second inference tokens in the prefix data corresponding to each of the second inference data; For each of the second inference data, based on the order of each of the second inference tokens, sequentially select a first quantity of second inference tokens from each of the second inference tokens and determine them as prefix data.

6. The model adjustment method according to claim 1, wherein The fine-tuning of the student model based on each of the prefix data and each of the target inference data to obtain a target student model includes: Fine-tune the student model using a first loss function based on each of the prefix data corresponding to each of the training data to obtain a first student model; Fine-tune the first student model using a second loss function based on the target inference data corresponding to each of the training data to obtain the target student model.

7. The model adjustment method according to claim 6, wherein The first loss function is determined based on a third probability of each of the problem text data; the third probability is the ratio between each of the prefix data corresponding to each of the problem text data and the corresponding subsequent inference tokens; the subsequent inference tokens are used to indicate the second inference tokens in the second inference data that are after the prefix data.

8. The model adjustment method according to claim 7, wherein The expression of the first loss function is: ； where q represents the question text data, represents the l-th prefix data corresponding to the question text data q; represents the post-inference token of the l-th prefix data corresponding to the question text data q, represents the third probability of the question text data q.

9. The model adjustment method according to claim 6, wherein The second loss function is determined based on a fourth probability of each of the problem text data, where the fourth probability is the ratio between each of the first inference tokens in the target inference data corresponding to each of the problem text data and the corresponding preceding inference tokens, and the preceding inference tokens are used to indicate each of the first inference tokens before the first inference token.

10. The model adjustment method according to claim 9, wherein The expression of the second loss function is: ； where q represents the question text data, represents the i-th first inference token in the target inference data corresponding to the question text data q, represents the prior inference token corresponding to the i-th first inference token, represents the fourth probability of the question text data q.

11. The model adjustment method according to any one of claims 1 to 10, characterized in that After the fine-tuning of the student model based on each of the prefix data and each of the target inference data to obtain a target student model, it includes: Obtain at least one set of target data; each of the target data includes target problem text data, target answer text data, and target inference text data; Input each of the target problem text data and the corresponding target answer text data into the target student model to obtain model inference data output by the target student model corresponding to each of the target problem text data; Determine a first similarity between each of the model inference data and the corresponding target inference text data; When each of the first similarities is greater than or equal to a first threshold, determine that the adjustment of the target student model is completed.

12. The model adjustment method according to claim 11, wherein The determining of the first similarity between each of the model inference data and the corresponding target inference text data includes: Determine at least one first token in each of the model inference data and at least one second token in the corresponding target inference text data; Determine the matching degree between the at least one first token and the at least one second token and determine the matching degree as the first similarity.

13. An electronic device, characterized in that, Includes: A memory for storing a computer program; A processor for implementing the steps of the model adjustment method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program implements the steps of the model adjustment method according to any one of claims 1 to 12 when executed by a processor.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the model adjustment method according to any one of claims 1 to 12 are implemented.

Images