Model adjustment method, device, medium and program product

Abstract

The present application discloses a model adjustment method, device, medium and program product, which relate to the field of data processing technology. The method includes determining target inference data and at least one prefix data corresponding to each group of training data through a teacher model, and fine-tuning the student model based on the prefix data and the target inference data. No annotation is required during the process, so there is no need to consume a large amount of computing resources. This solves the technical problem that related fine-tuning methods rely on manual annotation, and achieves the technical effect of improving the efficiency of model fine-tuning.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a model adjustment method, device, medium and program product. Background Art

[0002] Knowledge distillation refers to a model compression technique that allows users to use a large "teacher" model to train and fine-tune a small "student" model.

[0003] Related fine-tuning methods include supervised fine-tuning (SFT), rejection sampling fine-tuning (RFT), and self-learning thruster (STaR). These methods usually rely on manually labeled training data or large amounts of sampled data, consume large amounts of computing resources, are costly, and result in low model fine-tuning efficiency. Summary of the Invention

[0004] The present application provides a model adjustment method, device, medium and program product to at least solve the problem of low adjustment efficiency of related reasoning models in related technologies.

[0005] This application provides a model adjustment method, including:

[0006] Inputting at least one set of training data into a teacher model to obtain target inference data and at least one prefix data corresponding to each training data; wherein the training data includes question text data and answer text data; the target inference data includes first inference data generated by the teacher model based on a preset inference strategy; and the prefix data includes multiple inference tokens from multiple second inference data randomly generated by the teacher model;

[0007] The student model is fine-tuned based on each prefix data and each target inference data to obtain a target student model; wherein the number of first parameters in the student model is less than the number of second parameters in the teacher model.

[0008] The present application also provides a model adjustment device, comprising:

[0009] A first input module is configured to input at least one set of training data into a teacher model to obtain target inference data and at least one prefix data corresponding to each training data; wherein the training data includes question text data and answer text data; the target inference data includes first inference data generated by the teacher model based on a preset inference strategy; and the prefix data includes multiple inference tokens from multiple second inference data randomly generated by the teacher model;

[0010] A fine-tuning module is used to fine-tune the student model based on each prefix data and each target inference data to obtain a target student model; wherein the number of first parameters in the student model is less than the number of second parameters in the teacher model.

[0011] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned model adjustment methods when executing the computer program.

[0012] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model adjustment methods are implemented.

[0013] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned model adjustment methods when executed by a processor.

[0014] Through this application, the prefix data corresponding to each group of training data can be determined by the teacher model without adding annotations to the training data, and the student model can be fine-tuned based on the prefix data and target inference data. Therefore, the technical problem that the relevant fine-tuning method relies on manual annotation can be solved, and the technical effect of improving the efficiency of model fine-tuning can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A schematic diagram of an application of a model adjustment method provided in an embodiment of the present application;

[0017] Figure 2 One of the flow charts of a model adjustment method provided in an embodiment of the present application;

[0018] Figure 3 The second flowchart of a model adjustment method provided in an embodiment of the present application;

[0019] Figure 4 The third flowchart of a model adjustment method provided in an embodiment of the present application;

[0020] Figure 5 A schematic structural diagram of a model adjustment device provided in an embodiment of the present application;

[0021] Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0024] Chain of Thought (CoT) refers to a strategy used in Natural Language Processing (NLP) to improve model reasoning capabilities, particularly in complex question-answering tasks. Chain of Thought requires the model to generate a series of intermediate reasoning steps before providing a final answer. These reasoning steps detail how the model proceeds from the question to the answer through logical reasoning. This helps the model gradually simplify the problem until it arrives at the answer. Furthermore, by demonstrating the reasoning process, the model's interpretability is enhanced, allowing users to understand how the model arrives at a specific answer. It is widely used in various NLP tasks, such as math problem solving, commonsense reasoning, and program generation. In these tasks, the model is often required to demonstrate its understanding of the problem and provide a detailed reasoning process.

[0025] The reasoning steps in the chain of thought reasoning mainly include: (1) understanding the problem: input the problem into the model, and the model understands the problem; (2) generating intermediate steps: the model generates a series of intermediate reasoning steps, each of which is a logically coherent statement, which together constitute a complete reasoning process; (3) obtaining the answer: when all necessary intermediate reasonings are completed, the model finally gives the answer to the problem.

[0026] Pre-training is a strategy for training deep learning models. Its core is to initially train the model using large-scale datasets, enabling it to learn common feature representations. This process is similar to the basic learning stage humans go through before acquiring new knowledge, accumulating experience through extensive reading and observation.

[0027] Fine-tuning involves further training a pre-trained model on a task-specific dataset to adjust model parameters and better adapt it to the target task. During fine-tuning, most layers of the pre-trained model are typically frozen, with only newly added layers trained or a small number of key layers adjusted. This approach preserves the useful features learned by the pre-trained model while enabling rapid adaptation to the specific requirements of the new task. Choosing the appropriate learning rate and number of training rounds is also crucial for successful fine-tuning.

[0028] A pre-trained language model generally involves designing a language model training task based on a large-scale corpus (including language training materials such as sentences and paragraphs), and then training a large-scale neural network algorithm structure to learn and implement it. The resulting large-scale neural network algorithm structure and parameters are the pre-trained language model. Subsequent tasks can use this model to extract features or fine-tune it to achieve specific task objectives. The idea behind pre-training is to first train a task to obtain a set of model parameters, then use this set of model parameters to initialize the network model parameters. The initialized network model is then used to train other tasks to obtain a model adapted for these tasks. By pre-training on a large-scale corpus, neural language representation models can acquire powerful language representation capabilities and extract rich syntactic and semantic information from text. Pre-trained language models can provide token- and sentence-level features containing rich semantic information for use in downstream tasks. Fine-tuning can also be performed directly on the pre-trained model for downstream tasks, quickly and easily obtaining a dedicated downstream model.

[0029] The neural network algorithm structure used for pre-trained language models can be CNN, RNN, LSTM, etc., or a model built with an attention network, such as Transformer, BERT, GPT, Clip, etc., which is not limited in this application. An attention network refers to a network model that uses the attention mechanism for training. The model assigns different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence, so that the model ultimately obtains a more accurate output.

[0030] Knowledge distillation is a model compression technology that allows users to distill knowledge from a large model to obtain a dense student model that achieves similar accuracy on specific downstream tasks. This technology involves creating or selecting a pre-trained student model with far fewer parameters than a given teacher model, preparing a dataset related to the downstream task (such as the GSM8K dataset), using the teacher model to infer questions in the dataset, outputting soft labels (usually a probability distribution of the output layer), and fine-tuning the student model using the data and soft labels from the dataset, while optimizing both the distillation loss and the standard fine-tuning loss.

[0031] Related fine-tuning methods include supervised fine-tuning (SFT), rejection sampling fine-tuning (RFT), and self-learning thruster (STaR). SFT uses manually labeled reasoning process data to fine-tune large language models (LLMs), improving their reasoning capabilities. However, this method relies on a large amount of high-quality labeled data and is relatively expensive.

[0032] It is understandable that in some specific domains or tasks, it may be difficult to find enough labeled data to support model fine-tuning. In particular, supervised fine-tuning is more difficult when there are no labeled resources.

[0033] Furthermore, in some complex reasoning tasks (such as mathematical reasoning and code generation), annotating a complete reasoning process may take professionals hours or even longer. Furthermore, if the annotated data contains bias or noise, it may cause the model to overfit or perform poorly during fine-tuning.

[0034] Rejection sampling fine-tuning (RFT) generates multiple inference trajectories and selects the correct one for fine-tuning. While it does not require labeled data, it does require significant sampling and computational resources. Self-learning reasoners (STaR) are iterative self-learning methods that generate solutions from models, enabling self-verification and improvement. These methods also require significant computational resources and verification mechanisms. This limits their feasibility in practical applications, particularly for resource-constrained users or institutions.

[0035] In other words, the above methods usually have the problem of relying on manually labeled training data or a large amount of sampled data, which consumes a lot of computing resources and is costly, making the efficiency of model fine-tuning low.

[0036] In order to solve the above problems, the embodiments of the present application provide a model adjustment method, device, medium and program product. Through the teacher model, the target inference data and at least one prefix data corresponding to each group of training data are determined, and the student model is fine-tuned based on each prefix data and each target inference data. No annotation is required in the process, so there is no need to consume a large amount of computing resources, which reduces the fine-tuning cost and improves the efficiency of model fine-tuning.

[0037] The model adjustment method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, personal computers, and other devices, without specific limitation here.

[0038] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0039] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the model adjustment method depends, the specific application environment architecture or specific hardware architecture is described here. Figure 1 A schematic diagram of an application of a model adjustment method provided in an embodiment of the present application.

[0040] See also Figure 1 The electronic device includes a teacher model and a student model. The teacher model is a pre-trained first language model (also called a question-answering pre-trained language model) used to infer answers based on question text data. The student model is a second language model (also called a question-answering pre-trained language model) used to infer answers based on question text data. The number of first parameters in the student model is smaller than the number of second parameters in the teacher model. After a user inputs at least one set of training data (including question text data and answer text data) into the electronic device, the teacher model processes the training data to obtain corresponding prefix data and target inference data. The question text data, corresponding answer text data, prefix data, and target inference data are then input into the student model to fine-tune the student model and obtain a target student model. In this way, the target student model can generate corresponding model inference data based on the input question data and answer data. Model inference data refers to the information output by the model about the reasoning process used to obtain answer data based on the question data.

[0041] Figure 2 This is one of the flow charts of the model adjustment method provided in the embodiment of the present application, such as Figure 2 As shown, the embodiment of the present application provides a model adjustment method, which is described in detail as follows:

[0042] S201: Input at least one set of training data into the teacher model to obtain target inference data and at least one prefix data corresponding to each training data; wherein the training data includes question text data and answer text data; the target inference data includes first inference data generated by the teacher model based on a preset inference strategy; and the prefix data includes multiple inference words in multiple second inference data randomly generated by the teacher model.

[0043] Optionally, each set of training data includes, but is not limited to, question text data and answer text data. The question text data is text data used to describe question information, such as math question information or code generation question information; and the answer text data is text data used to describe answer information corresponding to the question information, such as math answer information corresponding to math question information or code answer information corresponding to code generation question information.

[0044] Optionally, at least one set of training data is input into the teacher model, and the teacher model is used to generate a text sequence containing multiple steps of reasoning explanations based on each question text data and the corresponding answer text data in the at least one set of training data. Each sequence includes question text data, reasoning steps (also referred to as reasoning data), and answer text data. Based on the above reasoning steps, target reasoning data and at least one prefix data corresponding to each training data are determined. The target reasoning data refers to the first reasoning data composed of multiple first reasoning words corresponding to each set of training data, generated by the teacher model based on a preset reasoning strategy. The prefix data refers to the text data composed of the second reasoning words located in the prefix part of each second reasoning data corresponding to each set of training data, randomly generated by the teacher model.

[0045] For example, the teacher model may be the large language model Llama3 model or the mathematical problem model Qwen-Math, and the student model may be the large language model Llam3-70b or the large language model Llam3-8b.

[0046] S202: Fine-tune the student model based on each prefix data and each target inference data to obtain a target student model; wherein the number of first parameters in the student model is less than the number of second parameters in the teacher model.

[0047] Optionally, each question text data and the corresponding answer text data, prefix data and target reasoning data are input into the student model, and the student model is preliminarily fine-tuned based on each question text data and the corresponding answer text data and prefix data, and then the student model is secondarily fine-tuned based on each question text data and the corresponding answer text data and the target reasoning data, that is, the intermediate reasoning steps corresponding to the question text data and the answer text data generated by the teacher model are used in the fine-tuning of the student model, so that the student model learns the core reasoning ability of the teacher model to obtain the target student model, so that the target student model can output the corresponding model reasoning data based on the input question data and answer data.

[0048] Alternatively, knowledge distillation involves training a small "student" model to mimic the behavior of a large "teacher" model, which can reduce the model's computational and storage requirements while maintaining performance, making it easier to integrate into practical applications. The model adjustment method provided in the embodiments of this application can adopt the COT distillation method.

[0049] For example, when the training data belongs to a dataset with inference steps, such as the GSM8K dataset, the machine learning method few-shot can be used to generate inference steps to increase the accuracy of the inference steps generated by the model.

[0050] The embodiment of the present application uses the prefix text data of the reasoning content output by the teacher model to generate a prefix data set that is easy to obtain and does not require additional labeling or calibration, and then generates a small amount of complete target reasoning data through the chain of thought reasoning CoT. The two types of data are combined and the student model is double-tuned to obtain a target student model with a smaller number of parameters than the teacher model and can achieve similar performance to the teacher model in logical reasoning tasks. It is suitable for downstream tasks that require theoretical inference or logical calculation. Moreover, since the target student model has a small number of parameters, it is used in various practical application scenarios.

[0051] Figure 3 The second flow chart of the model adjustment method provided in the embodiment of the present application is as follows: Figure 3 As shown, step S201 includes:

[0052] S2011: Inputting at least one set of training data into a teacher model to obtain first inference data corresponding to each training data, generated by the teacher model based on a preset inference strategy;

[0053] S2012: For each piece of training data, determine the corresponding first inference data as target inference data;

[0054] S2013: Inputting at least one set of training data into the teacher model, and obtaining at least one second inference data output by the teacher model corresponding to each set of training data;

[0055] S2014: For each training data, generate corresponding prefix data based on each second inference data.

[0056] Optionally, each group of training data is input into the teacher model to obtain the first inference word corresponding to each training data, which is generated by the teacher model based on a preset inference strategy. Each training data and the corresponding first inference word are used as new training data, and the above-mentioned step of the teacher model inferring the new first inference word based on the input training data is cyclically executed to obtain the first inference data corresponding to each training data and determine it as the target inference data.

[0057] Optionally, at least one set of training is input into the teacher model at least once, and at least one second inference data corresponding to each set of training data is output by the teacher model. For each set of training data, prefix data corresponding to each second inference data is generated in the corresponding at least one second inference data.

[0058] Optionally, each second inference data includes at least one second inference word element (the second inference word element may also be referred to as a character or token).

[0059] It is understandable that the process of generating multiple second inference data for the same set of question text data and answer text data by the teacher model has a certain degree of randomness. In other words, the teacher model has a certain degree of similarity in at least one second inference data corresponding to the same set of question text data and answer text data. The prefix data generated based on the above second inference data can be used as training data to fine-tune the student model, so that the student model learns the process of the teacher model performing multiple inferences on the same set of question text data and answer text data. In other words, the student model can learn the knowledge of prefix inference and retain the original inference structure, thereby ensuring overall performance. In addition, the prefix data generated by the teacher model does not need to be verified for accuracy or additional labeling, which can reduce the computing power cost required for manual labeling.

[0060] Optionally, the training data includes at least one word-gram, which may also be referred to as a training word-gram, a character, or a token. The teacher model includes a plurality of candidate word-grams, which may indicate word-grams (or characters) corresponding to the language type of the training data.

[0061] Figure 4 The third flow chart of the model adjustment method provided in the embodiment of the present application is as follows: Figure 4 As shown, step S2011 includes:

[0062] S20111: Input each training data into the teacher model and determine the target logarithmic difference corresponding to each training data; the target logarithmic difference is used to indicate the probability logarithmic difference between the training data and each candidate word;

[0063] S20112: For each training data, determine the candidate word corresponding to the largest logarithmic difference as the first inference word;

[0064] S20113: For each training data, the training data is updated based on the first inference word, and the process returns to input each training data into the teacher model, determines the target logarithmic difference corresponding to each training data, and subsequent steps until a stop character is detected, and obtains the first inference data corresponding to each training data.

[0065] Optionally, each training data is input into the teacher model to determine a target logarithmic difference corresponding to each training data, wherein the target logarithmic difference is used to indicate the logarithmic difference in probability between the training data and each candidate word.

[0066] Taking Chinese as the language type of training data as an example, assuming that the number of all Chinese word units (or characters) is 30,000, the target logarithmic difference can be used to indicate the probability logarithmic difference between the training data and each candidate word unit in the 30,000 candidate word units.

[0067] Optionally, for each training data, comparison is performed according to the numerical value of the target logarithmic difference, and the candidate word corresponding to the target logarithmic difference with the largest numerical value is determined as the first inference word corresponding to the training data.

[0068] Optionally, for each training data, the training data is updated based on each training data and the corresponding first inference word (i.e., the corresponding first inference word is added on the basis of the training data) to obtain new training data, and the new training data is input into the teacher model to obtain a new first inference word (i.e., return to execute the steps of inputting each training data into the teacher model, determining the target logarithmic difference corresponding to each training data and subsequent steps), until a stop character is detected, the first inference data corresponding to each training data is obtained based on each first inference word corresponding to each training data, and the first inference data is determined as the target inference data.

[0069] Optionally, a stop character is used to indicate a word in the candidate word that is used to indicate end information.

[0070] Exemplarily, the stop character may be a period. In another exemplary embodiment, the stop character may be a space character.

[0071] It can be understood that the target inference data includes the first inference word corresponding to the training data output by the teacher model each time.

[0072] For example, the word units in training data A include a1, a2, and a3. Among them, the candidate word unit corresponding to the target logarithmic difference of training data A with the largest value is a4, and a4 is determined to be the first inference word unit corresponding to training data A. a1, a2, a3, and a4 are updated to new training data A and input into the teacher model, the new target logarithmic difference of training data A is determined, and the candidate word unit corresponding to the new target logarithmic difference with the largest value is a5, which is determined as the new first inference word unit. a1, a2, a3, a4, and a5 are updated to another new training data A and input into the teacher model to obtain another new first inference word unit. The loop is executed until a stop character is detected, and each first inference word unit (such as a4, a5, etc.) is determined as the target inference data (i.e., including a4, a5...) corresponding to training data A (i.e., a1, a2, a3).

[0073] Optionally, each training data is input into a teacher model to determine a target logarithmic difference corresponding to each training data, including:

[0074] Input each training data into the teacher model, and determine a first probability based on the question text data and the answer text data of each training data;

[0075] Determining a second probability based on the question text data of each training data;

[0076] For each training data, the logarithmic difference between the corresponding first probability and the second probability is determined as the target logarithmic difference.

[0077] Optionally, each training data is input into the teacher model, and the first probability of each training data and each candidate word is determined based on the question text data and answer text data of each training data (including each word in the question text data and answer text data set and the first inference word corresponding to the training data).

[0078] Optionally, based on the question text data in each training data (including each word in the question text data, and the first inference word corresponding to the training data), the second probability of each training data and each candidate word is determined, and the logarithmic difference between the first probability and the second probability of each training data is calculated and determined as the target logarithmic difference corresponding to each training data.

[0079] As an example, not for illustration, when training data is updated based on the first inference tokens corresponding to each training data item to obtain new training data, and the new training data is input into the teacher model, the target logarithmic difference is used to indicate the logarithmic difference in probability between each first inference token corresponding to the training data item and each candidate token. That is, when the input data is new training data, the first probability is the probability determined based on the question text data, answer text data, and each first inference token corresponding to each training data item, and the second probability is the probability determined based on the question text data and each first inference token corresponding to each training data item.

[0080] For example, the target logarithmic difference can be calculated by the following formula:

[0081] ;

[0082] in, represents the first probability, represents the second probability, q represents the question text data, represents the answer text data corresponding to the question text data q, Indicates that the answer text data is empty. When the data input to the teacher model is the original training data, represents the inference word corresponding to the original training data (the inference word can also be called a candidate word, which is any one of the candidate words), Represents the word units contained in the question text data and answer text data in the original training data, Represents the word units included in the question text data in the original training data.

[0083] And, when the training data input to the teacher model is new training data, represents the i-th inference word corresponding to the training data (the inference word can also be called a candidate word, which is any one of the candidate words), 、 In this way, we can calculate the first inference word before the i-th inference word. When , the corresponding new inference word is generated (it can also be called determined in each candidate word) The first probability. And when the input question text data is q and the input empty answer text data is When , the corresponding new inference word is generated (it can also be called determined in each candidate word) The second probability of , and so on until the stop character is detected, thereby determining the training data (including question text data q, answer text data ) and the target logarithmic difference between each candidate word.

[0084] It can be understood that by calculating the logarithmic difference between the first probability and the second probability , we can get: the inference word corresponding to the training data input to the teacher model , the amount of credibility increase when inferring the correct answer compared to when inferring an empty character. The greater the credibility increase, the more likely the inference word is to be correct. The stronger the correlation with the correct answer, that is, the more the inference data generated by the model supports the correct answer. The reason for setting the comparison answer to an empty character is to penalize those who do not consider the correct answer. The above tokens may not be specific to the correct answer.

[0085] Therefore, among the target logarithmic differences between the training data and each candidate word, the candidate word corresponding to the largest target logarithmic difference can be determined as the first inference word, as shown in the following formula:

[0086] ;

[0087] in, represents the i-th inference word corresponding to the training data (the inference word can also be called a candidate word, which is any one of the candidate words), represents the i-th first inference token output by the teacher model.

[0088] Still taking the above training data A as an example, when the training data input to the teacher model is the original training data A (including word units a1, a2, and a3 of the question text data and the answer text data), represents a1, a2, a3, Indicates the inference word that can be inferred based on a1, a2, and a3 in each candidate word. When a4 is used, the data input to the teacher model is the new training data A (including the word units a1, a2, a3 of the question text data and the answer text data and the corresponding first inference word unit a4). represents a4, Indicates the inference word that can be inferred based on a4 among the candidate word units. And, the first inference word corresponding to a1, a2, a3, and a4 is determined based on the logarithmic difference. When a5 is used, the data input to the teacher model is the new training data A (including the word units a1, a2, a3 of the question text data and the answer text data and the corresponding first inference word units a4, a5). represents a4 and a5, Indicates the inference word that can be inferred based on a4 and a5 in each candidate word. Similarly, multiple first inference words output by the teacher model are obtained, forming a complete inference process including a small number of inference steps.

[0089] Optionally, the second inference data includes a plurality of second inference tokens, and for each piece of training data, corresponding prefix data is generated based on each piece of second inference data, including:

[0090] For each training data, determining the number of second inference tokens of each second inference data and the order of each second inference token;

[0091] Determine a first number based on the number of second inference tokens in each second inference data, where the first number is the number of second inference tokens in the prefix data corresponding to each second inference data;

[0092] For each second inference data, based on the order of each second inference word-gram, a first number of second inference word-grams are sequentially selected from each second inference word-gram and determined as prefix data.

[0093] Optionally, for each set of training data, the number of second inference tokens in each second inference data corresponding to the training data, as well as the order of each second inference token in the corresponding second inference data, are determined. Based on the number of second inference tokens in each second inference data, the number of second inference tokens in the corresponding prefix data (i.e., a first number) is determined. For each second inference data, based on the order of each second inference token in the corresponding second inference data, a first number of second inference tokens can be sequentially selected from each second inference token, and the first number of second inference tokens can be determined as the prefix data corresponding to the second inference data.

[0094] Assume that the teacher model generates m second reasoning data for each question q m can be set based on the size of the dataset, and there's no need to verify the correctness of the final result. For each piece of second inference data c, the first k second inference tokens are used as prefix data. For example, if the first number k = 8, the first eight second inference tokens of the second inference data are used as prefix data. The first number k can also be called a hyperparameter, and an appropriate value of k can be selected through experimentation.

[0095] It is understandable that if the teacher model has a small number of parameters (such as Llama), k can be set to a smaller value. If the teacher model has strong professional reasoning capabilities and a large number of parameters (such as the math problem model Qwen-Math), k can be set to a larger value. The number of prefix datasets should be larger than the training dataset.

[0096] It is understandable that by truncating the second inference data, only the first second inference token in the second inference data is obtained as prefix data, and there is no need to pay attention to whether the second inference data and the inference result are correct, so that the generated prefix data can be random. Different inference steps and thus different prefix data can be generated by adjusting the second inference parameters of the teacher model. For example, such as increasing the model parameter Temperature or increasing the value of the model parameter top-k.

[0097] It's understandable that the initial inference tokens (for example, the first 10-15 second inference tokens in the second inference data) in the inference data corresponding to different inference structures are generally consistent. This means that even if the subsequent second inference tokens in the second inference data may differ, the initial second inference tokens are generally similar, and logical reasoning errors generally occur in the inference tokens in the latter half of the second inference data. Therefore, fine-tuning the student model by only intercepting the initial second inference tokens in the second inference data as prefix data can guide the student model's reasoning process, enabling it to learn the key feature information of the problem and make correct inferences and judgments in the initial stages, thereby improving the overall reasoning accuracy of the learner.

[0098] Among them, the first number can be specifically set according to actual conditions. Exemplarily, the first number is the number of second inference word units in a preset ratio, and the preset ratio can be 40%, 45% or 50%. For example, the number of second inference word units of a certain second inference data is 30, and the preset ratio is 40%. The corresponding 12 second inference word units located before the second inference data are the prefix data of the second inference data. For another example, the output word units of a certain second inference data are 32, and the preset ratio is 45%. The corresponding 15 second inference word units located before the second inference data are the prefix data of the second inference data.

[0099] Optionally, fine-tuning the student model based on each prefix data and each target inference data to obtain a target student model includes:

[0100] Based on the prefix data corresponding to the training data, the student model is fine-tuned using the first loss function to obtain a first student model;

[0101] Based on the target inference data corresponding to each training data, the second loss function is used to fine-tune the first student model to obtain the target student model.

[0102] Optionally, the question text data in each training data, the inference data corresponding to each training data, and the prefix data are input into a student model, and a first loss function is used to fine-tune the student model so that the student model learns the inference information features of each prefix data in the process of inferring answer data from the question text data in the teacher model, thereby obtaining a first student model. The question text data in each training data, the answer text data, the inference data corresponding to each training data, and the target inference data are input into a first student model, and a second loss function is used to fine-tune the first student model so that the student model learns the inference information features of the target inference data in the process of inferring answer data from the question text data in the teacher model, thereby obtaining a target student model.

[0103] The embodiment of the present application determines the prefix data that does not need to be labeled and a small amount of complete target reasoning data from the reasoning data output by the teacher model to construct a fine-tuning data set, and performs two-stage fine-tuning on the student model so that the student model can not only learn the knowledge about prefix reasoning in the teacher model, but also retain the original reasoning structure, thereby ensuring the overall performance. In this way, the student model has the step-by-step reasoning ability of the teacher model, achieves performance similar to that of the teacher model in specific tasks, and is applicable to a variety of task scenarios.

[0104] Optionally, in order to avoid the student model losing its reasoning structure due to fine-tuning the student model using only prefix data, and to enable the student model to have complete logical reasoning capabilities similar to those of the teacher model, a second fine-tuning is required on a dataset that includes complete reasoning steps (such as text data for each question, text data for each answer, corresponding reasoning data, and target reasoning data, etc.).

[0105] Optionally, the first loss function is determined based on the third probability of each question text data; the third probability is the ratio between each prefix data corresponding to each question text data and the corresponding subsequent reasoning word; the subsequent reasoning word is used to indicate the second reasoning word located after the prefix data in the second reasoning data.

[0106] Optionally, the expression of the first loss function is:

[0107] ;

[0108] Among them, q represents the question text data, Indicates the lth prefix data corresponding to the question text data q; Indicates the subsequent inference word of the lth prefix data corresponding to the question text data q, represents the third probability of the question text data q.

[0109] Optionally, the first loss function is determined based on a third probability of each question text data; the third probability is a ratio between each prefix data corresponding to each question text data and the corresponding subsequent inference word.

[0110] The subsequent inference word is used to indicate an inference word that is located after the prefix data in the inference data.

[0111] For example, if the prefix data is the first 10 second inference tokens in the second inference data corresponding to the question text data, then the subsequent inference token is the 11th second inference token in the second inference data.

[0112] It can be understood that the smaller the third probability value is, the more the first loss function converges. When the first loss function converges, the first student model is obtained.

[0113] Optionally, the second loss function is determined based on the fourth probability of each question text data, and the fourth probability is the ratio between each first inference word in the target inference data corresponding to each question text data and the corresponding previous inference word, and the previous inference word is used to indicate each first inference word located before the first inference word.

[0114] Optionally, the expression of the second loss function is:

[0115] ;

[0116] Among them, q represents the question text data, represents the i-th first inference word in the target inference data corresponding to the question text data q, represents the previous inference word corresponding to the i-th first inference word, represents the fourth probability of the question text data q.

[0117] Optionally, the previous reasoning word can be used to indicate each first reasoning word before the current first reasoning word in the target reasoning data, and the corresponding It can also be referred to as each first inference word-unit located before the above-mentioned i-th first inference word-unit in the target inference data.

[0118] Still taking the above training data A as an example, the training data A includes question text data q. When the training data A input to the first student model includes a1, a2, a3, and the target reasoning data includes a4 and a5, then represents the first inference word a4 in the target inference data corresponding to the question text data q, represents the previous inference word corresponding to the first inference word a4 (which is empty), and Indicates the second first inference word a5 in the target inference data corresponding to the question text data q, Indicates the previous inference word a4 corresponding to the second first inference word a5 mentioned above, and so on.

[0119] It can be understood that the smaller the fourth probability value is, the more the second loss function converges. When the second loss function converges, the target student model is obtained.

[0120] Optionally, after fine-tuning the student model based on each prefix data and each target inference data to obtain a target student model, the following steps are included:

[0121] Acquire at least one set of target data; each target data includes target question text data, target answer text data, and target reasoning text data;

[0122] Input each target question text data and the corresponding target answer text data into the target student model, and obtain the model inference data corresponding to each target question text data output by the target student model;

[0123] Determining a first similarity between each model inference data and the corresponding target inference text data;

[0124] When all first similarities are greater than or equal to the first threshold, it is determined that the adjustment of the target student model is completed.

[0125] Optionally, at least one set of target data is obtained, each set of target data includes a target question text data, a target answer text data, and a target reasoning text data. Each target question text data and the corresponding target answer text data are input into the target student model, so that the target student model performs reasoning on the target question text data, obtains reasoning information of the target answer text data, and thus obtains model reasoning data corresponding to each target question text data output by the target student model. A first similarity between each model reasoning data and the target reasoning text data corresponding to each model reasoning data is calculated. When the first similarity corresponding to each target data is greater than or equal to a first threshold, it is determined that the adjustment of the target student model is completed, so that the adjusted target student model can be deployed in an actual application scenario.

[0126] The first threshold value may be determined according to actual conditions, for example, the first threshold value may be 85% or 90%.

[0127] Optionally, the first similarity may be a character similarity between the model inference data and the corresponding target inference text data.

[0128] Optionally, determining a first similarity between each model inference data and corresponding target inference text data includes:

[0129] Determining at least one first word in each model inference data and at least one second word in the corresponding target inference text data;

[0130] A degree of matching between at least one first word-gram and at least one second word-gram is determined, and the degree of matching is determined as a first similarity.

[0131] Optionally, at least one first word in each model inference data and at least one second word in the target inference text data are determined, and each first word is matched with each second word to determine the similarity between at least one first word in each model inference data and at least one second word in the target inference text data as the first similarity of the corresponding target data.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0133] Figure 5 This is a schematic diagram of the structure of the model adjustment device provided in the embodiment of the present application. Figure 5 As shown, an embodiment of the present application further provides a model adjustment device, comprising:

[0134] A first input module 501 is configured to input at least one set of training data into a teacher model to obtain target inference data and at least one prefix data corresponding to each training data; wherein the training data includes question text data and answer text data; the target inference data includes first inference data generated by the teacher model based on a preset inference strategy; and the prefix data includes multiple inference tokens from multiple second inference data randomly generated by the teacher model;

[0135] The fine-tuning module 502 is used to fine-tune the student model based on each prefix data and each target inference data to obtain a target student model; wherein the number of first parameters in the student model is less than the number of second parameters in the teacher model.

[0136] Optionally, the first input module includes:

[0137] A first input unit is configured to input at least one set of training data into a teacher model to obtain first inference data corresponding to each set of training data and generated by the teacher model based on a preset inference strategy;

[0138] a determining unit, configured to determine, for each piece of training data, the corresponding first inference data as target inference data;

[0139] A second input unit is configured to input at least one set of training data into the teacher model, and obtain at least one second inference data output by the teacher model corresponding to each set of training data;

[0140] The generating unit is configured to generate corresponding prefix data based on each second inference data for each training data.

[0141] Optionally, the teacher model includes a plurality of candidate word units; the first input unit includes:

[0142] The first determination subunit is used to input each training data into the teacher model and determine the target logarithmic difference corresponding to each training data; the target logarithmic difference is used to indicate the probability logarithmic difference between the training data and each candidate word;

[0143] A second determining subunit is configured to determine, for each training data, the candidate word corresponding to the largest logarithmic difference as the first inference word;

[0144] The third determination subunit is used to update the training data based on the first inference word for each training data, and return to execute the steps of inputting each training data into the teacher model, determining the target logarithmic difference corresponding to each training data, and subsequent steps until a stop character is detected to obtain the first inference data corresponding to each training data.

[0145] Optionally, the first determining subunit is specifically configured to:

[0146] Input each training data into the teacher model, and determine a first probability based on the question text data and the answer text data of each training data;

[0147] Determining a second probability based on the question text data of each training data;

[0148] For each training data, the logarithmic difference between the corresponding first probability and the second probability is determined as the target logarithmic difference.

[0149] Optionally, the second inference data includes a plurality of second inference tokens, and the generation unit is specifically configured to:

[0150] For each training data, determining the number of second inference tokens of each second inference data and the order of each second inference token;

[0151] Determining a first number based on the number of second inference tokens in each second inference data, where the first number is the number of second inference tokens in the prefix data corresponding to each second inference data;

[0152] For each second inference data, based on the order of each second inference word-gram, a first number of second inference word-grams are sequentially selected from each second inference word-gram and determined as prefix data.

[0153] Optionally, a fine-tuning module includes:

[0154] A first fine-tuning unit is configured to fine-tune the student model using a first loss function based on each prefix data corresponding to each training data to obtain a first student model;

[0155] The second fine-tuning unit is used to fine-tune the first student model using a second loss function based on the target inference data corresponding to each training data to obtain a target student model.

[0156] Optionally, the first loss function is determined based on the third probability of each question text data; the third probability is the ratio between each prefix data corresponding to each question text data and the corresponding subsequent reasoning word; the subsequent reasoning word is used to indicate the second reasoning word located after the prefix data in the second reasoning data.

[0157] Optionally, the expression of the first loss function is:

[0158] ;

[0159] Among them, q represents the question text data, Indicates the lth prefix data corresponding to the question text data q; Indicates the subsequent inference word of the lth prefix data corresponding to the question text data q, represents the third probability of the question text data q.

[0160] Optionally, the second loss function is determined based on the fourth probability of each question text data, and the fourth probability is the ratio between each first inference word in the target inference data corresponding to each question text data and the corresponding previous inference word, and the previous inference word is used to indicate each first inference word located before the first inference word.

[0161] Optionally, the expression of the second loss function is:

[0162] ;

[0163] Among them, q represents the question text data, represents the i-th first inference word in the target inference data corresponding to the question text data q, represents the previous inference word corresponding to the i-th first inference word, represents the fourth probability of the question text data q.

[0164] Optionally, the model adjustment device includes:

[0165] An acquisition module is used to acquire at least one set of target data; each target data includes target question text data, target answer text data and target reasoning text data;

[0166] The second input module is used to input each target question text data and the corresponding target answer text data into the target student model, and obtain the model inference data corresponding to each target question text data output by the target student model;

[0167] A first determining module, configured to determine a first similarity between each model inference data and the corresponding target inference text data;

[0168] The second determination module is used to determine that the adjustment of the target student model is completed when all first similarities are greater than or equal to a first threshold.

[0169] Optionally, the second determining module is specifically configured to:

[0170] Determining at least one first word in each model inference data and at least one second word in the corresponding target inference text data;

[0171] A degree of matching between at least one first word-gram and at least one second word-gram is determined, and the degree of matching is determined as a first similarity.

[0172] The description of the features in the embodiment corresponding to the model adjustment device can be found in the relevant description of the embodiment corresponding to the model adjustment method, and will not be repeated here.

[0173] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device provided by this embodiment includes: at least one processor 601 and a memory 602. Optionally, the electronic device further includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus.

[0174] During the specific implementation process, at least one processor 601 executes the computer-executable instructions stored in the memory 602, so that the at least one processor 601 executes the above-mentioned model adjustment method embodiment.

[0175] The specific implementation process of the processor 601 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0176] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0177] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage.

[0178] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0179] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned model adjustment method embodiments when run.

[0180] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0181] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned model adjustment method embodiments are implemented.

[0182] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned model adjustment method embodiments are implemented.

[0183] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0184] The above is a detailed introduction to the model adjustment method, device, medium and program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A model adjustment method, characterized in that: include: Inputting at least one set of training data into a teacher model to obtain target inference data and at least one prefix data corresponding to each of the training data; wherein the training data includes question text data and answer text data; the target inference data includes first inference data generated by the teacher model based on a preset inference strategy; and the prefix data includes multiple initial inference tokens intercepted from multiple second inference data randomly generated by the teacher model; Based on the prefix data corresponding to the training data, fine-tuning the student model using a first loss function to obtain a first student model; Based on the target inference data corresponding to each of the training data, the first student model is fine-tuned using a second loss function to obtain a target student model; wherein the number of first parameters in the student model is less than the number of second parameters in the teacher model.

2. The model adjustment method according to claim 1, characterized in that: The step of inputting at least one set of training data into a teacher model to obtain target inference data and at least one prefix data corresponding to each set of training data includes: Inputting the at least one set of training data into the teacher model to obtain first inference data corresponding to each of the training data, generated by the teacher model based on the preset inference strategy; For each of the training data, determining the corresponding first inference data as the target inference data; Inputting the at least one set of training data into the teacher model to obtain at least one second inference data output by the teacher model corresponding to each of the training data; For each of the training data, corresponding prefix data is generated based on each of the second inference data.

3. The model adjustment method according to claim 2, characterized in that: The teacher model includes a plurality of candidate word-grams; inputting the at least one set of training data into the teacher model to obtain first inference data corresponding to each of the training data and generated by the teacher model based on the preset inference strategy, including: Inputting each of the training data into the teacher model, and determining a target logarithmic difference corresponding to each of the training data; the target logarithmic difference is used to indicate the probability logarithmic difference between the training data and each candidate word; For each of the training data, determining the candidate word corresponding to the logarithmic difference with the largest value as the first inference word; For each of the training data, the training data is updated based on the first inference word, and the steps of inputting each of the training data into the teacher model, determining the target logarithmic difference corresponding to each of the training data, and subsequent steps are returned until a stop character is detected to obtain the first inference data corresponding to each of the training data.

4. The model adjustment method according to claim 3, characterized in that: Inputting each of the training data into the teacher model and determining the target logarithmic difference corresponding to each of the training data includes: Inputting each of the training data into the teacher model, and determining a first probability based on the question text data and the answer text data of each of the training data; Determining a second probability based on the question text data of each of the training data; For each of the training data, the logarithmic difference between the corresponding first probability and the second probability is determined as the target logarithmic difference.

5. The model adjustment method according to claim 2, characterized in that: The second inference data includes a plurality of second inference tokens, and the generating corresponding prefix data based on each second inference data for each training data includes: For each of the training data, determining the number of second inference tokens of each of the second inference data and the order of each of the second inference tokens; Determining a first number based on the number of second inference tokens in each of the second inference data, where the first number is the number of second inference tokens in the prefix data corresponding to each of the second inference data; For each of the second inference data, based on the order of each of the second inference word-grams, a first number of second inference word-grams are sequentially selected from each of the second inference word-grams and determined as prefix data.

6. The model adjustment method according to claim 1, characterized in that: The first loss function is determined based on the third probability of each of the question text data; the third probability is the ratio between each prefix data corresponding to each of the question text data and the corresponding subsequent inference word; the subsequent inference word is used to indicate the second inference word located after the prefix data in the second inference data.

7. The model adjustment method according to claim 6, characterized in that: The expression of the first loss function is: ； Among them, q represents the question text data, Indicates the lth prefix data corresponding to the question text data q; Indicates the subsequent inference word of the j-th prefix data corresponding to the question text data q, represents the third probability of the question text data q.

8. The model adjustment method according to claim 1, characterized in that: The second loss function is determined based on the fourth probability of each of the question text data, and the fourth probability is the ratio between each first inference word in the target inference data corresponding to each of the question text data and the corresponding previous inference word, and the previous inference word is used to indicate each first inference word located before the first inference word.

9. The model adjustment method according to claim 8, characterized in that: The expression of the second loss function is: ； Among them, q represents the question text data, represents the i-th first inference word in the target inference data corresponding to the question text data q, represents the previous inference word corresponding to the i-th first inference word, represents the fourth probability of the question text data q.

10. The model adjustment method according to any one of claims 1 to 9, characterized in that: After fine-tuning the student model based on each of the prefix data and each of the target inference data to obtain a target student model, the method includes: Acquire at least one set of target data; each of the target data includes target question text data, target answer text data, and target reasoning text data; Inputting each target question text data and corresponding target answer text data into the target student model, and obtaining model inference data corresponding to each target question text data output by the target student model; Determining a first similarity between each of the model inference data and the corresponding target inference text data; When each of the first similarities is greater than or equal to a first threshold, it is determined that the adjustment of the target student model is completed.

11. The model adjustment method according to claim 10, characterized in that: Determining the first similarity between each of the model inference data and the corresponding target inference text data includes: Determining at least one first word in each of the model inference data and at least one second word in the corresponding target inference text data; A degree of matching between the at least one first word-gram and the at least one second word-gram is determined, and the degree of matching is determined as a first similarity.

12. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the model adjustment method according to any one of claims 1 to 11 when executing the computer program.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the model adjustment method according to any one of claims 1 to 11 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the model adjustment method according to any one of claims 1 to 11 are implemented.

Images