Large model fine tuning method and device, equipment, storage medium and program product

Abstract

The invention discloses a large model fine tuning method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: a target training data set for performing fine tuning on a to-be-fine-tuned large model is generated based on the to-be-fine-tuned large model, and the target training data set comprises two types of question and answer pairs, wherein the answer in the second type of question and answer pair is distillation response data obtained by rewriting original response data corresponding to the instruction data in the second type of question and answer pair by the to-be-finely-tuned large model; the distillation response data and the corresponding original response data have the same reasoning logic and style, that is to say, the answers in the second class of question and answer pairs retain reasoning path characteristics of the large model to be finely tuned, so that the retention capability of the large model on the original reasoning logic in target task learning is enhanced; and the adaptability of the fine-tuned large model on the target task is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large model fine-tuning method, apparatus, device, storage medium, and program product. Background Art

[0002] Large model fine-tuning involves further training a pre-trained large model using data from a specific task, aiming to improve its performance on that task. However, in practice, while fine-tuning large models improves their performance on specific tasks, it often comes with the problem of catastrophic forgetting (i.e., a decrease in their original capabilities).

[0003] To prevent catastrophic forgetting in large models, current large model fine-tuning technology uses large models with good performance to generate question-answer pairs for specific tasks as training data for fine-tuning, and then fine-tunes the large model to be fine-tuned. This approach makes the large model less adaptable to specific tasks. Summary of the Invention

[0004] In view of the above problems, this application provides a large model fine-tuning method, device, equipment, storage medium and program product to improve the adaptability of the fine-tuned large model to specific tasks. The specific solution is as follows:

[0005] The first aspect of the present application provides a large model fine-tuning method, comprising:

[0006] Obtain a target training dataset for a target task, the target training dataset including a first type of question-answer pairs and a second type of question-answer pairs, wherein questions in different question-answer pairs are different instruction data corresponding to the target task; answers in the first type of question-answer pairs are original response data generated by the to-be-fine-tuned large model based on the instruction data in the first type of question-answer pairs, and answers in the second type of question-answer pairs are distilled response data obtained by rewriting the original response data corresponding to the instruction data in the second type of question-answer pairs by the to-be-fine-tuned large model; the distilled response data has the same reasoning logic and style as the corresponding original response data;

[0007] Fine-tune the large model to be fine-tuned based on the target training data set.

[0008] In one possible implementation, the target training dataset is generated in the following manner:

[0009] Using the large model to be fine-tuned, each instruction data in the instruction data set of the target task is processed separately to generate original response data corresponding to each instruction data;

[0010] Rewriting each original response data using the large model to be fine-tuned to obtain distilled response data corresponding to each original response data;

[0011] Performing quality assessment on each distillation response data, and retaining the distillation response data whose quality assessment results meet the quality conditions;

[0012] The target training dataset is constructed based on each instruction data and its corresponding original response data, as well as the retained distilled response data.

[0013] In a possible implementation, rewriting any original response data using the large model to be fine-tuned includes:

[0014] Parsing any of the original response data based on the instruction data corresponding to the any of the original response data by the large model to be fine-tuned to obtain key reasoning steps for the any of the original response data;

[0015] The distilled response data corresponding to any original response data is generated by the large model to be fine-tuned based on the instruction data corresponding to any original response data and the key reasoning steps.

[0016] In one possible implementation, quality assessment is performed on any distillation response data, including:

[0017] Obtaining the perplexity of any distilled response data and the degree of difference between any distilled response data and its corresponding original response data;

[0018] If the perplexity is less than or equal to the perplexity threshold corresponding to the target task type, and the difference degree is less than the target difference degree, it is determined that any of the distilled response data meets the quality condition; otherwise, it is determined that any of the distilled responses does not meet the quality condition.

[0019] In one possible implementation, quality assessment is performed on any distillation response data, including:

[0020] determining a first index for any distilled response data based on the perplexity of any distilled response data and the average perplexity of all distilled response data, wherein the first index represents performance of any distilled response data in terms of language fluency and distribution matching; the first index is negatively correlated with the perplexity of any distilled response data and positively correlated with the average perplexity;

[0021] Obtaining the similarity between any of the distilled response data and its corresponding original response data;

[0022] Calculating a comprehensive evaluation score of any distillation response based on the first indicator and the similarity;

[0023] If the comprehensive evaluation score is greater than the score threshold, it is determined that any of the distillation response data meets the quality condition; otherwise, it is determined that any of the distillation responses does not meet the quality condition.

[0024] In a possible implementation, fine-tuning the large model to be fine-tuned using the target training dataset includes:

[0025] Input the instruction data in each question-answer pair into the large model to be fine-tuned, and obtain the response data output by the large model to be fine-tuned;

[0026] Calculate learning loss based on the response data output by the large model to be fine-tuned, the answer in each question-answer pair, the key reasoning steps corresponding to each instruction data, and the distilled response data;

[0027] With the goal of reducing the learning loss, the parameters of the large model to be fine-tuned are updated.

[0028] In a possible implementation, the method further includes:

[0029] Perform a round of parameter update on the large model to be fine-tuned each time, and calculate the offset between the parameters updated in the current round and the parameters updated in the previous round;

[0030] If the offset is greater than the offset threshold, increase the proportion of the second type of question-answer pairs in the target training data set and reduce the proportion of the first type of question-answer pairs to obtain an updated training data set;

[0031] The updated training data set is used to perform the next round of parameter update on the large model to be fine-tuned.

[0032] In a possible implementation, the method further includes:

[0033] If the offset is less than or equal to the offset threshold, the ratio of the first category of question-answer pairs to the second category of question-answer pairs in the target training data set is adjusted to a target ratio.

[0034] A second aspect of the present application provides a large model fine-tuning device, comprising:

[0035] An acquisition module is configured to obtain a target training dataset for a target task, wherein the target training dataset includes a first type of question-answer pairs and a second type of question-answer pairs, wherein the questions in different question-answer pairs are different instruction data corresponding to the target task; the answers in the first type of question-answer pairs are original response data generated by the large model to be fine-tuned based on the instruction data in the first type of question-answer pairs, and the answers in the second type of question-answer pairs are distilled response data obtained by rewriting the original response data corresponding to the instruction data in the second type of question-answer pairs by the large model to be fine-tuned; and the distilled response data has the same reasoning logic and style as the corresponding original response data;

[0036] The fine-tuning module is used to perform fine-tuning training on the large model to be fine-tuned based on the target training data set.

[0037] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the large model fine-tuning method of the first aspect or any implementation of the first aspect.

[0038] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to execute the computer program so that the electronic device can implement the large model fine-tuning method of the first aspect or any implementation of the first aspect.

[0041] In a fifth aspect, the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the large model fine-tuning method of the above-mentioned first aspect or any implementation of the first aspect.

[0042] By means of the above-mentioned technical solution, the large model fine-tuning method, apparatus, equipment, storage medium and program product provided in the present application are used to generate a target training data set for fine-tuning the large model to be fine-tuned based on the large model to be fine-tuned, and the target training data set includes two types of question-answer pairs, wherein the answers in the second type of question-answer pairs are distilled response data obtained by rewriting the original response data corresponding to the instruction data in the second type of question-answer pairs by the large model to be fine-tuned; the distilled response data has the same reasoning logic and style as the corresponding original response data. Since the answers in the second type of question-answer pairs retain the reasoning path characteristics of the large model to be fine-tuned, fine-tuning the large model to be fine-tuned through the target training data set enhances the large model's ability to retain the original reasoning logic in target task learning, and improves the adaptability of the fine-tuned large model to the target task. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A flowchart for implementing the large model fine-tuning method provided in this application;

[0045] Figure 2 A flowchart for generating a target training dataset provided by this application;

[0046] Figure 3 A flowchart for implementing the rewriting of any original response data by using the large model to be fine-tuned provided in this application;

[0047] Figure 4 A flowchart for implementing quality assessment of any distillation response data provided in this application;

[0048] Figure 5 Another implementation flow chart for performing quality assessment on any distillation response data provided by this application;

[0049] Figure 6 A flowchart for implementing fine-tuning training of a large model to be fine-tuned using a target training dataset provided in this application;

[0050] Figure 7 A schematic diagram of the structure of a large model fine-tuning device provided in this application;

[0051] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0053] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0054] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0055] The inventors of this application have found that:

[0056] Fine-tuning a large model can significantly improve its performance on specific tasks, but this is often accompanied by a decrease in its original capabilities. The main reason for this phenomenon is that the task-specific datasets used to fine-tune the large model (composed of question-answer pairs adapted to the specific task, also known as the target task dataset) are typically real data collected from real-world scenarios or professional channels. The information distribution of the task-specific datasets differs significantly from that of the pre-trained large model (a large model pre-trained on general data, also known as the large model to be fine-tuned). Directly fine-tuning the pre-trained large model with the task-specific dataset can easily lead to the large model overfitting to new task data (i.e., the task-specific dataset), resulting in the loss of the pre-trained model's general capabilities while learning the new task (i.e., the specific task, also known as the target task).

[0057] To mitigate the catastrophic forgetting problem of fine-tuning large models, some traditional methods add distilled data to the collected task-specific datasets. This distillation method uses a large model with relatively high performance (a different model from the large model to be fine-tuned) to process the instruction data for a specific task (different from the instruction data in the collected task-specific dataset) to obtain response data corresponding to the instruction data for the specific task. The response data output by the large model with relatively high performance is then combined with the instruction data for the specific task to form question-answer pairs. The question-answer pairs obtained based on the large model with relatively high performance are then quality-checked, and the high-quality pairs are retained. These high-quality pairs are then mixed with the collected task-specific dataset to fine-tune the large model to be fine-tuned. This distillation method ignores the characteristics of the reasoning path of the large model to be fine-tuned and, therefore, fails to effectively utilize key information from the original reasoning path. This results in the responses generated by the fine-tuned large model being inconsistent in style and logic with the original model (the large model before fine-tuning), affecting the large model's performance and adaptability on new tasks.

[0058] Based on the above research findings, this application is proposed to improve the method of distilling target task data in order to improve the adaptability of the fine-tuned large model on specific tasks.

[0059] like Figure 1 As shown, a flowchart of an implementation of the large model fine-tuning method provided in an embodiment of the present application may include:

[0060] Step S101: Obtain a target training dataset for a target task.

[0061] The target task can be a task in any application field, for example, a task in the financial field, or a task in the customer service field, or a task in the marketing and business development field, etc.

[0062] The target training data set may be pre-generated and stored in a storage unit, and in this case the target training data set may be directly read from the storage unit.

[0063] Target training datasets can also be generated in real time.

[0064] The target training dataset includes the first type of question-answer pairs and the second type of question-answer pairs. The questions in different question-answer pairs are different instruction data corresponding to the target task, that is, different instruction data adapted to the target task; the answers in the first type of question-answer pairs are the response data generated by the large model to be fine-tuned based on the instruction data in the first type of question-answer pairs (referred to as original response data), and the answers in the second type of question-answer pairs are the response data obtained by rewriting the original response data corresponding to the instruction data in the second type of question-answer pairs by the large model to be fine-tuned (referred to as distilled response data); the distilled response data has the same reasoning logic and style as the corresponding original response data.

[0065] In this application, both the first-category question-answer pairs and the second-category question-answer pairs belong to the question-answer pairs of the target task, that is, both the first-category question-answer pairs and the second-category question-answer pairs are question-answer pairs adapted to the target task.

[0066] The answers in the first type of question-answer pairs can be obtained in the following way: the instruction data corresponding to the target task is input into the large model to be fine-tuned, and the large model to be fine-tuned generates original response data based on the input instruction data.

[0067] The answers in the second type of question-answer pairs can be obtained in the following way: the instruction data corresponding to the target task is input into the large model to be fine-tuned, and the large model to be fine-tuned generates original response data based on the input instruction data; the original response data is rewritten by the large model to be fine-tuned to obtain distilled response data; the distilled response data is different from the original response data, but retains the reasoning logic (including but not limited to: logical structures such as reasoning steps and the order between reasoning steps, as well as key features such as core actions and conclusions) and style (including but not limited to: speaking style such as professionalism and expression habits) of the original response data.

[0068] Compared with the original response data, the distilled response data may be a detailed description of the original response data. For example, "Put in the noodles and cook for 3 minutes" in the original response data can be "Put in the noodles and cook for 3 minutes until the noodles are soft" in the distilled impact data. By adding details, the interpretability of the reasoning path can be enhanced.

[0069] Compared to the original response data, the distilled response data may not provide a detailed description of the original response data, but may only change the sentence structure. If the original response data is clearly expressed, the rewrite may only adjust the sentence structure (for example, changing active sentences to passive sentences) without adding new information. The key is to ensure that the logical structure remains unchanged.

[0070] That is to say, regardless of whether the original response data is refined or not, the primary goal when rewriting the original response data is to "preserve the original reasoning path". The adjustment of the language expression serves the distribution consistency of the distilled response data and the original model, that is, the reasoning path of the distilled response data is consistent with that of the original response data.

[0071] Step S102: Fine-tune the large model to be fine-tuned based on the target training data set.

[0072] The process of fine-tuning the large model to be fine-tuned based on the target training dataset can refer to existing solutions, or you can use the training solution of this application (see below for details).

[0073] The large model fine-tuning method provided in the present application is a method for fine-tuning a large model to be fine-tuned, in which a target training dataset is generated based on the large model to be fine-tuned, and the target training dataset includes two types of question-answer pairs, wherein the answers in the second type of question-answer pairs are distilled response data obtained by rewriting the original response data corresponding to the instruction data in the second type of question-answer pairs by the large model to be fine-tuned; the distilled response data has the same reasoning logic and style as the corresponding original response data. Since the answers in the second type of question-answer pairs retain the reasoning path characteristics of the large model to be fine-tuned, fine-tuning the large model to be fine-tuned through the target training dataset enhances the large model's ability to retain the original reasoning logic in target task learning, and improves the adaptability of the fine-tuned large model to the target task.

[0074] In an optional embodiment, a flowchart for generating a target training data set is as follows: Figure 2 As shown, this may include:

[0075] Step S201: using the large model to be fine-tuned to process each instruction data in the instruction data set of the target task respectively, and generate response data (ie, original response data) corresponding to each instruction data.

[0076] Each instruction data in the instruction data set of the target task is a user instruction adapted to the target task. For example, if the target task is a question-and-answer question in the field of cooking, the instruction data may include "how to cook a bowl of instant noodles".

[0077] For each instruction data, the instruction data can be input into the large model to be fine-tuned (also called the seed model) to obtain the response of the large model to be fine-tuned to the instruction data, such as the response "boil water first, put in the noodles and cook for 3 minutes, add the seasoning packet and stir, and finally turn off the heat and serve."

[0078] Step S202: rewrite each original response data using the large model to be fine-tuned to obtain distilled response data corresponding to each original response data.

[0079] Optionally, for each original response data, the instruction data corresponding to the original response data and the original response data can be processed by the large model to be fine-tuned to obtain distilled response data corresponding to the original response data.

[0080] Step S203: performing quality assessment on each distillation response data, and retaining the distillation response data whose quality assessment results meet the quality conditions.

[0081] In order to ensure the quality of the distilled response data, ensure that the distilled response data not only conforms to the reasoning logic and style of the large model to be fine-tuned, but also retains the key features of the large model to be fine-tuned, it is necessary to perform a quality assessment on the distilled response data corresponding to each instruction data, so as to eliminate the distilled response data that does not meet the quality conditions and retain the distilled response data that meets the quality conditions, thereby ensuring that the quality of the distilled data matches the requirements of the target task.

[0082] Step S204: construct a target training dataset based on each instruction data and its corresponding original response data, as well as the retained distilled response data.

[0083] In the target training dataset, the ratio of the first type of question-answer pairs to the second type of question-answer pairs can be a target ratio, for example, 50% each.

[0084] Subsequently, the ratio of the first type of question-answer pairs to the second type of question-answer pairs can be adjusted in real time based on the actual fine-tuning situation.

[0085] The distilled response data in the second category of question-answer pairs is the retained distilled response data. In other words, for any instruction data, if the distilled response data corresponding to that instruction data is removed, that instruction data can only be used to construct a first category question-answer pair with its corresponding original response data. If the distilled response data corresponding to that instruction data is retained, that instruction data can be used to construct a first category question-answer pair with its corresponding original response data, or a second category question-answer pair with its corresponding distilled response data.

[0086] The distillation data in this application is obtained by distilling the large model to be fine-tuned itself, which is self-distillation.

[0087] In an optional embodiment, the present application provides a flowchart for rewriting any original response data by using a large model to be fine-tuned. Figure 3 As shown, this may include:

[0088] Step S301: parsing any original response data based on the instruction data corresponding to the any original response data by the large model to be fine-tuned to obtain the key reasoning steps of the any original response data.

[0089] Optionally, a parsing prompt word may be constructed, wherein the parsing prompt word includes a parsing task instruction, wherein the parsing task instruction indicates that the original response data is decomposed into multiple key reasoning steps according to the instruction data.

[0090] The key reasoning step is to input the parsing prompt words into the large model to be fine-tuned to obtain the original response data output by the large model to be fine-tuned.

[0091] As shown in Table 1, it is an example of the parsing prompt words provided in the embodiment of the present application.

[0092] Table 1

[0093]

[0094] In the prompt words shown in Table 1, "instruction" refers to "instruction data", that is, the user's question; "reference answer" refers to "original response data".

[0095] Corresponding to any original response data, replace "original_response" in the prompt word shown in Table 1 with the any original response data, and replace "instruction" in the prompt word shown in Table 1 with the instruction data corresponding to the any original response data to obtain the parsed prompt word corresponding to the any original response data, input the parsed prompt word into the large model to be fine-tuned, and obtain the key reasoning steps of the any original response data output by the large model to be fine-tuned.

[0096] For example, if the instruction data is "How to cook a bowl of instant noodles?" and this instruction data is input into the large model to be fine-tuned, the original response data obtained is "First boil water, add noodles and cook for 3 minutes, add seasoning packet and stir, finally turn off the heat and serve." Based on this, the following parsing prompt words can be constructed:

[0097] Please break down your reference answer into key reasoning steps according to the following instructions:

[0098] Instructions: {How to cook a bowl of instant noodles?}

[0099] Reference answer: {First boil water, add the noodles and cook for 3 minutes, add the seasoning packet and stir, then turn off the heat and serve.}

[0100] Reasoning steps:".

[0101] The above analysis prompt words are input into the large model to be fine-tuned, and the key reasoning steps output by the large model to be fine-tuned are obtained: "Boil water, cook noodles (3 minutes), add seasoning packet and stir, turn off the heat and serve."

[0102] Step S302: Generate distilled response data corresponding to any original response data based on the instruction data corresponding to any original response data and the above key reasoning steps through the large model to be fine-tuned.

[0103] Optionally, a reconstruction prompt word may be constructed, wherein the reconstruction prompt word includes a reconstruction task instruction, wherein the reconstruction task instruction instructs to generate a new answer that conforms to the data logical structure of the large model to be fine-tuned, using the key reasoning steps as a framework.

[0104] The key reasoning step is to input the reconstructed prompt word into the large model to be fine-tuned to obtain the original response data output by the large model to be fine-tuned.

[0105] As shown in Table 2, it is an example of the reconstruction prompt words provided in the embodiment of the present application.

[0106] Table 2

[0107]

[0108] In the prompt words shown in Table 2, "instruction" refers to "instruction data," that is, the user's question; "inference step" refers to the key inference step generated in step S301; "new answer" refers to the "distilled response data"; and "seed model" refers to the model that generates the "inference step," that is, the large model to be fine-tuned.

[0109] For any original response data, replace the "instruction" in the prompt word shown in Table 2 with the instruction data corresponding to the any original response data, and replace the "Reasoning Steps" in the prompt word shown in Table 2 with the key reasoning steps corresponding to the any original response data to obtain the reconstructed prompt word corresponding to the any original response data. Input the reconstructed prompt word into the large model to be fine-tuned to obtain the distilled response data corresponding to the any original response data output by the large model to be fine-tuned.

[0110] Continuing with the previous example of "how to cook a bowl of instant noodles", we can construct the following reconstruction prompt words:

[0111] Please follow the instructions below, using the reasoning steps as a framework, to generate new answers that conform to the logical structure of the seed model data. The answer style must be consistent with the distribution of the seed model training data and fully cover the information required by the instructions:

[0112] Instructions: {How to cook a bowl of instant noodles}

[0113] Reasoning steps: {Boil water, cook noodles (3 minutes), add seasoning packet and stir, turn off the heat and serve}

[0114] New answer: ".

[0115] The above reconstructed prompt words are input into the large model to be fine-tuned to obtain the distilled response data output by the large model to be questioned: "The steps for cooking instant noodles are as follows: first, pour an appropriate amount of water into the pot and boil it; then, put in the noodles and cook for 3 minutes until the noodles are soft; then, add the seasoning packet and stir evenly; finally, turn off the heat and serve the instant noodles in a bowl."

[0116] As can be seen from the above example, the new answer and the original response data have the same logic: "boil water -> cook noodles -> add seasoning -> serve", but they are expressed in different languages ​​to ensure that the reasoning path of the large model to be fine-tuned is preserved while adapting to the fine-tuning requirements of the new task.

[0117] In other words, this application introduces a multi-stage progressive guidance mechanism: in the first stage, the large model to be fine-tuned performs deep semantic abstraction decomposition on the original response data, accurately extracting a sequence of key intermediate reasoning nodes (also known as a logical architecture framework); in the second stage, based on the logical architecture framework obtained from the analysis, a new response instance that is highly isomorphic to the seed model data distribution is reconstructed through a constraint strategy. This two-stage process of "semantic deconstruction-logical reconstruction" can effectively capture and explicitly solidify the deep reasoning path characteristics of the large model to be fine-tuned, and then significantly enhance the feature distribution consistency of the distilled data and the seed model data in the distributed semantic space, laying the foundation for subsequent model knowledge inheritance and task adaptation, and ensuring a dynamic balance between the model's cognitive logic inheritance and the semantic fit of the new task during the fine-tuning process.

[0118] In an optional embodiment, the implementation flow chart of quality assessment of any distillation response data provided by the present application is as follows: Figure 4 As shown, this may include:

[0119] Step S401: Obtain the perplexity of any distilled response data and the degree of difference between the distilled response data and its corresponding original response data.

[0120] Perplexity is a common metric widely used in language model evaluation. For its specific calculation method, please refer to existing solutions and will not be described in detail here.

[0121] The degree of difference between any distilled response data and its corresponding original response data can be determined based on the similarity between the distilled response data and its corresponding original response data. The degree of difference between any distilled response data and its corresponding original response data is negatively correlated with the similarity between the distilled response data and its corresponding original response data. The degree of difference between any distilled response data and its corresponding original response data can be calculated using the following formula:

[0122] L contrast =((1-sim1)^2) / 2 (1)

[0123] Among them, L contrastrepresents the degree of difference between any distilled response data and its corresponding original response data, and sim represents the similarity between any distilled response data and its corresponding original response data. sim1 can be the cosine similarity between the feature vector of any distilled response data (which can be obtained by encoding the distilled response data using the encoding model) and the feature vector of its corresponding original response data (which can also be obtained by encoding the distilled response data using the encoding model). The encoding model can be the large model to be fine-tuned or other models, and this application does not impose any specific restrictions.

[0124] Step S402: Determine the evaluation result: If the perplexity is less than or equal to the perplexity threshold corresponding to the target task type, and the difference degree is less than the target difference degree, it is determined that any distilled response data meets the quality condition, indicating that the quality of any distilled data is high and should be retained; otherwise, it is determined that any distilled response does not meet the quality condition, indicating that the quality of any distilled data is low and should be eliminated.

[0125] This application predetermines the perplexity threshold based on the task type. Different task types may have the same or different perplexity thresholds. There are at least two task types with different perplexity thresholds.

[0126] Optionally, the perplexity threshold corresponding to any task type can be: the product of the average perplexity of the large model to be fine-tuned in tasks of the same type and the weight coefficient corresponding to the task type. As an example, assuming the target task type is mathematics, the perplexity threshold corresponding to the target task type can be: 0.8×ppl_seed=0.8×10=8. In this example, 0.8 is the weight coefficient corresponding to the mathematics task type, and the average perplexity of the large model to be fine-tuned in mathematics, ppl_seed=10, so the corresponding perplexity threshold is 8.

[0127] The average perplexity may be the average perplexity of the distilled response data obtained by the large model to be fine-tuned based on different instruction data under the same type of task. The different instruction data may be each instruction data in the instruction dataset of the target task.

[0128] This application evaluates distilled data using two comprehensive scoring branches. One branch (denoted as the main branch) evaluates the perplexity of the distilled response data, while the other branch (denoted as the auxiliary branch) evaluates the difference between the distilled response data and the corresponding original response data. This ensures that the distilled response data not only conforms to the "speaking style and logic" (distribution matching) of the original model (i.e., the large model to be fine-tuned), but also retains the key features of the original model (such as reasoning steps and expertise). Furthermore, the application dynamically adjusts the inspection criteria (i.e., perplexity threshold) based on the task type (e.g., math problems, general knowledge problems). For example, the requirements for math problems may be stricter, while those for general knowledge problems may be more relaxed. This avoids misjudgments caused by a "one-size-fits-all" approach and allows for flexible adaptation to different task requirements. The main branch acts as a "language quality checker," checking whether the distilled response data "speaks like the original model" (e.g., whether math problems are expressed using rigorous steps); the auxiliary branch acts as a "content quality checker," checking whether the new answers "have not lost core knowledge" (e.g., whether the calculation steps are correct).

[0129] In an optional embodiment, another implementation flow chart of quality assessment of any distillation response data provided by the present application is as follows: Figure 5 As shown, this may include:

[0130] Step S501: Determine a first index for any distilled response data based on the perplexity of any distilled response data and the average perplexity of all distilled response data. The first index represents the performance of the distilled response data in terms of language fluency and distribution matching. The first index is negatively correlated with the perplexity of the distilled response data and positively correlated with the average perplexity.

[0131] All distilled data may be all distilled response data obtained based on each instruction data in the instruction data set of the target task.

[0132] Optionally, the first indicator Target can be calculated as follows:

[0133] Target=1-ppl / ppl seed (2)

[0134] Among them, ppl represents the perplexity of any distilled response data, which reflects the matching degree between the distilled response data and the distribution of the model to be fine-tuned; ppl seed represents the average perplexity.

[0135] Step S502: obtaining the similarity between any distilled response data and its corresponding original response data.

[0136] The similarity between any distilled response data and its corresponding original response data may be: the cosine similarity between the feature vector of any distilled response data and its corresponding original response data. Specific implementation methods can be found in the aforementioned embodiments and will not be described in detail here.

[0137] Step S503: Calculate a comprehensive evaluation score of any distillation response based on the first indicator and the similarity.

[0138] The first indicator and the above similarity can be weighted and summed to obtain a comprehensive evaluation score. The formula can be expressed as:

[0139] (3)

[0140] Among them, S is the comprehensive evaluation score, which is used to comprehensively cover the quality of the distillation response; feat distill Represents the feature vector after encoding of any distillation response data; feat seed Represents the feature vector of the original response data corresponding to any distilled response data after encoding; sim (feat distill , feat seed ) represents the similarity between any distilled response data and its corresponding original response data, which is used to measure whether the distilled response retains the key features of the original response; α1 and β1 are weight coefficients, which can be adjusted according to the specific task type. That is, according to different task types, the weight coefficients may be different or the same to balance the importance of different evaluation indicators (the first indicator and similarity).

[0141] Step S504: Determine the evaluation result: if the comprehensive evaluation score is greater than the score threshold, determine that the any distillation response data meets the quality condition; otherwise, determine that the any distillation response does not meet the quality condition.

[0142] Optionally, the score threshold may be determined based on the task type, and the score thresholds corresponding to different task types may be the same or different. There may be at least two task types corresponding to different score thresholds.

[0143] This application evaluates distilled data using two comprehensive scoring branches: one branch (denoted as the main branch) evaluates the perplexity of the distilled response data, and the other branch (denoted as the auxiliary branch) evaluates the distilled response data based on the similarity between the distilled response data and the corresponding original response data. The main branch uses a quantitative measure of distribution matching (i.e., perplexity) to accurately calculate the conditional entropy (i.e., the first metric, Target) between the distilled response and the distribution generated by the fine-tuned model to characterize the fit of the semantic generation probability field. Semantic generation probability field fit is a high-level concept used to assess the degree of match between the output of a generative model (such as a language model or image generation model) and human semantic expectations. This metric goes beyond traditional surface statistical matching and delves into semantic alignment assessment. The auxiliary branch uses deep contrastive learning (i.e., contrastive similarity) to determine whether the distilled response data retains key cognitive features of the original model through similarity tests in the feature embedding space.

[0144] In addition, the weight coefficients of different branches can be adaptively adjusted according to the task type to avoid the uneven quality of distilled data caused by fixed weight coefficients, which makes it difficult for large models to balance the retention of original capabilities and adaptation to new tasks in new task learning, resulting in insufficient generalization ability and poor retention of reasoning logic for complex tasks. This application breaks through the bottleneck of existing solutions in comprehensive performance such as the ability to inherit original knowledge and adapt to new tasks in cross-domain tasks, ensuring that the quality of distilled data is accurately matched with task requirements, and achieving a precise balance between model knowledge inheritance and task adaptation.

[0145] The aforementioned process of generating the target training dataset belongs to the preprocessing process before fine-tuning the large model to be fine-tuned. The process of fine-tuning the large model to be fine-tuned is described in detail below.

[0146] In an optional embodiment, a flowchart for implementing fine-tuning training of the large model to be fine-tuned using the target training data set is as follows: Figure 6 As shown, this may include:

[0147] Step S601: Input the instruction data in each question-answer pair in the target training data set into the large model to be fine-tuned to obtain the response data output by the large model to be fine-tuned.

[0148] Step S602: Calculate the learning loss based on the response data output by the large model to be fine-tuned, the answer in each question-answer pair, the key reasoning steps corresponding to each instruction data, and the distilled response data.

[0149] The learning loss L can be calculated as follows total :

[0150] L total =L task +α2×L struct+β2×L cross (4)

[0151] Among them, L task The task loss represents the difference between the response data output by the fine-tuned large model and the answer in the question-answer pair. For example, it can be cross-entropy loss, mean squared error, negative log-likelihood loss, etc.

[0152] L struct The structural consistency loss represents the degree of difference between the distilled response data and the key reasoning steps corresponding to the instruction data input to the large model to be fine-tuned. It can be calculated based on the similarity between the distilled response data and the corresponding key reasoning steps. As an example, the structural consistency loss can be calculated using the following formula:

[0153] (5)

[0154] Among them, steps distilled Indicates the distilled response data corresponding to the instruction data input to the large model to be fine-tuned; steps seed Represents the key reasoning step corresponding to the distilled response data; Enc(·) is a hierarchical semantic encoder used to convert the distilled response data or key reasoning steps into vector form for easy calculation; sim(·,·) is a similarity calculation function, for example, it can be cosine similarity; Indicates the magnitude of a vector. The closer the value is to 0, the more consistent the structures of the two are.

[0155] L cross The contrastive learning loss represents the difference between the response data output by the fine-tuned large model and the original response data (i.e., ), which can be calculated as follows:

[0156] L cross =((1-sim2)^2) / 2 (6)

[0157] Here, sim2 represents the similarity between the feature vector of the response data output by the fine-tuned large model (which can be obtained by encoding the response data output by the fine-tuned large model using the encoding model) and the feature vector of the original response data (which can be obtained by encoding the original response data using the encoding model), such as cosine similarity. The encoding model can be the large model to be fine-tuned or another model, and this application does not impose any specific restrictions.

[0158] α2 and β2 are preset weight coefficients. These weight coefficients can vary depending on the task type. For example, if the target task type is mathematics, the weight coefficients can be: α2 = 0.7, β2 = 0.3. For another example, if the target task type is general knowledge question answering, the weight coefficients can be: α2 = 0.5, β2 = 0.5. It should be noted that the sum of α2 and β2 can be 1 or not.

[0159] Step S603: Update the parameters of the large model to be fine-tuned with the goal of minimizing the learning loss. That is, based on the same question-answer pair, the learning loss calculated based on the large model after the parameter update is smaller than the learning loss calculated before the parameter update.

[0160] In an optional embodiment, the present application can also adjust the ratio of the first and second question-answer pairs in the target training dataset in real time according to the parameter changes of the large model to be fine-tuned, so as to achieve a dynamic balance between "retaining the original ability" and "learning new tasks" for the large model, without losing the original ability and being able to effectively learn new tasks. Specifically, the large model fine-tuning method provided by this application can also include:

[0161] Each time a round of parameter update is performed on the large model to be fine-tuned, the offset between the updated parameters of the current round and the updated parameters of the previous round is calculated.

[0162] The large model to be fine-tuned includes several parameters, which can be regarded as a vector (denoted as a parameter vector). The difference between the parameter vector after the current round of update and the parameter vector after the previous round of update is calculated to obtain the difference vector, and the L2 norm of the difference vector is calculated as the parameter offset.

[0163] If the offset is greater than the offset threshold, the proportion of the second type of question-answer pairs in the target training dataset is increased, and the proportion of the first type of question-answer pairs is reduced to obtain an updated training dataset.

[0164] As an example, the offset threshold may be 0.1, or other values.

[0165] As an example, the proportion of the second type of question-answer pairs in the target training dataset can be determined as follows:

[0166] (7)

[0167] Among them, r t is the proportion of the second type of question-answer pairs in the target training dataset determined after the current round of parameter update (i.e., after the tth round of parameter update); r t-1 The proportion of the second type of question-answer pairs in the target training dataset determined after the previous round of parameter update (i.e., after the t-1th round of parameter update); The offset between the updated parameters of the current round and the updated parameters of the previous round; is the offset threshold, which is used to determine whether the parameter offset is significant; γ is the preset adjustment step (for example, it can be 0.05); sign (·) is the sign function. >ϵ (parameter offset is too large, overfitting may occur), , increasing the proportion of distilled response data by γ to strengthen the knowledge constraint of the seed model; when When ≤ϵ, , reducing the proportion of distilled response data to increase the weight of new task data, thereby dynamically balancing between "preserving original capabilities" and "learning new tasks".

[0168] Use the updated training dataset to perform the next round of parameter updates on the fine-tuned large model.

[0169] This application uses adaptive control theory to design a closed-loop feedback controller by online monitoring of the offset of model parameters. The controller dynamically adjusts the ratio (i.e., weight) of distilled data (i.e., the second type of question-answer pairs) to original task data (i.e., the first type of question-answer pairs) based on the parameter offset: when the parameter offset exceeds the preset offset threshold, the adaptive increase mechanism of the distilled data weight is triggered to strengthen the constraint of the large model to be fine-tuned on the original knowledge to suppress overfitting; when the parameter offset does not exceed the offset threshold, the weight of the original task data is dynamically increased to enhance the representation ability of the semantic space of the new task, ensuring the gradual improvement of the performance of the large model to be fine-tuned. Through this closed-loop feedback control mechanism based on adaptive control theory, the dynamic balanced optimization between knowledge inheritance and new task adaptation of the large model is achieved, ensuring that the model maintains the coherence of cognitive logic and the continuous improvement of new task performance during the fine-tuning process.

[0170] Furthermore, if the offset is less than or equal to the offset threshold, the ratio of the first category of question-answer pairs to the second category of question-answer pairs in the target training data set is adjusted to a target ratio.

[0171] The target ratio can be the initial ratio of the first type of question-answer pairs to the second type of question-answer pairs in the target training dataset, for example, 50% each.

[0172] For example, when fine-tuning a large medical Q&A model, the target training dataset consists of 50% each of distilled data (i.e., the second type of question-answer pairs) and original task data (i.e., the first type of question-answer pairs). In the first t-1 rounds, the parameter offset of the large medical Q&A model is detected to be less than or equal to the offset threshold. Therefore, the ratio of distilled data to original task data in the target training dataset remains unchanged. If the parameter offset of the large medical Q&A model is detected to be greater than the offset threshold in round t, it indicates that the large medical Q&A model may have forgotten general knowledge due to overfitting to new data. In this case, the proportion of distilled data can be increased to 60%, and the proportion of original task data can be reduced to 40% to strengthen the review of general knowledge and prevent overfitting. Subsequently, the rate of change of the parameter offset of the medical question-answering model will continue to be monitored. If the rate of change of the offset indicates that the offset is gradually decreasing, the proportion of distilled data and original task data can be kept unchanged. If the rate of change of the offset indicates that the offset is gradually increasing, the proportion of distilled data will be increased again, and the proportion of original task data will be reduced until the offset gradually decreases. When the offset gradually decreases and it is monitored that the parameter offset of the medical question-answering model is less than or equal to the offset threshold, the proportion of distilled data and original task data in the target training dataset will be adjusted to 50% each.

[0173] In summary, the large model fine-tuning method provided in this application first realizes deep feature mining and precise reconstruction of the reasoning path of the large model to be fine-tuned through multi-stage semantic structure and logic reconstruction technology. This not only retains the reasoning logic of the large model to be fine-tuned, but also enhances the semantic representation ability of the large model in new tasks. Secondly, the adaptive dual-path dynamic evaluation technology breaks through the limitations of traditional evaluation methods. Through the synergy of the main and auxiliary branches, combined with dynamic threshold adjustment, it realizes precise control of the quality of distillation data. Thirdly, the dynamic data mixing and multimodal loss collaborative optimization technology uses adaptive control theory to adjust the data ratio in real time, and through the dynamic balance of the comprehensive loss function, ensures the dynamic calibration of the model between new task learning and original ability retention. Through the joint action of the above three mechanisms on the large model fine-tuning process, a closed-loop process of "optimization-evaluation-adjustment" is formed. Compared with the existing technology, this application can more comprehensively and accurately mine and aggregate the semantic information of the large model, dynamically adjust the training strategy, and achieve a dynamic balance and collaborative optimization between the large model's learning of new tasks and the retention of original capabilities. It continuously improves the model performance and provides a new technical solution for the efficient fine-tuning of large models on specific tasks, significantly improving the generalization performance and adaptability of large models. In practical applications, whether it is complex reasoning tasks or specific field adaptation, this application can effectively avoid the problem of catastrophic forgetting and provide a more efficient and reliable solution for large model fine-tuning. It provides an innovative technical paradigm for the efficient fine-tuning of large models in vertical fields.

[0174] Corresponding to the method embodiment, the present application also provides a large model fine-tuning device. A structural diagram of the large model fine-tuning device provided in the embodiment of the present application is shown as follows: Figure 7 As shown, this may include:

[0175] Acquisition module 701 and fine-tuning module 702;

[0176] Among them, the acquisition module 701 is used to obtain a target training data set for the target task, wherein the target training data set includes a first type of question-answer pairs and a second type of question-answer pairs, where the questions in different question-answer pairs are different instruction data corresponding to the target task; the answers in the first type of question-answer pairs are original response data generated by the large model to be fine-tuned based on the instruction data in the first type of question-answer pairs, and the answers in the second type of question-answer pairs are distilled response data obtained by rewriting the original response data corresponding to the instruction data in the second type of question-answer pairs by the large model to be fine-tuned; the distilled response data has the same reasoning logic and style as the corresponding original response data;

[0177] The fine-tuning module 702 is used to perform fine-tuning training on the large model to be fine-tuned based on the target training data set.

[0178] The large model fine-tuning device provided in the present application is used to generate a target training dataset for fine-tuning the large model to be fine-tuned based on the large model to be fine-tuned. Moreover, the target training dataset includes two types of question-answer pairs, wherein the answers in the second type of question-answer pairs are distilled response data obtained by rewriting the original response data corresponding to the instruction data in the second type of question-answer pairs by the large model to be fine-tuned; the distilled response data has the same reasoning logic and style as the corresponding original response data. Since the answers in the second type of question-answer pairs retain the reasoning path characteristics of the large model to be fine-tuned, fine-tuning the large model to be fine-tuned through the target training dataset enhances the large model's ability to retain the original reasoning logic in target task learning, and improves the adaptability of the fine-tuned large model to the target task.

[0179] In an optional embodiment, the large model fine-tuning apparatus may further include: a generation module for generating the target training data set, specifically for:

[0180] Using the large model to be fine-tuned, each instruction data in the instruction data set of the target task is processed separately to generate original response data corresponding to each instruction data;

[0181] Rewriting each original response data using the large model to be fine-tuned to obtain distilled response data corresponding to each original response data;

[0182] Performing quality assessment on each distillation response data, and retaining the distillation response data whose quality assessment results meet the quality conditions;

[0183] The target training dataset is constructed based on each instruction data and its corresponding original response data, as well as the retained distilled response data.

[0184] In an optional embodiment, when the generation module rewrites any original response data using the large model to be fine-tuned, it is configured to:

[0185] Parsing any of the original response data based on the instruction data corresponding to the any of the original response data by the large model to be fine-tuned to obtain key reasoning steps for the any of the original response data;

[0186] The distilled response data corresponding to any original response data is generated by the large model to be fine-tuned based on the instruction data corresponding to any original response data and the key reasoning steps.

[0187] In an optional embodiment, when the generation module performs quality assessment on any distillation response data, it is configured to:

[0188] Obtaining the perplexity of any distilled response data and the degree of difference between any distilled response data and its corresponding original response data;

[0189] If the perplexity is less than or equal to the perplexity threshold corresponding to the target task type, and the difference degree is less than the target difference degree, it is determined that any of the distilled response data meets the quality condition; otherwise, it is determined that any of the distilled responses does not meet the quality condition.

[0190] In an optional embodiment, when the generation module performs quality assessment on any distillation response data, it is configured to:

[0191] determining a first index for any distilled response data based on the perplexity of any distilled response data and the average perplexity of all distilled response data, wherein the first index represents performance of any distilled response data in terms of language fluency and distribution matching; the first index is negatively correlated with the perplexity of any distilled response data and positively correlated with the average perplexity;

[0192] Obtaining the similarity between any of the distilled response data and its corresponding original response data;

[0193] Calculating a comprehensive evaluation score of any distillation response based on the first indicator and the similarity;

[0194] If the comprehensive evaluation score is greater than the score threshold, it is determined that any of the distillation response data meets the quality condition; otherwise, it is determined that any of the distillation responses does not meet the quality condition.

[0195] In an optional embodiment, when fine-tuning the large model to be fine-tuned using the target training data set, the fine-tuning module 702 is configured to:

[0196] Input the instruction data in each question-answer pair into the large model to be fine-tuned, and obtain the response data output by the large model to be fine-tuned;

[0197] Calculate learning loss based on the response data output by the large model to be fine-tuned, the answer in each question-answer pair, the key reasoning steps corresponding to each instruction data, and the distilled response data;

[0198] With the goal of reducing the learning loss, the parameters of the large model to be fine-tuned are updated.

[0199] In an optional embodiment, the fine-tuning module 702 is further configured to:

[0200] Perform a round of parameter update on the large model to be fine-tuned each time, and calculate the offset between the parameters updated in the current round and the parameters updated in the previous round;

[0201] If the offset is greater than the offset threshold, increase the proportion of the second type of question-answer pairs in the target training data set and reduce the proportion of the first type of question-answer pairs to obtain an updated training data set;

[0202] The updated training data set is used to perform the next round of parameter update on the large model to be fine-tuned.

[0203] In an optional embodiment, the fine-tuning module 702 is further configured to:

[0204] If the offset is less than or equal to the offset threshold, the ratio of the first category of question-answer pairs to the second category of question-answer pairs in the target training data set is adjusted to a target ratio.

[0205] Each module in the large model fine-tuning device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0206] An electronic device is also provided in an embodiment of the present application. Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application can be a terminal device (such as a car computer, a large-screen device, a smart home, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.), or a server (which can be a single server, a server cluster, or a cloud server, etc.). Figure 8The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0207] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 802 or programs loaded from a storage device 808 into a random access memory (RAM) 803. When the electronic device is powered on, the RAM 803 also stores various programs and data required for the operation of the electronic device. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0208] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a memory card, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 8 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0209] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any large model fine-tuning method provided in the embodiment of the present application.

[0210] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any large model fine-tuning method provided in the embodiment of the present application.

[0211] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0212] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0213] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0214] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0215] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0216] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large model fine-tuning method, characterized in that: include: Obtain a target training dataset for a target task, wherein the target training dataset includes a first type of question-answer pairs and a second type of question-answer pairs, wherein questions in different question-answer pairs are different instruction data corresponding to the target task; The answer to the first type of question-answer pair is the original response data generated by the large model to be fine-tuned based on the instruction data in the first type of question-answer pair, and the answer to the second type of question-answer pair is the distilled response data obtained by rewriting the original response data corresponding to the instruction data in the second type of question-answer pair by the large model to be fine-tuned; The distilled response data has the same reasoning logic and style as the corresponding original response data; Fine-tune the large model to be fine-tuned based on the target training data set.

2. The method according to claim 1, characterized in that The target training dataset is generated in the following way: Using the large model to be fine-tuned, each instruction data in the instruction data set of the target task is processed separately to generate original response data corresponding to each instruction data; Rewriting each original response data using the large model to be fine-tuned to obtain distilled response data corresponding to each original response data; Performing quality assessment on each distillation response data, and retaining the distillation response data whose quality assessment results meet the quality conditions; The target training dataset is constructed based on each instruction data and its corresponding original response data, as well as the retained distilled response data.

3. The method according to claim 2, characterized in that Rewriting any original response data by using the large model to be fine-tuned includes: Parsing any of the original response data based on the instruction data corresponding to the any of the original response data by the large model to be fine-tuned to obtain key reasoning steps for the any of the original response data; The distilled response data corresponding to any original response data is generated by the large model to be fine-tuned based on the instruction data corresponding to any original response data and the key reasoning steps.

4. The method according to claim 2, characterized in that Perform quality assessment on any distillation response data, including: Obtaining the perplexity of any distilled response data and the degree of difference between any distilled response data and its corresponding original response data; If the perplexity is less than or equal to the perplexity threshold corresponding to the type of the target task, and the difference degree is less than the target difference degree, it is determined that any of the distilled response data meets the quality condition; otherwise, it is determined that any of the distilled responses does not meet the quality condition.

5. The method according to claim 2, characterized in that Perform quality assessment on any distillation response data, including: determining a first index for any distilled response data based on the perplexity of any distilled response data and the average perplexity of all distilled response data, wherein the first index represents performance of any distilled response data in terms of language fluency and distribution matching; the first index is negatively correlated with the perplexity of any distilled response data and positively correlated with the average perplexity; Obtaining the similarity between any of the distilled response data and its corresponding original response data; Calculating a comprehensive evaluation score of any distillation response based on the first indicator and the similarity; If the comprehensive evaluation score is greater than the score threshold, it is determined that any of the distillation response data meets the quality condition; otherwise, it is determined that any of the distillation responses does not meet the quality condition.

6. The method according to claim 3, characterized in that The fine-tuning training of the large model to be fine-tuned using the target training data set includes: Input the instruction data in each question-answer pair into the large model to be fine-tuned, and obtain the response data output by the large model to be fine-tuned; Calculate learning loss based on the response data output by the large model to be fine-tuned, the answer in each question-answer pair, the key reasoning steps corresponding to each instruction data, and the distilled response data; With the goal of reducing the learning loss, the parameters of the large model to be fine-tuned are updated.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: Each time a round of parameter update is performed on the large model to be fine-tuned, the offset between the updated parameters of the current round and the updated parameters of the previous round is calculated; If the offset is greater than the offset threshold, increase the proportion of the second type of question-answer pairs in the target training data set and reduce the proportion of the first type of question-answer pairs to obtain an updated training data set; The updated training data set is used to perform the next round of parameter update on the large model to be fine-tuned.

8. The method according to claim 7, characterized in that Also includes: If the offset is less than or equal to the offset threshold, the ratio of the first category of question-answer pairs to the second category of question-answer pairs in the target training data set is adjusted to a target ratio.

9. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the large model fine-tuning method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so as to enable the electronic device to implement the large model fine-tuning method according to any one of claims 1 to 8.

Images