Model fine tuning method and device, storage medium and electronic equipment

Abstract

The invention provides a model fine tuning method and device, a storage medium and electronic equipment. The method comprises the steps that firstly, based on first training data, full-amount fine adjustment is conducted on a pre-training model, and a model after full-amount fine adjustment is obtained; performing direct preference optimization training on the model after the full fine tuning based on the second training data to obtain a direct preference optimization model; and then performing efficient parameter fine tuning on the direct preference optimization model based on the third training data to obtain a fine-tuned model corresponding to the target task. According to the fine-tuned model obtained by applying the technical scheme of the invention, the processing performance in the specific target task can be improved, and the processing accuracy of the model in the specific target task can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a model fine-tuning method, device, storage medium, and electronic device. Background Art

[0002] Large models generally refer to AI models with large parameters and complex architectures, particularly deep learning models. Currently, large models can be pre-trained on large, general-purpose datasets, known as pre-trained models. However, due to domain or usage scenarios, these pre-trained models may perform poorly in specific target tasks, affecting the model's accuracy in handling these tasks. Summary of the Invention

[0003] The present disclosure provides a model fine-tuning method, device, storage medium and electronic device, the main purpose of which is to improve the technical problem that the current pre-trained model obtained by pre-training on a large-scale general data set has poor performance in a specific target task, affecting the processing accuracy of the model in the target task.

[0004] According to a first aspect of an embodiment of the present disclosure, a model fine-tuning method is provided, comprising:

[0005] Obtain first training data and second training data for a target task;

[0006] Performing full fine-tuning on the pre-trained model based on the first training data to obtain a fully fine-tuned model;

[0007] Performing direct preference optimization training on the fully fine-tuned model based on the second training data to obtain a direct preference optimization model;

[0008] According to the direct preference optimization model, a fine-tuned model corresponding to the target task is determined.

[0009] Optionally, obtaining first training data for the target task includes:

[0010] Obtaining different request texts and correct output information corresponding to the request texts;

[0011] Combining the constructed full fine-tuning instruction with the request text to obtain a first input field, and using the correct output information as a first output field corresponding to the first input field;

[0012] The first training data is constructed based on the first input field and the first output field.

[0013] Optionally, obtaining second training data for the target task includes:

[0014] Obtain different request texts, as well as correct output information and error output information corresponding to the request texts;

[0015] Combine the constructed full-scale fine-tuning instructions with the request text to obtain a second input field, and combine the correct output information and the error output information into paired preference data as a second output field corresponding to the second input field;

[0016] Construct the second training data based on the second input field and the second output field.

[0017] Optionally, the method further includes:

[0018] Directly perform parameter-efficient fine-tuning on the pre-trained model that has not been fine-tuned according to the target task;

[0019] Use the pre-trained model after parameter-efficient fine-tuning for testing;

[0020] Analyze the rules based on the error output information and correct output information in the test results, and combine with the corresponding test request text;

[0021] Construct different request texts according to the rule analysis results, as well as correct output information and / or error output information corresponding to the request texts.

[0022] Optionally, determining the fine-tuned model corresponding to the target task according to the direct preference optimization model includes:

[0023] Obtain the third training data of the target task;

[0024] Perform parameter-efficient fine-tuning on the direct preference optimization model based on the third training data to obtain a fine-tuned model corresponding to the target task.

[0025] Optionally, obtaining the third training data of the target task includes:

[0026] Obtain different request texts and correct output information corresponding to the request texts;

[0027] Combine the constructed parameter-efficient fine-tuning instructions with the request text to obtain a third input field, and use the correct output information as a third output field corresponding to the third input field;

[0028] Construct the third training data based on the third input field and the third output field.

[0029] Optionally, after performing parameter-efficient fine-tuning on the direct preference optimization model based on the third training data to obtain a fine-tuned model corresponding to the target task, the method further includes:

[0030] Obtain the evaluation data corresponding to the parameter efficient fine-tuning instruction;

[0031] Based on the parameter efficient fine-tuning instruction and the evaluation data, evaluate the fine-tuned model;

[0032] Determine whether to further fine-tune the fine-tuned model according to the evaluation result.

[0033] Optionally, the data volume of the first training data is less than the data volume of the third training data, and the data volume of the second training data is less than the data volume of the third training data.

[0034] According to a second aspect of the embodiments of the present disclosure, there is provided a model fine-tuning device, including:

[0035] An acquisition module, configured to acquire first training data and second training data of a target task;

[0036] A fine-tuning module, configured to perform full-scale fine-tuning on a pre-trained model based on the first training data to obtain a fully fine-tuned model; perform direct preference optimization training on the fully fine-tuned model based on the second training data to obtain a direct preference optimization model; and determine a fine-tuned model corresponding to the target task according to the direct preference optimization model.

[0037] According to a third aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the model fine-tuning method described in the first aspect is implemented.

[0038] According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, and when the processor executes the computer program, the model fine-tuning method described in the first aspect is implemented.

[0039] By means of the above technical solutions, the present disclosure provides a model fine-tuning method, device, storage medium, and electronic device. For a pre-trained model pre-trained on a large-scale general data set, the present disclosure can effectively fine-tune it. Specifically, first, acquire first training data and second training data of a target task; then perform full-scale fine-tuning on the pre-trained model based on the first training data to obtain a fully fine-tuned model; then further perform direct preference optimization training on the fully fine-tuned model based on the second training data to obtain a direct preference optimization model; and further determine a fine-tuned model corresponding to the target task according to the direct preference optimization model. The fine-tuned model obtained by applying the technical solutions of the present disclosure can improve the processing performance in a specific target task and can improve the processing accuracy of the model in a specific target task.

[0040] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0042] Figure 1 It shows a schematic flowchart of a model fine-tuning method provided by an embodiment of the present disclosure;

[0043] Figure 2 It shows a schematic diagram of an example provided by an embodiment of the present disclosure;

[0044] Figure 3 It shows a schematic flowchart of another model fine-tuning method provided by an embodiment of the present disclosure;

[0045] Figure 4 It shows a schematic structural diagram of another example provided by an embodiment of the present disclosure;

[0046] Figure 5 It shows a schematic structural diagram of a model fine-tuning device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] Some embodiments of the present disclosure will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely exemplary and is not limited to those set forth herein, but may be changed as will be apparent after understanding the present disclosure, except for operations that must be performed in a specific order. Additionally, descriptions of features known in the art may be omitted for the sake of clarity and conciseness. It should be noted that, without conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0048] The embodiments described in some embodiments of the present disclosure below do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0049] To address the technical problem that the performance of a pre-trained model pre-trained on a large-scale general dataset is poor in a specific target task, which affects the processing accuracy of the model in that target task, the embodiments of the present disclosure provide a model fine-tuning method, as Figure 1 shown. This method can be applied to a model fine-tuning device or equipment for execution, and can be configured on the terminal or server side. The method includes the following steps.

[0050] Step 101: Obtain the first training data and the second training data for the target task.

[0051] Among them, the target task can refer to a specific task or problem that the pre-trained model needs to complete in business. The pre-trained model can be pre-trained on a large-scale general dataset. For example, the target task can be text classification, information extraction (or information retrieval), named entity recognition, text summary generation, semantic similarity calculation, voice control instruction recognition, or machine translation, etc.

[0052] The first training data can be training data related to the target task. The first training data can include model input data related to the target task and the correct model output data corresponding to the model input data, etc.

[0053] For example, for the target task of text summary generation, the model input data can be a long text (request text), and the corresponding correct model output data can be the correct summary text (correct output information) corresponding to the long text. By fine-tuning the pre-trained model with this training data, the pre-trained model can further strengthen the learning of how to better perform text summary generation, and the subsequent fine-tuned model can handle the text summary generation task more accurately.

[0054] For another example, for the target task of machine translation from Chinese to English, the model input data can be a Chinese text (request text), and the corresponding correct model output data can be the correct English translation text (correct output information) corresponding to the Chinese text. By fine-tuning the pre-trained model with this training data, the pre-trained model can further strengthen the learning of how to better perform machine translation from Chinese to English, and the subsequent fine-tuned model can handle the Chinese-to-English machine translation task more accurately.

[0055] The second training data can be training data related to the target task. The second training data can include model input data related to the target task and the correct model output data, model incorrect output data, etc. corresponding to the model input data.

[0056] For example, for the target task of voice control instruction recognition, the model input data can be the user's voice text (request text), the corresponding correct output data of the model can be the correct control instruction (correct output information) corresponding to the voice text, and the incorrect output data of the model can be the incorrect control instruction (incorrect output information) corresponding to the voice text. The model is further fine-tuned with this training data, enabling the model to further strengthen its learning of how to better recognize voice control instructions. The subsequent fine-tuned model processes the task of voice control instruction recognition more accurately.

[0057] In some embodiments, the first training data can be used to perform full-scale fine-tuning on the pre-trained model, while the second training data can be used to further perform direct preference optimization training on the model. Among them, full-scale fine-tuning can be used to update all parameters of the pre-trained model, requiring more computing resources and time. Through full-scale fine-tuning, the model can be more comprehensively adapted to the processing of the target task. Direct preference optimization training can be used to strengthen the model's intent recognition ability, improve the understanding ability of instructions, improve the accuracy of intent type classification, strengthen the learning of the problem of fuzzy category boundaries, and correct the model's classification ability.

[0058] Step 102: Perform full-scale fine-tuning on the pre-trained model based on the first training data to obtain a fully fine-tuned model.

[0059] In some embodiments, the full-scale fine-tuning instruction and the request text can be combined into an input field, and the correct output information corresponding to the request text can be used as the field that the model should output. These input fields and their corresponding output fields are used as the first training data to be input into the pre-trained model for full-scale fine-tuning, updating all parameters of the pre-trained model, enabling the model to gradually learn the basic features and patterns of the target task, adapt to the core requirements of the target task, and help the model understand what the target task is specifically for.

[0060] Among them, the full-scale fine-tuning instruction can be used to perform full-scale fine-tuning on the pre-trained model. During the process of performing full-scale fine-tuning on the pre-trained model, the full-scale fine-tuning instruction can be used to instruct the model to output corresponding information and can be constructed according to the target task. For example, for the target task of text summary generation, the model input data can be a long text, and the corresponding correct output data of the model can be the correct summary text corresponding to the long text. In order for the model to output the summary text, the full-scale fine-tuning instruction can be an instruction to instruct the model to output the summary text corresponding to the long text, such as please output the summary content in the following text (the long text).

[0061] Optionally, after constructing the full-scale fine-tuning instructions according to the target task, the constructed full-scale fine-tuning instructions can be further adjusted to better meet the actual requirements and better instruct the model to output the corresponding information. For example, for the target task of text summary generation, where a summary needs to be generated for specific parts of a long text, the full-scale fine-tuning instructions can be constructed as Please output the summary content of the third to fifth paragraphs in the following text (this long text).

[0062] In the embodiments of the present disclosure, the pre-trained model is fine-tuned in full scale with the first training data, which can accurately guide the model to learn the core features of the target task and significantly improve the model's intention recognition ability.

[0063] Step 103: Perform direct preference optimization training on the model fine-tuned in full scale based on the second training data to obtain a directly preference-optimized model.

[0064] Combine the full-scale fine-tuning instructions and the request text into an input field, and form paired preference data from the correct output information and incorrect output information corresponding to the request text as the output field. Input these input fields and their corresponding output fields into the model fine-tuned in full scale as the second training data for direct preference optimization training, which is used to strengthen the model's intention recognition ability, improve the understanding ability of instructions, improve the accuracy of intention type classification, strengthen the learning of the problem of fuzzy category boundaries, and correct the model's classification ability.

[0065] Step 104: Determine the fine-tuned model corresponding to the target task according to the directly preference-optimized model.

[0066] The fine-tuned model obtained by applying the technical solution of the embodiments of the present disclosure can improve the processing performance in a specific target task and improve the processing accuracy of the model in a specific target task.

[0067] For the method of model fine-tuning, as an example, parameter-efficient fine-tuning can be directly performed on the pre-trained model, as Figure 2 shown. First, collect the training data for the corresponding target task, construct parameter-efficient fine-tuning instructions according to the requirements of the target task, and complete the instruction adjustment of the training data. For example, combine the parameter-efficient fine-tuning instructions and the request text in the training data into an input field, and use the correct output information corresponding to the request text in the training data as the field that the model should output. Input these input fields and their corresponding output fields into the pre-trained model for parameter-efficient fine-tuning training to obtain the fine-tuned model.

[0068] In some examples, in the relevant applications of large models for target tasks such as information extraction, due to the high requirements for the effect in business, simply generating corresponding responses by means of instructions is difficult to meet the business requirements in terms of effect. Moreover, the instructions constructed in this way need to be detailed and complete, which will result in very long instructions and affect the generation speed. Therefore, it is necessary to fine-tune the large model. Additionally, considering the generation speed, when constructing the large model data, it often tends to construct short instruction data, and then through the training of the large model, learn the information extraction format and establish relevant mappings. Specifically, the training can be carried out by means of parameter-efficient fine-tuning, such as Figure 2 the method shown. However, in business, the problem of generation speed will make the constructed parameter-efficient fine-tuning instructions relatively simple, mainly relying on the mapping relationship learning task of the training data, without making efficient use of the capabilities of the large model. The simple parameter-efficient fine-tuning instructions can achieve format compliance after training, but they cannot understand what the current task is specifically for. Therefore, complete instruction information is needed to enable the model to better learn and understand the current task. And in specific business, the need for classification tasks is often encountered. In this type of task, due to the problem of blurred boundaries between categories, some data are easily misclassified into other categories. Therefore, this solution can construct a preference learning algorithm, which is beneficial for the model to efficiently learn human preferences and reduce the number of such errors, so as to improve the processing performance of the model.

[0069] For this reason, based on Figure 1 the method shown, as an optional way, this disclosure further proposes a method for improving the business understanding ability of a pre-trained model based on three-stage fine-tuning, such as Figure 3 shown, the method includes:

[0070] Step 201, obtain the first training data, the second training data, and the third training data of the target task.

[0071] Among them, the target task may refer to the specific task or problem that the pre-trained model needs to complete in business. The pre-trained model can be pre-trained on a large-scale general dataset. The first training data, the second training data, and the third training data can be training data related to the target task.

[0072] In some embodiments, the first training data can be used for full-scale fine-tuning training in the first stage. The first training data can be generated according to full-scale fine-tuning instructions, different request texts, and the correct output information corresponding to these request texts respectively.

[0073] In some examples, different request texts and the correct output information corresponding to the request texts can be obtained first; then, the constructed full-scale fine-tuning instructions are combined with the request texts to obtain a first input field, and the correct output information is used as a first output field corresponding to the first input field; then, first training data is constructed based on the first input field and the first output field.

[0074] The second training data can be used for direct preference optimization training in the second stage. The second training data can be generated according to the full-scale fine-tuning instructions, different request texts, and the correct output information, incorrect output information, etc. respectively corresponding to these request texts.

[0075] In some examples, different request texts, the correct output information, and the incorrect output information corresponding to the request texts can be obtained first; then, the constructed full-scale fine-tuning instructions are combined with the request texts to obtain a second input field, and the correct output information and the incorrect output information are combined into paired preference data as a second output field corresponding to the second input field; then, second training data is constructed based on the second input field and the second output field.

[0076] In some examples, first, parameter-efficient fine-tuning is directly performed on the pre-trained model that has not been fine-tuned according to the target task; then, the pre-trained model after parameter-efficient fine-tuning is used for testing; then, based on the incorrect output information and the correct output information in the test results, and in combination with the corresponding test request texts, rule analysis is performed; furthermore, different request texts, and the correct output information and / or incorrect output information corresponding to the request texts are constructed according to the rule analysis results, and then, in cooperation with the full-scale fine-tuning instructions, first training data and second training data are respectively constructed.

[0077] For example, request texts related to the target task and the correct output information corresponding to the request texts are obtained; the parameter-efficient fine-tuning instructions are combined with the request texts to obtain an input field, and the correct output information is used as the output field corresponding to the input field; then, a training set is constructed based on these input fields and the output fields corresponding to these input fields, and the constructed training set is used to train the pre-trained model for parameter-efficient fine-tuning to obtain the pre-trained model after parameter-efficient fine-tuning. Based on the above Figure 2Description of an example. For a model obtained by directly performing parameter-efficient fine-tuning, its ability to recognize intents is lacking, its understanding of instructions is not in place, and there are many cases of incorrect classification of intent types. Therefore, the embodiments of the present disclosure can use the pre-trained model after parameter-efficient fine-tuning for testing. For example, testing is performed through a test set. The test instructions and the test request text are combined to obtain an input field, which is then input into the model. The difference between the model output result and the corresponding correct result is compared, and then error output information and correct output information are collected. Then, based on the error output information and correct output information in the test results, and combined with the corresponding test request text, a pattern analysis is performed. Furthermore, different request texts are constructed according to the summarized patterns, as well as the correct output information and error output information corresponding to these request texts respectively.

[0078] The third training data can be used for parameter-efficient fine-tuning training in the third stage. Parameter-efficient fine-tuning is different from full-scale fine-tuning. Parameter-efficient fine-tuning aims to minimize the number of parameters that need to be updated while maintaining or approaching the performance of full-scale fine-tuning. It can include various techniques, such as only fine-tuning some layers, introducing an adapter module, using specific regularization methods, etc. Since parameter-efficient fine-tuning only needs to update a small part of the parameters in the model, the required computing resources and time are both less.

[0079] In some embodiments, different request texts and the correct output information corresponding to the request texts are first obtained; then the constructed parameter-efficient fine-tuning instructions are combined with the request texts to obtain a third input field, and the correct output information is used as the third output field corresponding to the third input field; then the third training data is constructed based on the third input field and the third output field.

[0080] In some embodiments, the parameter-efficient fine-tuning instructions can be used to perform parameter-efficient fine-tuning on the model. During the process of performing parameter-efficient fine-tuning on the model, the parameter-efficient fine-tuning instructions can be used to instruct the model to output corresponding information, and can be constructed according to the target task. For example, for the target task of machine translation from Chinese to English, the model input data can be a Chinese text, and the corresponding correct output data of the model can be the correct English translation text corresponding to the Chinese text. In order for the model to output the English translation text, the parameter-efficient fine-tuning instructions can be instructions to instruct the model to output the English translation text corresponding to the Chinese text, such as Please translate the following text (this Chinese text) into English.

[0081] Optionally, after constructing the parameter-efficient fine-tuning instruction according to the target task, the constructed parameter-efficient fine-tuning instruction can be further adjusted to better meet the actual requirements and better instruct the model to output the corresponding information. For example, for the target task of machine translation from Chinese to English, the input data of the model can be a Chinese text, but specific parts of the Chinese text need to be translated into English. Then the parameter-efficient fine-tuning instruction can be constructed as "Please translate the content from line 6 to line 10 of the following text (this Chinese text) into English."

[0082] In some embodiments, the data volume of the first training data can be smaller than that of the third training data. Since full fine-tuning requires more computing resources and time, less training data can be used for full fine-tuning of the model to save computing resources and time, while achieving the purpose of more comprehensively adapting to the target task processing. In addition, the data volume of the second training data can also be smaller than that of the third training data, so as to achieve the learning purpose of strengthening the learning of the problem of fuzzy class boundaries through a small amount of data.

[0083] In some examples, more complete full fine-tuning instructions can be constructed so that the subsequent model training can strictly execute the target tasks, such as performing tasks like text classification, information extraction, named entity recognition, or text summary generation.

[0084] In some examples, compared with constructing more complete full fine-tuning instructions, the embodiments of the present disclosure can construct short parameter-efficient fine-tuning instructions, which can improve the output result generation speed of the model in the parameter-efficient fine-tuning stage, and thus improve the efficiency of model fine-tuning.

[0085] Step 202: Perform full fine-tuning on the pre-trained model based on the first training data to obtain the fully fine-tuned model.

[0086] For example, for the target task of machine translation from English to Japanese, obtain the English text and the corresponding Japanese translation text of the English text. Combine the full fine-tuning instruction (such as "Please translate the following text into Japanese") with the English text to obtain the input field, and use the Japanese translation text as the output field corresponding to the input field. In this way, each input field and the output field corresponding to each input field can be obtained, and then these input fields and their corresponding output fields are used to construct a training set, and the pre-trained model is trained with the constructed training set for full fine-tuning, so that the pre-trained model can further strengthen the learning of how to perform the machine translation task from English to Japanese.

[0087] In this way, the intention recognition ability of the model can be significantly improved, and then the understanding ability of the model is improved, so that the fine-tuned model can accurately execute the corresponding target task.

[0088] Step 203: Perform direct preference optimization training on the fully fine-tuned model based on the second training data to obtain a direct preference optimization model.

[0089] In the embodiments of the present disclosure, the pre-trained model can be sequentially subjected to full-scale fine-tuning training in the first stage and direct preference optimization training in the second stage to obtain a direct preference optimization model.

[0090] For the model directly obtained by parameter-efficient fine-tuning, its intention recognition ability is lacking, its understanding of instructions is not in place, and there are many cases of incorrect classification of intention types. In the embodiments of the present disclosure, errors caused by fuzzy category boundaries in the error data output by the model can be used to construct similar data as error output information. The construction of this data can strengthen the learning of the fuzzy category boundary problem and correct the classification ability of the model. In addition, the corresponding request text and correct output information are constructed together to obtain the first training data. In this case, if it is necessary to perform the second-stage direct preference optimization training on the model after full-scale fine-tuning training, the full-scale fine-tuning instruction can be combined with the request text to obtain an input field, and the correct output information and error output information are combined into paired preference data as the output field corresponding to the input field. According to this method, each input field and the output field corresponding to each input field can be obtained, and then these input fields and their corresponding output fields are used to construct a training set, and the model after full-scale fine-tuning training is trained by the constructed training set for direct preference optimization to obtain a direct preference optimization model.

[0091] For example, for the target task of text classification, the text to be classified and its corresponding correct classification and incorrect classification are obtained. The full-scale fine-tuning instruction (such as please determine the category of this text) is combined with the text to be classified to obtain an input field, and the correct classification and incorrect classification are combined into paired preference data as the output field corresponding to the input field. According to this method, each input field and the output field corresponding to each input field can be obtained, and then these input fields and their corresponding output fields are used to construct a training set, and the model after full-scale fine-tuning training is trained by the constructed training set for direct preference optimization to enable the model to further strengthen the learning of category errors and improve the discrimination ability of intention-fuzzy errors.

[0092] Step 204: Perform parameter-efficient fine-tuning on the direct preference optimization model based on the third training data to obtain a fine-tuned model corresponding to the target task.

[0093] Combine the parameter-efficient fine-tuning instruction and the request text into an input field, and use the correct output information corresponding to the request text as the field that the model should output. Input these input fields and their corresponding output fields into the direct preference optimization model for parameter-efficient fine-tuning, update some parameters of the model, further optimize the model performance, and at the same time reduce the demand for computing resources. After parameter-efficient fine-tuning, a fine-tuned model corresponding to the target task can be obtained, and this fine-tuned model can be used later for recognition related to the target task.

[0094] For example, when the target task is a text classification task, compared with the pre-trained model that has not been fine-tuned by this solution, the fine-tuned model corresponding to text classification obtained by this solution can better handle the text classification task and obtain more accurate text classification results.

[0095] Another example is that when the target task is a text summarization generation task, compared with the pre-trained model that has not been fine-tuned by this solution, the fine-tuned model corresponding to text summarization generation obtained by this solution can better handle the text summarization generation task and can accurately extract the corresponding text summary from the text provided by the user.

[0096] After the model is fine-tuned in three stages, a fine-tuned model corresponding to the target task can be obtained. To verify whether the model meets the standard, the fine-tuned model can be further evaluated through evaluation data. Specifically, the process shown in steps 205 to 206 can be executed.

[0097] Step 205: Obtain the evaluation data corresponding to the parameter-efficient fine-tuning instruction, and based on the parameter-efficient fine-tuning instruction and the evaluation data, evaluate (or can be called infer) the fine-tuned model.

[0098] For example, the evaluation data may include the request text of the target task and the corresponding correct output information. Construct an input field based on the parameter-efficient fine-tuning instruction and the request text, input the input field into the fine-tuned model, and compare the result output by the model with the correct output information to achieve the evaluation of the fine-tuned model.

[0099] Step 206: Determine whether to fine-tune the fine-tuned model again according to the evaluation result.

[0100] After evaluation, if the fine-tuned model does not meet the standard, the fine-tuned model can be fine-tuned again according to the process shown in steps 201 to 204, or the training data can be replaced for fine-tuning again, etc., until a model that meets the requirements (the evaluation meets the standard) is obtained. This model can be used for subsequent target task processing, such as performing text classification, information extraction (or can be called information retrieval), named entity recognition, text summarization generation, semantic similarity calculation, speech control instruction recognition, or machine translation and other task processing through this model.

[0101] Further, to illustrate the implementation process of the embodiments of the present disclosure, the following examples are given, but are not limited thereto:

[0102] For example, usually, for the instruction data required for information extraction (target task) related services, it is difficult for large models (pre-trained on large-scale general datasets) to fully follow and understand instructions. Conventional instruction methods are not sufficient to meet business requirements, so it is necessary to train large models to follow instructions. However, due to the slow generation speed of large models, the required time increases significantly as the text length increases. If complete and rich instructions are adopted, the business latency requirements cannot be met. Therefore, simple instructions tend to be adopted when designing business instructions. Through a large amount of training data, learn the mapping information related to information extraction to meet business requirements. In some examples, the method of parameter-efficient fine-tuning can be used for training, such as Figure 2 the method shown, but there are the following disadvantages:

[0103] Disadvantage 1: Due to the requirement for generation speed, the instructions are generally short. Large models cannot understand the instructions well, so they can only learn through training. This method does not efficiently utilize the understanding ability of large models themselves. The data quality determines the model's effect, so a large amount of high-quality matching data is required. For example, the model lacks the ability of information extraction and intention recognition, there are many information extraction errors in the data, and the model does not understand the instructions well.

[0104] Disadvantage 2: It is difficult to correct the wrong data with ambiguous intentions. For example, in the wrong data, there are many cases of incorrect classification of intention types, and it is necessary to strengthen the learning of the boundary between different categories. The data for preference learning training can be constructed by imitating this type of error to strengthen the learning of category errors.

[0105] Therefore, the embodiments of the present disclosure provide a three-stage fine-tuning method with a preference learning algorithm for the problem that it is not easy for large models after pre-training to extract and understand the key information required for tasks, which significantly improves the information extraction and intention recognition tasks of large models. The overall process is as Figure 4 shown.

[0106] First, according to the product requirements of information extraction (target task), collect the training data for the corresponding task and construct the corresponding evaluation data. And construct more complete full-scale fine-tuning instructions and short parameter-efficient fine-tuning instructions.

[0107] Data construction in the first stage:

[0108] Output fields in the first stage: According to the output information required for the information extraction service, extract from the correct output text of the training data and convert it into a specific dictionary format for easy model recognition;

[0109] The input field in the first stage: It is obtained by combining the full-scale fine-tuning instruction and the request text in the training data, and is converted into a specific dictionary format.

[0110] Among them, the embodiments of the present disclosure construct a more complete full-scale fine-tuning instruction, enabling the large model to output information extraction data strictly according to the above dictionary format.

[0111] Data construction in the second stage:

[0112] Data source: Based on the large model obtained by directly performing parameter-efficient fine-tuning as shown in Figure 2 For the errors caused by blurred category boundaries in the analyzed error data, similar data is constructed as the error output in the output field of the second stage. The construction of this data can strengthen the learning of the problem of blurred category boundaries and correct the classification ability of the model.

[0113] Output field in the second stage: Using the same construction method as the output field in the first stage, error output and correct output are respectively constructed, combined into paired preference data and converted into a specific dictionary format as the output field in the second stage.

[0114] Input field in the second stage: The construction of the full-scale fine-tuning instruction can be the same as that of the full-scale fine-tuning instruction in the input field of the first stage. The input field in the second stage is obtained by combining the full-scale fine-tuning instruction and the request text in the training data, and is converted into a specific dictionary format.

[0115] Data construction in the third stage:

[0116] Output field in the third stage: It is the correct output corresponding to the request text.

[0117] Input field in the third stage: It is obtained by combining the parameter-efficient fine-tuning instruction and the request text. Among them, the parameter-efficient fine-tuning instruction is a short instruction, which is convenient for improving the generation speed of the model's output result.

[0118] After the data construction in the three stages is completed, first, based on the data in the first stage, full-scale fine-tuning training in the first stage is carried out. Then, based on the output model of the first stage and the training data in the second stage, direct preference optimization training in the second stage is carried out. Then, based on the output model of the second stage and the training data in the third stage, parameter-efficient fine-tuning training in the third stage is completed. Through the training in the three stages, not only the understanding ability of the large model for the current task is improved, but also short instructions consistent with the original scheme can be maintained, meeting the business requirements in both generation effect and generation speed, and then a large model with stronger information extraction and intention recognition capabilities is obtained. Finally, after reasoning with the evaluation data, if it is determined that the large model passes the evaluation, the large model can be better used for the execution of information extraction tasks, that is, in terms of task extraction, the fine-tuned large model can show better processing performance and obtain more accurate information extraction results.

[0119] The embodiments of the present disclosure can improve the understanding ability of the large model, and it can be verified that it can be optimized through complete instruction data learning and preference learning algorithm training. Through fine-tuning, the ability of the large model to follow relevant instructions can be improved, helping the model to achieve the text processing ability required by the business. Among them, through preference learning training, the ability to distinguish intention ambiguity errors is improved, and multiple data with incorrect intention discrimination are effectively corrected. Through full-scale fine-tuning training, the information extraction and intention recognition capabilities are significantly improved. Based on only a data volume of no more than 5000, compared with directly performing parameter-efficient fine-tuning, it has increased by 4 points, significantly improving the understanding ability of the large model, and meeting the business requirements in both generation speed and effect.

[0120] Figure 5 is a block diagram of a model fine-tuning device shown according to some embodiments of the present disclosure. Refer to Figure 5 , the device includes: an acquisition module 31 and a fine-tuning module 32.

[0121] The acquisition module 31 is configured to acquire the first training data and the second training data of the target task;

[0122] The fine-tuning module 32 is configured to perform full-scale fine-tuning on the pre-trained model based on the first training data to obtain a fully fine-tuned model; perform direct preference optimization training on the fully fine-tuned model based on the second training data to obtain a direct preference optimization model; and determine a fine-tuned model corresponding to the target task according to the direct preference optimization model.

[0123] In some embodiments of the present disclosure, the acquisition module 31 is specifically configured to acquire different request texts and correct output information corresponding to the request texts; combine the constructed full-scale fine-tuning instructions with the request texts to obtain a first input field, and use the correct output information as a first output field corresponding to the first input field; construct the first training data based on the first input field and the first output field.

[0124] In some embodiments of the present disclosure, the acquisition module 31 is specifically configured to acquire different request texts, correct output information, and incorrect output information corresponding to the request texts; combine the constructed full-scale fine-tuning instructions with the request texts to obtain a second input field, and combine the correct output information and the incorrect output information into paired preference data as a second output field corresponding to the second input field; construct the second training data based on the second input field and the second output field.

[0125] In some embodiments of the present disclosure, the acquisition module 31 is further configured to directly perform parameter-efficient fine-tuning on the pre-trained model without fine-tuning according to the target task; use the pre-trained model after parameter-efficient fine-tuning for testing; perform rule analysis based on the incorrect output information and correct output information in the test results and in combination with the corresponding test request texts; construct different request texts, and correct output information and / or incorrect output information corresponding to the request texts according to the rule analysis results.

[0126] In some embodiments of the present disclosure, the fine-tuning module 32 is specifically configured to acquire third training data for the target task; perform parameter-efficient fine-tuning on the direct preference optimization model based on the third training data to obtain a fine-tuned model corresponding to the target task.

[0127] In some embodiments of the present disclosure, the acquisition module 31 is specifically configured to acquire different request texts and correct output information corresponding to the request texts; combine the constructed parameter-efficient fine-tuning instructions with the request texts to obtain a third input field, and use the correct output information as a third output field corresponding to the third input field; construct the third training data based on the third input field and the third output field.

[0128] In some embodiments of the present disclosure, the fine-tuning module 32 is further configured to acquire evaluation data corresponding to the parameter-efficient fine-tuning instructions; evaluate the fine-tuned model based on the parameter-efficient fine-tuning instructions and the evaluation data; determine whether to perform re-fine-tuning on the fine-tuned model according to the evaluation results.

[0129] In some embodiments of the present disclosure, the data volume of the first training data is less than that of the third training data, and the data volume of the second training data is less than that of the third training data.

[0130] It should be noted that for other corresponding descriptions of each functional unit involved in a model fine-tuning device provided in the embodiments of the present disclosure, reference can be made to Figures 1 to 4 the corresponding description in, which will not be elaborated herein.

[0131] Based on the method as described above Figures 1 to 4 shown, correspondingly, the embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method as described above Figures 1 to 4 shown is implemented.

[0132] Based on such an understanding, the technical solution of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods in various implementation scenarios of the present disclosure.

[0133] Based on the method as described above Figures 1 to 4 shown, and Figure 5 the virtual device embodiments shown, for the purpose of achieving the above object, the embodiments of the present disclosure further provide an electronic device, such as a terminal or a server, etc., and the device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to implement the method as described above Figures 1 to 4 shown.

[0134] Optionally, the above-mentioned physical device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, sensors, an audio circuit, a WI-FI module, etc. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), etc., and optionally the user interface may further include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0135] Those skilled in the art can understand that the above-mentioned physical device structure provided in the embodiments of the present disclosure does not constitute a limitation to the physical device, and may include more or fewer components, or combine some components, or have different component arrangements.

[0136] The storage medium may further include an operating system and a network communication module. The operating system is a program for managing the hardware and software resources of the above-mentioned physical devices, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to implement communication between components inside the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0137] Based on the method as described above Figures 1 to 4 shown, and Figure 5 the virtual device embodiment shown, the embodiments of the present disclosure further provide a chip, including one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from the memory of the electronic device and send the signal to the processor, and the signal includes computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is caused to execute the method as described above Figures 1 to 4 shown.

[0138] Through the description of the above embodiments, those skilled in the art can clearly understand that the present disclosure can be implemented by means of software plus a necessary general hardware platform, or can also be implemented by hardware. For the pre-trained model pre-trained on a large-scale general data set, the embodiments of the present disclosure can effectively fine-tune it. The fine-tuned model obtained by applying the technical solution of the embodiments of the present disclosure can improve the processing performance in a specific target task and can improve the processing accuracy of the model in a specific target task.

[0139] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the element.

[0140] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather to the broadest scope consistent with the principles and novel features claimed herein.

Claims

1. A model fine-tuning method, characterized in that, Including: Obtain the first training data and the second training data of the target task; Perform full-scale fine-tuning on the pre-trained model based on the first training data to obtain a full-scale fine-tuned model; Perform direct preference optimization training on the full-scale fine-tuned model based on the second training data to obtain a direct preference optimization model; Determine a fine-tuned model corresponding to the target task according to the direct preference optimization model.

2. The method according to claim 1, wherein Obtain the first training data of the target task, including: Obtain different request texts and correct output information corresponding to the request texts; Combine the constructed full-scale fine-tuning instruction with the request text to obtain a first input field, and use the correct output information as a first output field corresponding to the first input field; Construct the first training data based on the first input field and the first output field.

3. The method according to claim 1, characterized in that, Obtain the second training data of the target task, including: Obtain different request texts and correct output information and incorrect output information corresponding to the request texts; Combine the constructed full-scale fine-tuning instruction with the request text to obtain a second input field, and combine the correct output information and the incorrect output information into paired preference data as a second output field corresponding to the second input field; Construct the second training data based on the second input field and the second output field.

4. The method according to claim 1 or 2, characterized in that, The method further includes: Directly perform parameter-efficient fine-tuning on the pre-trained model that has not been fine-tuned according to the target task; Use the pre-trained model after parameter-efficient fine-tuning for testing; Perform rule analysis based on the incorrect output information and correct output information in the test results and in combination with the corresponding test request text; Construct different request texts according to the rule analysis results, and correct output information and / or incorrect output information corresponding to the request texts.

5. The method according to claim 1, wherein Determine a fine-tuned model corresponding to the target task according to the direct preference optimization model, including: Obtain the third training data of the target task; Perform parameter-efficient fine-tuning on the direct preference optimization model based on the third training data to obtain a fine-tuned model corresponding to the target task.

6. The method according to claim 5, characterized in that, Obtain the third training data of the target task, including: Obtain different request texts and correct output information corresponding to the request texts; Combine the constructed parameter-efficient fine-tuning instruction with the request text to obtain a third input field, and use the correct output information as a third output field corresponding to the third input field; Construct the third training data based on the third input field and the third output field.

7. The method according to claim 6, wherein After performing parameter-efficient fine-tuning on the direct preference optimization model based on the third training data to obtain a fine-tuned model corresponding to the target task, the method further includes: Obtain evaluation data corresponding to the parameter-efficient fine-tuning instruction; Evaluate the fine-tuned model based on the parameter-efficient fine-tuning instruction and the evaluation data; Determine whether to perform re-fine-tuning on the fine-tuned model according to the evaluation result.

8. The method according to any one of claims 5 to 7, characterized in that The data volume of the first training data is less than the data volume of the third training data, and the data volume of the second training data is less than the data volume of the third training data.

9. A model fine-tuning device, characterized in that, Including: An acquisition module, configured to acquire first training data and second training data of a target task; A fine-tuning module, configured to perform full-scale fine-tuning on a pre-trained model based on the first training data to obtain a full-scale fine-tuned model; Perform direct preference optimization training on the full-scale fine-tuned model based on the second training data to obtain a direct preference optimization model; determine a fine-tuned model corresponding to the target task according to the direct preference optimization model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

11. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images