Method and system for enhancing domain specific capability of large language model based on preference learning

Abstract

The invention provides a method and system for enhancing the domain specific capability of a large language model based on preference learning, and the method comprises the steps: obtaining unlabeled data, carrying out the reasoning of the unlabeled data through a preset domain expert model, and generating corresponding output distribution; inputting the output distribution into a large language model, and generating preference enhancement data based on the output distribution through the large language model; summarizing the preference enhancement data, arranging the preference enhancement data into a new training data set, and performing fine adjustment on the large language model through the new training data set; and performing model evaluation on the fine-tuned large language model, and performing secondary adjustment on the large language model based on an evaluation result to complete field specific capability enhancement of the large language model. The problem that text generation of an existing large language model in a specific field is inaccurate is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of model preference enhancement, and in particular to a method and system for enhancing domain-specific capabilities of a large language model based on preference learning. Background Art

[0002] Large language models have strong general capabilities. However, in applications in specific fields, large language models usually lack sufficient domain-specific capabilities, resulting in the generated text being inaccurate and not relevant enough. Existing methods usually enhance the performance of the model by adding domain-specific training data. However, domain-specific data labels often require the participation of domain experts in labeling, which is costly; and the labeled data may not be open to the holder. There are also methods that transfer the domain knowledge of domain-specific models to large language models through knowledge distillation. In addition, in order to further improve the distillation effect, there are methods that first use a large language model to generate explanations for the specified labels and add the explanations to the training process together with the original data. However, this method cannot fully learn the complete and comprehensive knowledge of the domain expert model. Summary of the invention

[0003] The present invention provides a method and system for enhancing the domain-specific capabilities of a large language model based on preference learning, so as to solve the problem that the existing large language model generates inaccurate text in a specific domain.

[0004] The present invention provides a method for enhancing the domain-specific capabilities of a large language model based on preference learning, comprising: Obtain unlabeled data, and perform reasoning on the unlabeled data using a preset domain expert model to generate a corresponding output distribution; Inputting the output distribution into a large language model, and generating preference enhancement data based on the output distribution through the large language model; Summarizing the preference enhancement data and organizing them into a new training data set, and fine-tuning the large language model through the new training data set; The fine-tuned large language model is evaluated, and the large language model is adjusted again based on the evaluation results to enhance the domain-specific capabilities of the large language model.

[0005] According to a method for enhancing the domain-specific capabilities of a large language model based on preference learning provided by the present invention, the unlabeled data is obtained, and the unlabeled data is inferred by a preset domain expert model to generate a corresponding output distribution, which specifically includes: Obtain unlabeled data and preprocess the data in a unified format; Input the preprocessed unlabeled data into the preset domain expert model.

[0006] The domain expert model generates an output distribution that implies a comprehensive understanding of the input data and relevant knowledge.

[0007] According to a method for enhancing domain-specific capabilities of a large language model based on preference learning provided by the present invention, the output distribution includes multiple output values ​​and their corresponding cumulative probabilities.

[0008] According to a method for enhancing the domain-specific capabilities of a large language model based on preference learning provided by the present invention, the output distribution is input into the large language model, and preference enhancement data is generated based on the output distribution by the large language model, specifically comprising: The large language model compares the cumulative probabilities of the output distributions of the inputs; Constructing a text prompt for generating a preferred interpretation within the large language model, and selecting a maximum cumulative probability of an output distribution based on the text prompt; The maximum cumulative probability of the selected output distribution is used as the preference enhancement data output by the large language model.

[0009] According to a method for enhancing the domain-specific capabilities of a large language model based on preference learning provided by the present invention, the preference enhancement data is summarized and organized into a new training data set, and the large language model is fine-tuned through the new training data set, specifically including: Summarize multiple preference enhancement data and organize them into a new training data set; The large language model is trained multiple times using new training data sets, so that it can better understand expert knowledge and complete fine-tuning of the large language model.

[0010] According to a method for enhancing the domain-specific capability of a large language model based on preference learning provided by the present invention, the fine-tuned large language model is evaluated, and the large language model is adjusted again based on the evaluation result to enhance the domain-specific capability of the large language model, which specifically includes: Evaluate on the fine-tuned large language model and use domain-specific evaluation datasets to verify the performance of the large language model; Based on the evaluated accuracy, the large language model is adjusted a second time to enhance its domain-specific capabilities.

[0011] The present invention also provides a system for enhancing the domain-specific capabilities of a large language model based on preference learning, the system comprising: An expert reasoning module is used to obtain unlabeled data, reason on the unlabeled data through a preset domain expert model, and generate a corresponding output distribution; A preference enhancement module, configured to input the output distribution into a large language model, and generate preference enhancement data based on the output distribution through the large language model; A model fine-tuning module, used to summarize the preference enhancement data, organize it into a new training data set, and fine-tune the large language model through the new training data set; The model evaluation module is used to evaluate the fine-tuned large language model, make secondary adjustments to the large language model based on the evaluation results, and enhance the domain-specific capabilities of the large language model.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for enhancing the domain-specific capabilities of a large language model based on preference learning as described in any one of the above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for enhancing the domain-specific capabilities of a large language model based on preference learning.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above methods for enhancing the domain-specific capabilities of a large language model based on preference learning.

[0015] The present invention provides a method and system for enhancing the domain-specific capabilities of a large language model based on preference learning. It does not require domain experts to annotate unlabeled data, thereby reducing labor costs. When generating learning data, an output distribution of the domain expert model for each input data is generated. The large language model interprets the output distribution of the domain expert model to generate preference enhanced data. Compared with classic knowledge distillation or knowledge distillation enhancement through generating intermediate steps, this method learns the domain knowledge contained in the domain expert model more comprehensively and completely, and has a more obvious effect on improving the domain-specific capabilities of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flowchart of the method for enhancing the domain-specific capabilities of a large language model based on preference learning provided by the present invention.

[0018] Figure 2 It is a schematic diagram of module connections of a system for enhancing the domain-specific capabilities of a large language model based on preference learning provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention.

[0020] Reference numerals: 110: expert reasoning module; 120: preference enhancement module; 130: model fine-tuning module; 140: model evaluation module; 310: processor; 320: communication interface; 330: memory; 340: communication bus. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Combine the following Figure 1 A method of the present invention for enhancing domain-specific capabilities of a large language model based on preference learning is described, comprising: Step 100: Obtain unlabeled data, and use a preset domain expert model to infer the unlabeled data to generate a corresponding output distribution; In the present invention, without the need for manual annotation of data, the output of the domain expert model is used to train the large language model, thereby improving its performance in a specific domain.

[0023] Specifically, unlabeled data is obtained and preprocessed to a unified data format; the preprocessed unlabeled data is input into a preset domain expert model.

[0024] The domain expert model generates an output distribution that implies a comprehensive understanding of the input data and relevant knowledge.

[0025] Collect a large amount of unlabeled data from a specific field. This unlabeled data does not need to be manually labeled and can be textbooks, papers, or datasets in any form of problems in the field.

[0026] Use the domain expert model to infer the collected unlabeled data. The expert model is the top model in a specific field and has deep domain knowledge. For each input data, the expert model generates a corresponding output distribution, which implies the expert model's comprehensive understanding of the input data and related knowledge. The number of samples of the output distribution should be related to the strength of the large language model, but the value is relatively robust, and 2-4 can be used.

[0027] In the present invention, the expert model is used Represented by Q as input, A as output, and P as cumulative probability, beam search can be used to allow the expert model to generate K outputs for a single Q.

[0028] .

[0029] Where P(A|Q) represents the cumulative probability of output A when the input is Q. The K outputs and their corresponding cumulative probabilities constitute the output distribution.

[0030] Step 200: Input the output distribution into a large language model, and generate preference enhancement data based on the output distribution through the large language model.

[0031] Specifically, the large language model compares the cumulative probability of the output distribution of the input; Constructing a text prompt for generating a preferred interpretation within the large language model, and selecting a maximum cumulative probability of an output distribution based on the text prompt; The maximum cumulative probability of the selected output distribution is used as the preference enhancement data output by the large language model.

[0032] In the present invention, the output distribution of the domain expert model is input into the large language model, and the large language model generates explanations based on the output distribution of the expert model. These explanations are called preference enhancement data. The preference enhancement data represents the understanding of the output of the expert model by the large language model, and contains domain-specific knowledge and preference information.

[0033] In a specific example, assuming that for a given input Q, if the cumulative probability of output A1 given by the expert model is greater than the cumulative probability of A2, then the expert model is considered to believe that A1 is a better response to Q than A2 (i.e., preference information). In actual use, except for the output with the largest cumulative probability, the relative cumulative probabilities of the remaining outputs have little effect on the actual effect, so it is only necessary to determine the output with the largest cumulative probability. That's it.

[0034] .

[0035] When generating explanations, we first construct a text prompt for the target large language model to generate the preference explanation based on the preference information obtained from the output distribution. The text prompt template of the present invention is As follows: "For an input problem { }, the expert model believes that { } is better than { For a better answer, please provide an explanation based on your understanding of the problem and the preferences of the expert model."

[0036] In the text prompt template, The outputs other than the output with the maximum cumulative probability generated by the expert model for the input Q can be one or more. Based on the text prompt template, each time the Q in the unlabeled data and the output distribution obtained by the expert model are filled into the template, and then input to the target large language model. , which can generate a preference explanation .

[0037] .

[0038] Step 300: Summarize the preference enhancement data and organize them into a new training data set, and fine-tune the large language model through the new training data set.

[0039] Specifically, it includes: summarizing multiple preference enhancement data and organizing them into a new training data set; The large language model is trained multiple times using new training data sets, so that it can better understand expert knowledge and complete fine-tuning of the large language model.

[0040] In the present invention, all the generated preference enhancement data are collected and organized into a new training data set for subsequent fine-tuning of the large language model. The preference enhancement data includes: the original data (paired question-answer pairs) and the preference interpretation generated by the large language model for the output distribution of the expert model.

[0041] The preference enhancement data combined with the above formula can be expressed as follows.

[0042] .

[0043] In a specific embodiment, the generated preference enhancement data is used to fine-tune the large language model. During the fine-tuning process, the large language model repeatedly learns the data and continuously adjusts internal parameters to better adapt to the needs of a specific field.

[0044] The fine-tuning process requires configuration of hyperparameters such as learning rate and number of training rounds. The recommended learning rate range is 1e-5 to 5e-5, and the number of training rounds is 3-5 rounds.

[0045] Compared with ordinary fine-tuning, the loss function of fine-tuning has an additional loss for calculating preference data, which is used to help the target large language model better understand expert knowledge. Refer to the following formula.

[0046] .

[0047] in and They represent the estimates of preference data and answer data generated during the training of the large language model. The loss function uniformly adopts the classic cross entropy function.

[0048] Step 400: perform model evaluation on the fine-tuned large language model, and perform secondary adjustments on the large language model based on the evaluation results to enhance the domain-specific capabilities of the large language model.

[0049] Specifically, it includes: evaluating on the fine-tuned large language model and using domain-specific evaluation datasets to verify the performance of the large language model; Based on the evaluated accuracy, the large language model is adjusted a second time to enhance its domain-specific capabilities.

[0050] In the present invention, according to experimental results, the performance of the large language model after secondary adjustment on specific domain tasks should be significantly better than that before adjustment.

[0051] The method for enhancing the domain-specific capabilities of a large language model based on preference learning provided by the present invention does not require domain experts to annotate unlabeled data, thereby reducing labor costs. When generating learning data, an output distribution of the domain expert model for each input data is generated. The large language model interprets the output distribution of the domain expert model to generate preference enhanced data. Compared with classic knowledge distillation or knowledge distillation enhancement methods that generate intermediate steps, this method learns the domain knowledge contained in the domain expert model more comprehensively and completely, and has a more obvious effect on improving the domain-specific capabilities of the large language model.

[0052] refer to Figure 2 The present invention also discloses a system for enhancing the domain-specific capabilities of a large language model based on preference learning, the system comprising: The expert reasoning module 110 is used to obtain unlabeled data, reason on the unlabeled data through a preset domain expert model, and generate a corresponding output distribution; A preference enhancement module 120, configured to input the output distribution into a large language model, and generate preference enhancement data based on the output distribution through the large language model; A model fine-tuning module 130, for summarizing the preference enhancement data, arranging them into a new training data set, and fine-tuning the large language model through the new training data set; The model evaluation module 140 is used to perform model evaluation on the fine-tuned large language model, and perform secondary adjustments on the large language model based on the evaluation results to enhance the domain-specific capabilities of the large language model.

[0053] The process of obtaining unlabeled data and inferring the unlabeled data through a preset domain expert model to generate a corresponding output distribution specifically includes: Obtain unlabeled data and preprocess the data in a unified format; Input the preprocessed unlabeled data into the preset domain expert model.

[0054] The domain expert model generates an output distribution that implies a comprehensive understanding of the input data and relevant knowledge.

[0055] The output distribution includes a plurality of output values ​​and their corresponding cumulative probabilities.

[0056] Inputting the output distribution into a large language model, and generating preference enhancement data based on the output distribution through the large language model, specifically includes: The large language model compares the cumulative probabilities of the output distributions of the inputs; Constructing a text prompt for generating a preferred interpretation within the large language model, and selecting a maximum cumulative probability of an output distribution based on the text prompt; The maximum cumulative probability of the selected output distribution is used as the preference enhancement data output by the large language model.

[0057] The preference enhancement data is aggregated and organized into a new training data set, and the large language model is fine-tuned using the new training data set, specifically including: Summarize multiple preference enhancement data and organize them into a new training data set; The large language model is trained multiple times using new training data sets, so that it can better understand expert knowledge and complete fine-tuning of the large language model.

[0058] We evaluate the fine-tuned large language model and make secondary adjustments to the large language model based on the evaluation results to enhance the domain-specific capabilities of the large language model, including: Evaluate on the fine-tuned large language model and use domain-specific evaluation datasets to verify the performance of the large language model; Based on the evaluated accuracy, the large language model is adjusted a second time to enhance its domain-specific capabilities.

[0059] The system provided by the present invention for enhancing the domain-specific capabilities of a large language model based on preference learning does not require domain experts to annotate unlabeled data, thereby reducing labor costs. When generating learning data, an output distribution of the domain expert model for each input data is generated. The large language model interprets the output distribution of the domain expert model to generate preference enhanced data. Compared with classic knowledge distillation or knowledge distillation enhancement methods that generate intermediate steps, this method learns the domain knowledge contained in the domain expert model more comprehensively and completely, and has a more obvious effect on improving the domain-specific capabilities of the large language model.

[0060] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute a method for enhancing the domain-specific capabilities of a large language model based on preference learning, the method comprising: obtaining unlabeled data, reasoning the unlabeled data through a preset domain expert model, and generating a corresponding output distribution; inputting the output distribution into a large language model, and generating preference enhancement data based on the output distribution through the large language model; summarizing the preference enhancement data and organizing it into a new training data set, and fine-tuning the large language model through the new training data set; performing model evaluation on the fine-tuned large language model, and performing secondary adjustment on the large language model based on the evaluation result, so as to enhance the domain-specific capabilities of the large language model.

[0061] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0062] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a method provided by the above methods for enhancing the domain-specific capabilities of a large language model based on preference learning, the method including: obtaining unlabeled data, reasoning the unlabeled data through a preset domain expert model, and generating a corresponding output distribution; inputting the output distribution into a large language model, and generating preference enhancement data based on the output distribution through the large language model; summarizing the preference enhancement data and organizing it into a new training data set, and fine-tuning the large language model through the new training data set; performing model evaluation on the fine-tuned large language model, and performing secondary adjustments to the large language model based on the evaluation results to complete the enhancement of the domain-specific capabilities of the large language model.

[0063] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute a method for enhancing the domain-specific capabilities of a large language model based on preference learning provided by the above methods, the method comprising: obtaining unlabeled data, reasoning the unlabeled data through a preset domain expert model, and generating a corresponding output distribution; inputting the output distribution into a large language model, and generating preference enhancement data based on the output distribution through the large language model; summarizing the preference enhancement data and organizing it into a new training data set, and fine-tuning the large language model through the new training data set; performing model evaluation on the fine-tuned large language model, and performing secondary adjustments to the large language model based on the evaluation results to complete the enhancement of the domain-specific capabilities of the large language model.

[0064] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for enhancing domain-specific capabilities of large language models based on preference learning, characterized in that: include: Obtain unlabeled data, and perform reasoning on the unlabeled data using a preset domain expert model to generate a corresponding output distribution; Inputting the output distribution into a large language model, and generating preference enhancement data based on the output distribution through the large language model; Summarizing the preference enhancement data and organizing them into a new training data set, and fine-tuning the large language model through the new training data set; The fine-tuned large language model is evaluated, and the large language model is adjusted again based on the evaluation results to enhance the domain-specific capabilities of the large language model.

2. The method for enhancing domain-specific capabilities of a large language model based on preference learning according to claim 1, characterized in that: The step of obtaining unlabeled data and inferring the unlabeled data through a preset domain expert model to generate a corresponding output distribution specifically includes: Obtain unlabeled data and preprocess the data in a unified format; Input the preprocessed unlabeled data into the preset domain expert model. The domain expert model generates an output distribution that implies a comprehensive understanding of the input data and relevant knowledge.

3. The method for enhancing the domain-specific capabilities of a large language model based on preference learning according to claim 2, characterized in that: The output distribution includes a plurality of output values ​​and their corresponding cumulative probabilities.

4. The method for enhancing domain-specific capabilities of a large language model based on preference learning according to claim 1, characterized in that: Inputting the output distribution into a large language model, and generating preference enhancement data based on the output distribution through the large language model, specifically includes: The large language model compares the cumulative probabilities of the output distributions of the inputs; Constructing a text prompt for generating a preferred interpretation within the large language model, and selecting a maximum cumulative probability of an output distribution based on the text prompt; The maximum cumulative probability of the selected output distribution is used as the preference enhancement data output by the large language model.

5. The method for enhancing domain-specific capabilities of a large language model based on preference learning according to claim 1, characterized in that: The preference enhancement data is aggregated and organized into a new training data set, and the large language model is fine-tuned by the new training data set, specifically including: Summarize multiple preference enhancement data and organize them into a new training data set; The large language model is trained multiple times using new training data sets, so that it can better understand expert knowledge and complete fine-tuning of the large language model.

6. The method for enhancing domain-specific capabilities of a large language model based on preference learning according to claim 1, characterized in that: The fine-tuned large language model is evaluated, and the large language model is adjusted again based on the evaluation result to enhance the domain-specific capabilities of the large language model, specifically including: Evaluate on the fine-tuned large language model and use domain-specific evaluation datasets to verify the performance of the large language model; Based on the evaluated accuracy, the large language model is adjusted a second time to enhance its domain-specific capabilities.

7. A system for enhancing domain-specific capabilities of large language models based on preference learning, characterized in that The system comprises: An expert reasoning module is used to obtain unlabeled data, reason on the unlabeled data through a preset domain expert model, and generate a corresponding output distribution; A preference enhancement module, configured to input the output distribution into a large language model, and generate preference enhancement data based on the output distribution through the large language model; A model fine-tuning module, used to summarize the preference enhancement data, organize it into a new training data set, and fine-tune the large language model through the new training data set; The model evaluation module is used to evaluate the fine-tuned large language model, make secondary adjustments to the large language model based on the evaluation results, and enhance the domain-specific capabilities of the large language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for enhancing the domain-specific capabilities of a large language model based on preference learning as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for enhancing the domain-specific capabilities of a large language model based on preference learning as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for enhancing the domain-specific capabilities of a large language model based on preference learning as described in any one of claims 1 to 6 is implemented.

Images