Instruction fine tuning method and device for large language model, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence development, in particular to an instruction fine tuning method, device and equipment of a large language model and a medium, which can be applied to training of an AI doctor family inquiry model, and the instruction fine tuning method of the large language model comprises the following steps: obtaining an initial data set; selecting sample data conforming to preset preferences from the initial data set to construct a preference data set; dividing the initial data set into a plurality of fine tuning data sets; and according to the plurality of fine tuning data sets, performing batch fine tuning on the pre-trained large language model, obtaining a preset number of preference sample data from the preference data set after a preset number of batches of fine tuning, and inserting the preference sample data into the next batch of fine tuning data set. According to the method, the big language model is trained by using the preference sample data in the instruction fine tuning stage, so that the output of the obtained big language model can be accurately aligned with the preference, meanwhile, a reinforcement learning alignment step is omitted, and resource consumption during training of the big language model is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence development technology, and is suitable for artificial intelligence development in AI doctor home consultation scenarios, and in particular to a method, device, equipment and medium for fine-tuning instructions of a large language model. Background Art

[0002] With the emergence of dialogue models such as chatGPT, a large number of open source models have emerged in academia and industry, and the overall ecology is extremely prosperous. At present, the training of such large language models mainly includes three steps, namely large-scale pre-training, supervised instruction fine-tuning, and reinforcement learning alignment. In the instruction fine-tuning stage, by learning the written prompt plus output pairs, the model is able to follow the instructions, and the user's needs are aligned through reinforcement learning. During reinforcement learning, the model is usually first allowed to generate multiple answers for specific prompts, and then multiple answers are labeled. The ultimate learning goal is to make the output probability of the answer that meets the preference higher than the rest of the answers. In the prior art, when training models with high professional requirements such as the AI ​​doctor family consultation model, there is often a gap between the data preference in the instruction fine-tuning stage and the data preference in the reinforcement learning alignment stage, which will cause the output of the trained large language model to not accurately align with the preference. In addition, the reinforcement learning alignment stage also consumes a lot of resources. Summary of the invention

[0003] The present invention provides a method, device, computer equipment and readable storage medium for fine-tuning instructions of a large language model to solve the technical problems in the prior art that the output of the large language model cannot accurately align preferences and the training consumes high resources.

[0004] In a first aspect, a method for fine-tuning instructions of a large language model is provided, and the method for fine-tuning instructions of a large language model comprises:

[0005] Acquire an initial data set, wherein sample data in the initial data set includes prompt data and corresponding output data;

[0006] Selecting the sample data that meets the preset preference from the initial data set to construct a preference data set;

[0007] Dividing the initial dataset into multiple fine-tuning datasets;

[0008] The pre-trained large language model is fine-tuned in batches according to the multiple fine-tuning data sets. After each preset number of batches of fine-tuning, a preset number of preference sample data are obtained from the preference data set, and the preference sample data are inserted into the fine-tuning data set of the next batch until all the fine-tuning data sets are used to fine-tune the pre-trained large language model, which is obtained based on pre-training of a basic large language model type.

[0009] In a second aspect, a device for fine-tuning instructions of a large language model is provided, wherein the device for fine-tuning instructions of an agent large language model comprises:

[0010] A first acquisition module is used to acquire an initial data set, wherein the sample data in the initial data set includes prompt data and corresponding output data;

[0011] A second acquisition module, used for selecting the sample data that meets the preset preference from the initial data set to construct a preference data set;

[0012] A batch module, used to divide the initial data set into multiple fine-tuning data sets;

[0013] A preference training module is used to fine-tune the pre-trained large language model in batches according to the multiple fine-tuning data sets, and after each preset number of batches of fine-tuning, obtain a preset number of preference sample data from the preference data set, and insert the preference sample data into the fine-tuning data set of the next batch, until all the fine-tuning data sets are used to fine-tune the pre-trained large language model, which is obtained based on pre-training of the basic large language model type.

[0014] According to a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for fine-tuning instructions of a large language model when executing the computer program.

[0015] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the instruction fine-tuning method of the above-mentioned large language model are implemented.

[0016] In the scheme implemented by the instruction fine-tuning method, device, computer equipment and readable storage medium of the above-mentioned large language model, in the instruction fine-tuning stage of the large language model, the initial data set is divided into several batches of fine-tuning data sets to fine-tune the pre-trained large language model in batches. After each preset number of batches of fine-tuning, a preset number of preference sample data are obtained from the preference data set, and the preference sample data are inserted into the fine-tuning data set of the next batch; by using the preference sample data to train the large language model in the instruction fine-tuning stage, the output of the large language model can be accurately aligned with the preference, and at the same time, the reinforcement learning alignment step is omitted, thereby reducing the resource consumption when training the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 It is a flowchart of a first implementation of a method for fine-tuning instructions of a large language model provided by an embodiment of the present invention.

[0019] Figure 2 yes Figure 1 A schematic flow chart of a specific implementation of step S20 and step S40.

[0020] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S20 in FIG.

[0021] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S40 in FIG.

[0022] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S40 in FIG.

[0023] Figure 6 4 is a schematic diagram of a device for fine-tuning instructions of a large language model according to an embodiment of the present invention.

[0024] Figure 7 It is a structural diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The instruction fine-tuning method of a large language model provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, Figure 1 FIG. 1 is a flow chart of a first implementation of a method for fine-tuning instructions of a large language model provided by an embodiment of the present invention. Figure 1 As shown, the instruction fine-tuning method of the large language model may specifically include the following steps:

[0027] S10, obtaining an initial data set, wherein the sample data in the initial data set includes prompt data and corresponding output data.

[0028] In the implementation manner of the present application, the initial data set is obtained by collecting public corpus supervision data in combination with the target task, and the initial data set includes multiple prompt data and corresponding output data.

[0029] S20: Select the sample data that meets the preset preference from the initial data set to construct a preference data set.

[0030] The sample data of the preference data set is the data that meets the preference requirements screened out from the initial data set, that is, the output data of the prompt data meets the preference requirements.

[0031] S30: Divide the initial data set into multiple fine-tuning data sets.

[0032] S40. Fine-tune the pre-trained large language model in batches according to the multiple fine-tuning data sets. After each preset number of batches of fine-tuning, obtain a preset number of preference sample data from the preference data set, and insert the preference sample data into the fine-tuning data set of the next batch, until all the fine-tuning data sets are used to fine-tune the pre-trained large language model, which is obtained based on pre-training of a basic large language model type.

[0033] In this step, after each preset number of batches of fine-tuning, the preference sample data is inserted into the fine-tuning data set of the next batch, which can increase the training ratio of the preference sample data. The model trained with the guidance of the preference sample data has a certain reward effect on the relevant data involved in the preference sample data, and a certain inhibitory effect on the irrelevant data not involved in the preference sample data. By inserting the preference data into the fine-tuning data set for multiple rounds to fine-tune the large language model, the output of the obtained large language model can be accurately aligned with the preference, while omitting the reinforcement learning alignment step, reducing the resource consumption when training the large language model.

[0034] In the above steps, the preset number after each preset number of batches of fine-tuning is less than the batch number of the fine-tuning dataset. In practical applications, the preset number and the batch number of the fine-tuning dataset can be set according to the capacity of the initial dataset and the training requirements.

[0035] The pre-trained large language model in this embodiment is determined by pre-training the public basic large language model to a certain extent in combination with the target task, wherein the basic large language model can be BLOOM, etc. The embodiment of the present application does not limit the type of basic large language model selected here. Specifically, in combination with the target task (purpose of the model) of the target large language model to be finally obtained, the public large language model is trained to a certain extent by collecting public corpus data to obtain a pre-trained large language model. For example, if the target large language model to be finally obtained is an AI doctor family consultation model, it is necessary to pre-train the public large language model by collecting public medical question and answer corpus data. Through this pre-training method, the cost of collecting training data and data annotation can be greatly reduced, which also reduces the cost of obtaining a pre-trained large language model and improves the efficiency of obtaining a target large language model.

[0036] In addition, those skilled in the art may further perform reinforcement learning alignment on the large language model that has been fine-tuned in the present embodiment. At this time, since the large language model that has been fine-tuned already has a good alignment preference capability, when performing reinforcement learning alignment, when faced with a situation where the same prompt data has multiple corresponding output data, the output probability of the output data that meets the preference will be higher.

[0037] In a specific implementation, the instruction fine-tuning method of the large language model is applied to the AI ​​doctor's family consultation model, and a real doctor-patient dialogue set is obtained as the initial data set. The prompt data and the corresponding output data in the initial data set are the patient's questions and the doctor's answers, respectively. One prompt data can correspond to multiple output data to form multiple groups of sample data; such as the prompt data "What should I do with a cough?" and the output data "Can I drink cough syrup for coughing?", "Coughing is considered to be caused by a cold", "Is there sputum for coughing?", "How long has the cough lasted?" and "Is the body temperature normal?" can form 5 groups of sample data. One of the functions of the AI ​​doctor's family consultation model is to analyze the patient's symptoms according to the patient's description. The sample data of the output data involving the analysis of the patient's symptoms, such as "Is there sputum for coughing?", "How long has the cough lasted?" and "Is the body temperature normal?", can be marked as preferred sample data. By increasing the training ratio of the above-mentioned preferred sample data, the obtained AI doctor's family consultation model, when faced with the prompt data "What should I do with a cough?", the output data is more inclined to the preferred sample data "Is there sputum for coughing?", "How long has the cough lasted?" and "Is the body temperature normal?" It has a certain inhibitory effect on the preferences that are not included in the sample data, such as "cough can be treated with cough syrup" and "cough may be caused by cold". Therefore, the obtained AI doctor family consultation model can accurately align the preferences of family medical consultation.

[0038] Figure 2 yes Figure 1 A specific implementation flow diagram of step S20 and step S40 is shown in FIG. Figure 2 As shown, in some embodiments, step S20 may include:

[0039] S21. Mark the sample data in the initial data set that meets the preset preference as positive sample data, and mark the sample data in the initial data set that does not meet the preset preference as negative sample data; and construct the preference data set based on the positive sample data.

[0040] In the above steps, the sample data in the initial data set are labeled, and the sample data whose data pairs of prompt data and output data meet the preset preference requirements are marked as positive sample data; and the sample data whose data pairs of prompt data and output data do not meet the preference requirements are marked as negative sample data.

[0041] Step S40 may include:

[0042] S41. When the prompt data of the positive sample data is the same as the prompt data of the negative sample data, obtain positive sample data from the preference data set, whose difference between the output probability of output data and the output probability of output data of the negative sample data is less than a preset value, as preference sample data, to obtain a preset number of preference sample data.

[0043] In the above steps, first, when the prompt data of the positive sample data and the prompt data of the negative sample data are the same under the current model parameters, the output probability a of the output data of the positive sample data and the output probability b of the output data of the negative sample data are calculated by the current model, and the positive sample data whose difference c between a and b is less than the preset value d is obtained. The value range of d is [0, 1], which can be set according to the actual situation and training requirements.

[0044] Figure 3 yes Figure 1 A specific implementation flow diagram of step S20 is shown in FIG. Figure 3 As shown, in some embodiments, step S20 may include:

[0045] S201. Obtain evaluation information of the output data corresponding to the prompt data, and save the prompt data, the output data and the evaluation information to a data pool.

[0046] S202: Aggregate the multiple output data of the same prompt data in the data pool and sort them according to the evaluation information to obtain a sorted data set.

[0047] S203: Combine each of the prompt data in the sorted data set with the corresponding first-ranked output data to obtain the preference data set.

[0048] In the above steps, the evaluation information may include one or more information such as scores and comment words. In a specific embodiment, the prompt data A corresponds to output data a, output data b, and output data c, and the evaluation information is a score on a 10-point scale. The scores of output data a, output data b, and output data c corresponding to the prompt data A are 8, 9, and 4, respectively. In the sorted data set, the sorting of the output data is output data b>output data a>output data c. The prompt data A is combined with the output data b ranked first in the output data to obtain the preference sample data. The preference data set can be obtained by combining multiple prompt data with the output data ranked first in the output data. Specifically, when there are multiple types of evaluation information, weights are set for the multiple types of evaluation information respectively, and the sorted data set is obtained after comprehensive consideration.

[0049] Figure 4 yes Figure 1 A specific implementation flow diagram of step S40 is shown in FIG. Figure 4 As shown, in some implementations, step S40 may include:

[0050] S401, before fine-tuning the pre-trained large language model for the first time, setting model parameters.

[0051] The model parameters set for the first time are set according to the specific training task and training plan.

[0052] S402: After each fine-tuning of the pre-trained large language model, the model parameters of the pre-trained large language model are updated.

[0053] In the above steps, by updating the model parameters, the model can be better adapted to these specific scenarios, improve task performance, save computing resources, enhance model flexibility, improve model generalization ability, and optimize model behavior.

[0054] Figure 5 yes Figure 2 A specific implementation flow diagram of step S40 is shown in FIG. Figure 5 As shown, in some implementations, step S40 may include:

[0055] S403: Determine the loss function L of the fine-tuning data set after inserting the preferred sample data.

[0056] The loss function L=L1*β+L2, wherein L1 is the loss function of the fine-tuning data set when the preference sample data is not inserted, L2 is the loss function of the preference sample data, and β is the weight.

[0057] In the above steps, the loss function L of the fine-tuning dataset after inserting the preferred sample data is determined. The fused dataset can provide a smoother gradient update path during the training process, thereby reducing the oscillation of the model during the training process and easily updating and optimizing the model. In specific applications, the value of β is designed according to the actual situation in order to calculate the most suitable loss function L.

[0058] In some embodiments, in step S40, when obtaining a preset number of preference sample data from the preference data set, when inserting the preference sample data into the next batch of the fine-tuning data set, the number of the preset number of preference sample data is made smaller than the number of sample data of the fine-tuning data set.

[0059] In this embodiment, the amount of obtained preference sample data can be made smaller than the capacity of the fine-tuning dataset. When the pre-trained large model is fine-tuned using the fine-tuning dataset to which the preference sample data has been added, the model can be guided to obtain alignment knowledge faster without increasing computing power.

[0060] In some embodiments, in step S30, when the initial data set is divided into several batches of fine-tuning data sets, the initial data set may be evenly divided into several batches of fine-tuning data sets, or the initial data set may be unevenly divided into several batches of fine-tuning data sets. The specific situation is flexibly applied according to the capacity of the initial data set and the specific requirements of the fine-tuning task.

[0061] The instruction fine-tuning method for a large language model provided in an embodiment of the present invention can be constructed based on artificial intelligence, and relevant data can be acquired and processed based on artificial intelligence technology to achieve unattended instruction fine-tuning of a large language model. Among them, artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0062] In one embodiment, a device for fine-tuning instructions for a large language model is provided, and the device for fine-tuning instructions for a large language model corresponds one-to-one to the method for fine-tuning instructions for a large language model in the above embodiment. Figure 6 As shown, the instruction fine-tuning device of the large language model includes:

[0063] A first acquisition module 101 is used to acquire an initial data set, where sample data in the initial data set includes prompt data and corresponding output data;

[0064] A second acquisition module 102 is used to select the sample data that meets the preset preference from the initial data set to construct a preference data set;

[0065] A batch module 103, used to divide the initial data set into multiple fine-tuning data sets;

[0066] The preference training module 104 is used to fine-tune the pre-trained large language model in batches according to the multiple fine-tuning data sets, and obtain a preset number of preference sample data from the preference data set after each preset number of batches of fine-tuning, and insert the preference sample data into the fine-tuning data set of the next batch, until all the fine-tuning data sets are used to fine-tune the pre-trained large language model, and the pre-trained large language model is obtained based on pre-training of the basic large language model type.

[0067] Specifically, the second acquisition module 102 includes a label acquisition unit, and the preference training module 104 includes a preference acquisition unit.

[0068] A marking acquisition unit is used to mark the sample data in the initial data set that meets the preset preference as positive sample data, and mark the sample data in the initial data set that does not meet the preset preference as negative sample data; and construct the preference data set based on the positive sample data.

[0069] A preference acquisition unit is used to acquire, from the preference data set, positive sample data whose difference between the output probability of output data and the output probability of output data of the negative sample data is less than a preset value, as preference sample data, so as to obtain a preset number of preference sample data, when the prompt data of the positive sample data is the same as the prompt data of the negative sample data.

[0070] Specifically, the second acquisition module 102 further includes an evaluation unit, a summarization unit, and a sorting unit.

[0071] The evaluation unit is used to obtain evaluation information of the output data corresponding to the prompt data, and save the prompt data, the output data and the evaluation information to a data pool.

[0072] The summarizing unit is used to summarize the multiple output data of the same prompt data in the data pool and sort them according to the evaluation information to obtain a sorted data set.

[0073] A sorting unit is used to combine each of the prompt data in the sorted data set with the corresponding first-ranked output data to obtain the preference data set.

[0074] Specifically, the preference training module 104 further includes a parameter updating unit.

[0075] The parameter updating unit is used to set the model parameters before fine-tuning the pre-trained large language model for the first time; and is also used to update the model parameters of the pre-trained large language model after each fine-tuning of the pre-trained large language model.

[0076] Specifically, the preference training module 104 further includes a loss determination unit.

[0077] The loss determination unit is used to determine the loss function L of the fine-tuning data set after the preference sample data is inserted, and the loss function L=L1*β+L2, wherein L1 is the loss function of the fine-tuning data set when the preference sample data is not inserted, L2 is the loss function of the preference sample data, and β is the weight.

[0078] Specifically, when the preference training module 104 is used to obtain a preset number of preference sample data from the preference data set and insert the preference sample data into the fine-tuning data set of the next batch, the number of the preset number of preference sample data is smaller than the number of sample data of the fine-tuning data set.

[0079] Specifically, when dividing the initial data set into several batches of fine-tuning data sets, the batching module 103 may divide the initial data set into several batches of fine-tuning data sets evenly or unequally.

[0080] For the specific definition of the instruction fine-tuning device for a large language model, please refer to the definition of the instruction fine-tuning method for a large language model mentioned above, which will not be repeated here. Each module in the above-mentioned instruction fine-tuning device for a large language model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0081] Figure 7 is a schematic diagram of the structure of an embodiment of a computer device according to the present application, such as Figure 7 As shown, the present application also provides a computer device, including:

[0082] A memory and a processor, wherein computer-readable instructions are stored in the processor, and when the computer-readable instructions are executed by the processor, the processor executes any step of the instruction fine-tuning method for the large language model.

[0083] The present application also provides a computer-readable storage medium. When the computer-readable instructions are executed by one or more processors, the one or more processors execute any step in the instruction fine-tuning method of the large language model. It can be understood that the readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0084] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for fine-tuning instructions for a large language model, characterized in that: include: Acquire an initial data set, wherein sample data in the initial data set includes prompt data and corresponding output data; Selecting the sample data that meets the preset preference from the initial data set to construct a preference data set; Dividing the initial dataset into multiple fine-tuning datasets; The pre-trained large language model is fine-tuned in batches according to the multiple fine-tuning data sets. After each preset number of batches of fine-tuning, a preset number of preference sample data are obtained from the preference data set, and the preference sample data are inserted into the fine-tuning data set of the next batch until all the fine-tuning data sets are used to fine-tune the pre-trained large language model, which is obtained based on pre-training of a basic large language model type.

2. The instruction fine-tuning method for a large language model as claimed in claim 1, characterized in that: The selecting the sample data that meets the preset preference from the initial data set to construct the preference data set includes: Marking the sample data in the initial data set that meets the preset preference as positive sample data, and marking the sample data in the initial data set that does not meet the preset preference as negative sample data; constructing the preference data set based on the positive sample data; The acquiring a preset number of preference sample data from the preference data set includes: When the prompt data of the positive sample data is the same as the prompt data of the negative sample data, positive sample data whose difference between the output probability of output data and the output probability of output data of the negative sample data is less than a preset value is obtained from the preference data set as preference sample data to obtain a preset number of preference sample data.

3. The instruction fine-tuning method for a large language model as claimed in claim 1, characterized in that: The selecting sample data that meets the preset preference from the initial data set to construct the preference data set includes: Acquire evaluation information of the output data corresponding to the prompt data, and save the prompt data, the output data and the evaluation information to a data pool; Aggregating a plurality of output data of the same prompt data in the data pool and sorting them according to the evaluation information to obtain a sorted data set; Each of the prompt data in the sorted data set is combined with the corresponding first-ranked output data to obtain the preference data set.

4. The instruction fine-tuning method of a large language model as claimed in claim 1, characterized in that: Also includes: Before fine-tuning the pre-trained large language model for the first time, setting model parameters; After each fine-tuning of the pre-trained large language model, the model parameters of the pre-trained large language model are updated.

5. The instruction fine-tuning method for a large language model as claimed in claim 1, characterized in that: After inserting the preference sample data into the next batch of the fine-tuning data set, the method further includes: Determine the loss function L of the fine-tuning data set after inserting the preference sample data, the loss function L=L1*β+L2, wherein L1 is the loss function of the fine-tuning data set when the preference sample data is not inserted, L2 is the loss function of the preference sample data, and β is the weight.

6. The instruction fine-tuning method of a large language model as claimed in claim 1, characterized in that: Also includes: When obtaining a preset number of preference sample data from the preference data set and inserting the preference sample data into the next batch of the fine-tuning data set, the number of the preset number of preference sample data is made smaller than the number of sample data of the fine-tuning data set.

7. The instruction fine-tuning method for a large language model as claimed in claim 1, characterized in that: The step of dividing the initial data set into a plurality of fine-tuning data sets comprises: Divide the initial dataset evenly into multiple fine-tuning datasets, or The initial dataset is unequally divided into a plurality of fine-tuning datasets.

8. A device for fine-tuning instructions for a large language model, characterized in that: include: A first acquisition module is used to acquire an initial data set, wherein the sample data in the initial data set includes prompt data and corresponding output data; A second acquisition module, used for selecting the sample data that meets the preset preference from the initial data set to construct a preference data set; A batch module, used to divide the initial data set into multiple fine-tuning data sets; A preference training module is used to fine-tune the pre-trained large language model in batches according to the multiple fine-tuning data sets, and after each preset number of batches of fine-tuning, obtain a preset number of preference sample data from the preference data set, and insert the preference sample data into the fine-tuning data set of the next batch, until all the fine-tuning data sets are used to fine-tune the pre-trained large language model, which is obtained based on pre-training of the basic large language model type.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the instruction fine-tuning method for a large language model as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the instruction fine-tuning method for a large language model as claimed in any one of claims 1 to 7 are implemented.

Images