Fine adjustment method for LLM model and related equipment

Abstract

The invention provides a fine tuning method for an LLM model and related equipment. The fine tuning aiming at the LLM model comprises a plurality of fine tuning stages which are sequentially carried out, and the fine tuning stages comprise fine tuning stages based on reinforcement learning. The method comprises the following steps: acquiring a fine tuning data set, wherein the fine tuning data set comprises first fine tuning data in various formats corresponding to a plurality of fine tuning stages; based on a preset target format, performing format conversion processing on the first fine tuning data of the multiple formats to obtain second fine tuning data of the target format corresponding to the first fine tuning data; wherein the target format is a format of first fine tuning data corresponding to the fine tuning stage based on reinforcement learning; and based on the second fine tuning data in the target format, further performing fine tuning training on the LLM model after pre-training is completed.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of large model technology, and more particularly, to a fine-tuning method for an LLM model and related equipment. Background Art

[0002] Large Language Models (LLM), or pre-trained models, are language models built with deep neural networks containing more than tens of billions of parameters. They use self-supervised learning methods to master many language phenomena through pre-training on large and diverse public datasets.

[0003] Furthermore, in order to enable the LLM model to complete specific downstream tasks, the pre-trained LLM model needs to be fine-tuned. Generally speaking, fine-tuning of the LLM model is often carried out in stages, usually including two fine-tuning stages: supervised fine-tuning (SFT) stage based on supervised learning and fine-tuning stage based on reinforcement learning (such as reinforcement learning with human feedback (RLHF) or direct preference optimization (DPO)). First, in the SFT stage, the LLM model will be fine-tuned based on labeled data, which helps to improve the performance of the LLM model in a specific field, but it is easy to make the LLM model lose its generalization ability, and the LLM model is not enough to fully capture subtle human preferences. Furthermore, in the fine-tuning stage based on reinforcement learning, the LLM model will be fine-tuned based on interaction with the environment and human feedback, so that the output of the LLM model conforms to human preferences, but the fine-tuning training process based on reinforcement learning is slow and inefficient.

[0004] As a result, the fine-tuning training process in multiple stages needs to rely on a large amount of fine-tuning data in different formats, and takes a long time and is costly. Summary of the invention

[0005] In view of this, one or more embodiments of the present specification provide a fine-tuning method and related device for an LLM model.

[0006] In a first aspect, the present specification provides a fine-tuning method for an LLM model, wherein the fine-tuning of the LLM model comprises a plurality of fine-tuning stages performed sequentially, wherein the plurality of fine-tuning stages include a fine-tuning stage based on reinforcement learning; the method comprises:

[0007] Acquire a fine-tuning data set, wherein the fine-tuning data set includes first fine-tuning data in multiple formats corresponding to the multiple fine-tuning stages;

[0008] Based on a preset target format, format conversion processing is performed on the first fine-tuning data in the multiple formats respectively to obtain second fine-tuning data in a target format corresponding to the first fine-tuning data; wherein the target format is the format of the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning;

[0009] Based on the second fine-tuning data in the target format, further fine-tuning training is performed on the pre-trained LLM model.

[0010] In a second aspect, the present specification provides a fine-tuning device for an LLM model, wherein the fine-tuning of the LLM model includes a plurality of fine-tuning stages performed sequentially, wherein the plurality of fine-tuning stages include a fine-tuning stage based on reinforcement learning; the device includes:

[0011] An acquiring unit, configured to acquire a fine-tuning data set, wherein the fine-tuning data set includes first fine-tuning data in multiple formats corresponding to the multiple fine-tuning stages;

[0012] a format conversion unit, configured to perform format conversion processing on the first fine-tuning data in the plurality of formats respectively based on a preset target format, so as to obtain second fine-tuning data in a target format corresponding to the first fine-tuning data; wherein the target format is a format of the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning;

[0013] A fine-tuning unit is used to further perform fine-tuning training on the pre-trained LLM model based on the second fine-tuning data in the target format.

[0014] Accordingly, this specification also provides a computing device, comprising: a memory and a processor; the memory stores a computer program / instruction that can be executed by the processor; when the processor executes the computer program / instruction, it executes the fine-tuning method for the LLM model described in the first aspect above.

[0015] Accordingly, the present specification also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the fine-tuning method for the LLM model as described in the first aspect above is executed.

[0016] Accordingly, the present specification also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the fine-tuning method for the LLM model as described in the first aspect above is performed.

[0017] In summary, after obtaining fine-tuning data in multiple formats corresponding to multiple fine-tuning stages of the LLM model, the present application can perform format conversion processing on the fine-tuning data in multiple formats respectively, so as to unify the fine-tuning data in each format into the format of the fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning in multiple fine-tuning stages. Subsequently, these fine-tuning data in a unified format can be used to perform unified and comprehensive fine-tuning training on the LLM model, which is equivalent to unifying the fine-tuning of multiple stages into one stage of fine-tuning, simplifying the fine-tuning process, and greatly improving the fine-tuning efficiency of the LLM model. Moreover, fine-tuning data in all formats can be applied to the fine-tuning method of the present application after format unification, so the demand for fine-tuning data is reduced, for example, a large amount of label data and a large amount of preference pair data are no longer strictly required, and the two can be mixed and used together, which reduces the acquisition cost of fine-tuning data as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a system architecture provided by an exemplary embodiment;

[0019] Figure 2 is a flowchart of a fine-tuning method for an LLM model provided by an exemplary embodiment;

[0020] Figure 3 It is a structural schematic diagram of a fine-tuning device for an LLM model provided by an exemplary embodiment;

[0021] Figure 4 It is a structural diagram of a computing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0023] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0024] It should be noted that the “plurality” mentioned in this application refers to two or more than two.

[0025] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0026] As mentioned above, after the pre-training of the LLM model is completed, it is often necessary to go through multiple stages of fine-tuning training (such as SFT and RLHF, or SFT and DPO) in sequence to improve the generation quality of the LLM model in a specific field and meet the actual needs of users. It is understandable that each fine-tuning stage often requires the use of fine-tuning data that matches the fine-tuning method of that stage. For example, SFT requires the use of label data, while SLHF requires the use of preference pair data that reflects user preferences and non-preferences, and so on. In this way, multiple fine-tuning in stages not only consumes a lot of time, but also relies on a large amount of fine-tuning data in different formats, resulting in a slow, inefficient and costly fine-tuning process. In addition, the optimization goals of different fine-tuning stages are not unified and have low scalability.

[0027] Based on this, this specification provides a technical solution that can unify the formats of fine-tuning data in multiple formats corresponding to multiple fine-tuning stages, and then use the fine-tuning data in a unified format to perform unified and comprehensive fine-tuning training on the LLM model, thereby greatly improving the fine-tuning efficiency of the LLM model.

[0028] During implementation, a fine-tuning data set is first obtained, which may include first fine-tuning data in multiple formats corresponding to multiple fine-tuning stages of the LLM model. Further, based on a preset target format, the first fine-tuning data in multiple formats are format-converted to obtain second fine-tuning data in a target format corresponding to the first fine-tuning data. The target format may be the format of the first fine-tuning data corresponding to a fine-tuning stage based on reinforcement learning in multiple fine-tuning stages. Further, based on the second fine-tuning data in the target format, fine-tuning training can be further performed on the pre-trained LLM model.

[0029] In the above technical solution, after obtaining fine-tuning data of multiple formats corresponding to multiple fine-tuning stages of the LLM model, the present application can perform format conversion processing on the fine-tuning data of multiple formats respectively, so as to unify the fine-tuning data of each format into the format of the fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning in multiple fine-tuning stages. Subsequently, these fine-tuning data in unified formats can be used to perform unified and comprehensive fine-tuning training on the LLM model, which is equivalent to unifying the fine-tuning of multiple stages into one stage of fine-tuning, simplifying the fine-tuning process, and greatly improving the fine-tuning efficiency of the LLM model. Moreover, fine-tuning data of all formats can be applied to the fine-tuning method of the present application after format unification, so the demand for fine-tuning data is reduced, for example, a large amount of label data and a large amount of preference pair data are no longer strictly required, and the two can be mixed and used together, which reduces the acquisition cost of fine-tuning data as a whole.

[0030] See also Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided by an exemplary embodiment. One or more embodiments provided in this specification may be Figure 1 The system architecture shown in the figure or a similar system architecture is specifically implemented. Figure 1 As shown, the system architecture may include a computing device, in which an LLM model is mounted. The LLM model may be a pre-trained LLM basic model. Fine-tuning the LLM model may include multiple fine-tuning stages performed in sequence, such as Figure 1 The fine-tuning stage A and the fine-tuning stage B are shown, wherein the fine-tuning stage A can be performed before the fine-tuning stage B.

[0031] like Figure 1 As shown, in order to further perform fine-tuning training on the pre-trained LLM model, the computing device may first obtain the fine-tuning data set required for fine-tuning training. In one embodiment shown, the fine-tuning training for the LLM model may be to apply the LLM model to tasks related to the target application scenario, and accordingly, the fine-tuning data set may be a fine-tuning data set related to the target application scenario. Exemplarily, the target application scenario may be a medical scenario, a financial scenario, a legal scenario, etc., which is not specifically limited in this specification.

[0032] The fine-tuning data set may include first fine-tuning data in different formats corresponding to multiple fine-tuning stages, for example, first fine-tuning data in fine-tuning stage A and first fine-tuning data in fine-tuning stage B.

[0033] It should be noted that the present application does not specifically limit the method for obtaining the fine-tuning dataset. In one illustrated embodiment, the computing device may obtain the fine-tuning dataset stored in other remote devices through the network, or may directly obtain the fine-tuning dataset stored locally, etc., and this specification does not specifically limit this.

[0034] In an illustrated embodiment, the fine-tuning stage A may be a fine-tuning stage based on supervised learning, and accordingly, the first fine-tuning data corresponding to the fine-tuning stage of supervised learning may be supervised fine-tuning data (SFT dataset). The supervised fine-tuning data may include given inputs and outputs, such as given question samples and answer samples as labels.

[0035] In an illustrated embodiment, the fine-tuning stage B may be a fine-tuning stage based on reinforcement learning, and accordingly, the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning may be a preference pair data (Pairwise alignment dataset). The preference pair data may include a question sample, and a positive and negative sample pair corresponding to the answer to the question sample, wherein the positive sample in the positive and negative sample pair may be an answer sample that meets the user's preference (i.e., preference data), and the negative sample in the positive and negative sample pair may be an answer sample that does not meet the user's preference (i.e., non-preference data).

[0036] It should be noted that this specification does not specifically limit the specific type of SFT.

[0037] In one illustrated implementation, SFT may specifically be question answering tuning and instruction tuning, etc., and this specification does not specifically limit this.

[0038] It should be noted that this specification does not specifically limit the specific fine-tuning method adopted in the fine-tuning stage based on reinforcement learning.

[0039] In an illustrated embodiment, the fine-tuning method adopted in the fine-tuning stage based on reinforcement learning can be various reinforcement learning methods such as RLHF or DPO, which is not specifically limited in this specification. Among them, RLHF can include traditional proximal policy optimization (PPO).

[0040] Furthermore, if Figure 1As shown, the computing device can perform format conversion processing on the obtained fine-tuning data in multiple formats based on the preset target format, so as to unify the fine-tuning data in multiple formats into the target format, thereby obtaining the second fine-tuning data in the target format corresponding to each first fine-tuning data. For details, please refer to the following Figure 2 The description in the corresponding embodiment will not be repeated here.

[0041] In an illustrated embodiment, the preset target format may be a format of the first fine-tuning data corresponding to any fine-tuning stage in a plurality of fine-tuning stages. Alternatively, in an illustrated embodiment, the target format may also be a format of the first fine-tuning data corresponding to the last fine-tuning stage in a plurality of fine-tuning stages. Alternatively, in an illustrated embodiment, the target format may be a format of the first fine-tuning data corresponding to a fine-tuning stage based on reinforcement learning, and so on, which is not specifically limited in this specification.

[0042] Furthermore, if Figure 1 As shown, the computing device can further perform fine-tuning training on the pre-trained LLM model based on the obtained second fine-tuning data in the target format, thereby obtaining a fine-tuned LLM model (i.e., an LLM service model). It should be understood that the fine-tuned LLM model can be applied to the above-mentioned target application scenario to perform tasks related to the target application scenario, such as medical diagnosis tasks in medical scenarios, etc., and this specification does not specifically limit this.

[0043] In one illustrated embodiment, a user can communicate with the fine-tuned LLM model through an input device provided by the computing device (such as a keyboard or a touch screen, etc.) to consult the LLM model related to the target application scenario and obtain corresponding answers, etc., such as conducting a medical consultation dialogue to obtain medical diagnosis results or medication recommendations, etc. This specification does not specifically limit this.

[0044] As described above, compared with the traditional method of performing multiple fine-tuning in stages, the present application provides a unified alignment optimization (UAO) fine-tuning method, which unifies the formats of fine-tuning data in multiple formats corresponding to the original multiple fine-tuning stages, and then uses the fine-tuning data in a unified format to perform unified and comprehensive fine-tuning training on the LLM model, thereby greatly improving the fine-tuning efficiency of the LLM model, simplifying the fine-tuning steps, and reducing the fine-tuning cost.

[0045] In one illustrated embodiment, Figure 1The computing device shown may be a smart wearable device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a server, or a server cluster consisting of multiple servers, etc., which has the above functions. This specification does not make any specific limitations on this.

[0046] It should be understood that Figure 1 The system architecture shown is for exemplary purposes only. In some possible implementations, the system architecture may further include more devices, such as multiple servers storing fine-tuning data sets, etc., which is not specifically limited in this specification.

[0047] In addition, it should be noted that the "first" and "second" in the "first fine-tuning data" and "second fine-tuning data" described in this specification are only used to distinguish the fine-tuning data before and after the format conversion, and do not limit the order or priority of the fine-tuning data.

[0048] See also Figure 2 , Figure 2 is a flowchart of a fine-tuning method for an LLM model provided by an exemplary embodiment. The method can be applied to Figure 1 The system architecture shown can be applied to Figure 1 The computing devices in the system architecture shown. Figure 2 As shown, the method may specifically include the following steps S201 to S203.

[0049] Step S201 : obtaining a fine-tuning data set, wherein the fine-tuning data set includes first fine-tuning data in multiple formats corresponding to multiple fine-tuning stages of the LLM model.

[0050] First, in order to further perform fine-tuning training on the pre-trained LLM model, the computing device can first obtain the fine-tuning data set required for fine-tuning training. The fine-tuning data set may include first fine-tuning data in multiple formats corresponding to multiple fine-tuning stages of the LLM model. For details, please refer to the above Figure 1 The description of the corresponding embodiments will not be repeated here.

[0051] Step S202: based on a preset target format, format conversion processing is performed on the first fine-tuning data in the multiple formats respectively to obtain second fine-tuning data in a target format corresponding to the first fine-tuning data; wherein the target format is the format of the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning included in the multiple fine-tuning stages.

[0052] Furthermore, the computing device may unify the formats of the first fine-tuning data in multiple formats corresponding to the multiple fine-tuning stages, so as to subsequently perform unified fine-tuning training on the LLM model without performing different fine-tuning in stages.

[0053] In an illustrated embodiment, the computing device may perform format conversion processing on first fine-tuning data of multiple formats respectively based on a preset target format, so as to obtain second fine-tuning data of a target format corresponding to each first fine-tuning data. As described above, the multiple fine-tuning stages of the LLM model may include a fine-tuning stage based on supervised learning and a fine-tuning stage based on reinforcement learning performed in sequence, wherein the preset target format may be a format supported by the fine-tuning stage based on reinforcement learning, that is, the format of the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning, which is equivalent to aligning the fine-tuning based on supervised learning to the fine-tuning based on reinforcement learning. In this way, the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning may not be format-converted, or in other words, the format of the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning remains unchanged after the format conversion processing.

[0054] It should be noted that the core idea of ​​RLHF or DPO is to use the feedback provided by humans to improve learning efficiency and performance. This feedback can be a reward signal (i.e. positive feedback) or a punishment signal (i.e. negative feedback). The LLM model can adjust its behavior according to these signals to better meet human needs and expectations.

[0055] Correspondingly, the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning (such as RLHF or DPO) can be preference pair data, which can include question samples and positive and negative sample pairs corresponding to the answers to the question samples, wherein the positive samples (Pair-Choosen Sample) in the positive and negative sample pairs can be answer samples that meet the user's preferences (i.e., the above-mentioned reward signals), and the negative samples (Pair-Reject Sample) in the positive and negative sample pairs can be answer samples that do not meet the user's preferences (i.e., the above-mentioned penalty signals).

[0056] Similarly, the second fine-tuning data in the target format also includes question samples and positive and negative sample pairs corresponding to the answers to the question samples; wherein the positive samples in the positive and negative sample pairs may include answers that meet the user's preferences, and the negative samples in the positive and negative sample pairs may include answers that do not meet the user's preferences.

[0057] In one illustrated embodiment, as described above, the target format is a format supported by the fine-tuning stage based on reinforcement learning, and therefore, in the second fine-tuning data of the target format corresponding to the above-mentioned preference pair data, the positive sample is the positive sample in the preference pair data (i.e., the answer sample that meets the user's preference), and the negative sample is the negative sample in the preference pair data (i.e., the answer sample that does not meet the user's preference).

[0058] In an illustrated embodiment, the first fine-tuning data corresponding to the fine-tuning stage based on supervised learning may be supervised fine-tuning data (ie, SFT data), which is usually labeled data.

[0059] Exemplarily, the supervised fine-tuning data may include a question sample and an answer sample corresponding to the question sample as a positive label (Positive Labels). Among them, the answer sample as a positive label usually contains the correct or expected output result, and the positive label is usually used to tell the LLM model what kind of output should be generated under a given input (such as a question sample). Correspondingly, in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data, the positive sample can be the answer sample as the positive label in the supervised fine-tuning data, and the negative sample can be composed of a placeholder that does not contain semantic information. This is equivalent to using the positive label in the supervised fine-tuning data as the positive sample in the preference pair data, and leaving the negative sample in the preference pair data vacant, thereby achieving a unified alignment of the formats of the supervised fine-tuning data and the preference pair data.

[0060] Exemplarily, the supervised fine-tuning data may also include question samples and answer samples corresponding to the question samples as negative labels (Negative Labels). Among them, the answer samples as negative labels usually contain output results that do not meet expectations, or wrong output results. Negative labels are usually used to tell the LLM model what kind of output should not be generated under given inputs (such as question samples), or what kind of output should be avoided. Correspondingly, in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data, the positive samples are composed of placeholders that do not contain semantic information, and the negative samples are the answer samples as negative labels in the supervised fine-tuning data. This is equivalent to using the negative labels in the supervised fine-tuning data as negative samples in the preference pair data, and leaving the positive samples in the preference pair data vacant, thereby achieving a unified alignment of the formats of the supervised fine-tuning data and the preference pair data.

[0061] Exemplarily, the supervised fine-tuning data may also include a question sample, and an answer sample as a positive label and an answer sample as a negative label corresponding to the question sample, that is, it includes both positive labels and negative labels. Accordingly, in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data, the positive sample is the answer sample as a positive label in the supervised fine-tuning data, and the negative sample is the answer sample as a negative label in the supervised fine-tuning data.

[0062] Generally speaking, SFT generally uses positive labels to guide the learning process of the LLM model to make its output more in line with expectations. However, in some scenarios, it is also possible to provide negative labels to help the LLM model identify and avoid generating inappropriate or harmful content. For example, in a dialogue system, negative labels can be used to prevent the LLM model from generating offensive or inappropriate responses. Introducing negative labels on the basis of positive labels allows the LLM model to learn the correct patterns and effectively avoid incorrect patterns during fine-tuning training. This manual does not specifically limit this.

[0063] In an illustrated embodiment, the fine-tuning dataset may also include collected negative feedback data (Reject Dataset), such as historical answers with low user evaluation or low scores, and historical answers that users have "rejected", etc. Similar to the answer samples as negative labels, in the second fine-tuning data in the target format corresponding to the negative feedback data, the positive sample is composed of a placeholder that does not contain semantic information, and the negative sample is the negative feedback data, etc. This specification does not specifically limit this.

[0064] Step S203: further performing fine-tuning training on the pre-trained LLM model based on the second fine-tuning data in the target format.

[0065] Furthermore, after completing the format unification of the first fine-tuning data in multiple formats corresponding to multiple fine-tuning stages, fine-tuning training can be further performed on the pre-trained LLM model based on the second fine-tuning data in the target format corresponding to the first fine-tuning data.

[0066] In one illustrated embodiment, considering that each first fine-tuning data contains a different amount of information, correspondingly, the second fine-tuning data in the target format corresponding to each first fine-tuning data will also contain a different amount of information. As described in step S202 above, some positive samples or negative samples in the second fine-tuning data do not even contain any semantic information (i.e., the negative samples are empty). Therefore, when fine-tuning the LLM model, the present application needs to consider the differences between the second fine-tuning data.

[0067] In an illustrated embodiment, the training factors of the positive samples and negative samples contained in the second fine-tuning data in the target format corresponding to the first fine-tuning data can be determined according to the length of each first fine-tuning data, wherein the training factor can be used to represent the weight of the corresponding positive sample / negative sample in the fine-tuning training.

[0068] Furthermore, fine-tuning training may be further performed on the pre-trained LLM model based on the second fine-tuning data in the target format and the training factors of the positive samples and the negative samples included in the second fine-tuning data.

[0069] It should be noted that this specification does not particularly limit the specific implementation method of determining the training factors of the positive samples and negative samples included in the second fine-tuning data.

[0070] In an illustrated embodiment, the target length of the longest first fine-tuning data among multiple formats of first fine-tuning data can be determined first; then, based on the ratio of the length of each first fine-tuning data to the target length, the training factors of the positive samples and negative samples contained in the second fine-tuning data corresponding to each first fine-tuning data can be determined.

[0071] Exemplarily, the first fine-tuning data with the longest length may be an answer sample with a positive label in the supervised fine-tuning data, or an answer sample with a negative label in the supervised fine-tuning data, or a positive sample or a negative sample in the preferred data, etc. This specification does not make any specific limitation on this.

[0072] In an illustrated embodiment, if the supervised fine-tuning data only contains answer samples as positive labels and does not contain answer samples as negative labels, the training factor of the positive samples contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length of the answer samples as positive labels in the supervised fine-tuning data to the above-mentioned target length, and since the negative samples are all placeholders, the training factor of the negative samples is 0.

[0073] Exemplarily, assuming that the first fine-tuning data with the longest length is "Based on your symptoms, the preliminary diagnosis is that it is an early stage of a cold" and the target length is 15, and assuming that the answer sample contained in the above supervised fine-tuning data as a positive label is "It may be a cold" and has a length of 5, then the training factor of the positive sample "It may be a cold" contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length 5 to the target length 15, that is, the training factor of the positive sample "It may be a cold" is equal to 1 / 3.

[0074] Exemplarily, assuming that the first fine-tuning data with the longest length is "Today's weather is cloudy and then sunny", and the target length is 9, and assuming that the answer sample contained in the above-mentioned supervised fine-tuning data as a positive label is "Today's weather forecast is sunny", and the length is also 9, then the training factor of the positive sample "Today's weather forecast is sunny" contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length 9 to the target length 9, that is, the training factor of the positive sample "Today's weather forecast is sunny" is equal to 1.

[0075] In an illustrated embodiment, if the supervised fine-tuning data only contains answer samples as negative labels and does not contain answer samples as positive labels, the training factor of the negative samples contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length of the answer samples as negative labels in the supervised fine-tuning data to the above-mentioned target length, and since the positive samples are all placeholders, the training factor of the positive samples is 0.

[0076] Exemplarily, assuming that the first fine-tuning data with the longest length is "Based on your symptoms, the preliminary diagnosis is that you have an early cold" and the target length is 15, and assuming that the answer sample contained in the above supervised fine-tuning data as a negative label is "I don't know" and has a length of 4, then the training factor of the negative sample "I don't know" contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length 4 to the target length 15, that is, the training factor of the negative sample "I don't know" is equal to 4 / 15.

[0077] In an illustrated embodiment, if the supervised fine-tuning data includes both answer samples as positive labels and answer samples as negative labels, the training factor of the positive sample contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length of the answer sample as the positive label in the supervised fine-tuning data to the above target length, and the training factor of the negative sample is the ratio of the length of the answer sample as the negative label in the supervised fine-tuning data to the above target length.

[0078] Exemplarily, assuming that the first fine-tuning data with the longest length is "Based on your symptoms, the preliminary diagnosis is the early stage of a cold" and the target length is 15, and assuming that the answer sample contained in the above supervised fine-tuning data as a positive label is "Your symptoms may be the early stage of a cold" with a length of 11, and the answer sample contained in the negative label is "I don't know your situation" with a length of 8, then the training factor of the positive sample "Your symptoms may be the early stage of a cold" contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of length 11 to target length 15 (i.e. 11 / 15), and the training factor of the negative sample "I don't know your situation" is the ratio of length 8 to target length 15 (i.e. 8 / 15).

[0079] In one illustrated embodiment, the training factor of the positive sample contained in the second fine-tuning data of the target format corresponding to the preference pair data (equivalent to the training factor of the positive sample in the preference pair data) is the ratio of the length of the positive sample in the preference pair data (i.e., the answer sample that meets the user preference) to the above target length, and the training factor of the negative sample (equivalent to the training factor of the negative sample in the preference pair data) is the ratio of the length of the negative sample in the preference pair fine-tuning data (i.e., the answer sample that does not meet the user preference) to the above target length.

[0080] For example, assume that the first fine-tuning data with the longest length is "Based on your symptoms, it is recommended that you use drug A and go to the hospital for treatment in time", the target length is 22, and assume that the positive sample in the above preference pair data is "It is recommended that you use drug A", the length is 8, and the negative sample is "I don't know", the length is 4. Accordingly, the training factor of the positive sample "It is recommended that you use drug A" is the ratio of length 8 to target length 22 (i.e. 4 / 11), and the training factor of the negative sample "I don't know" is the ratio of length 4 to target length 22 (i.e. 2 / 11).

[0081] In an illustrated embodiment, a mask vector may be set for each positive sample / negative sample in the second fine-tuning data to calculate the training factor of the positive sample / negative sample through the mask vector. In an illustrated embodiment, the length of the mask vector may be the longest target length of all first fine-tuning data. The length of the positive sample / negative sample is the same as the number of elements with a value of 1 contained in the corresponding mask vector. If the positive sample / negative sample is composed of placeholders, all elements in the corresponding mask vector are 0. Furthermore, the training factor of the positive sample / negative sample can be determined based on the ratio of the number of elements with a value of 1 in the mask vector of the positive sample / negative sample to the total number.

[0082] For example, still taking the first fine-tuning data with the longest length as "Based on your symptoms, the preliminary diagnosis is the early stage of a cold" and the target length as 15, the length of the mask vector is 15, that is, it contains 15 elements in total.

[0083] For example, it is still assumed that the positive sample contained in the second fine-tuning data is "Your symptoms may be the early stage of a cold" and its length is 11. The mask vector corresponding to the positive sample contains a total of 11 elements with a value of 1, and the remaining 4 elements have a value of 0, for example, "1111111111110000". Correspondingly, the ratio of the number of elements with a value of 1 to the total number is 11 / 15, so it can be determined that the training factor of the positive sample "Your symptoms may be the early stage of a cold" is equal to 11 / 15.

[0084] For example, assuming that the negative sample contained in the second fine-tuning data is "I don't understand your situation" and has a length of 8, the mask vector corresponding to the negative sample contains a total of 8 elements with a value of 1, and the remaining 7 elements have a value of 0, for example, "111111110000000", and accordingly, the ratio of the number of elements with a value of 1 to the total number is 8 / 15, so it can be determined that the training factor of the negative sample "I don't understand your situation" is equal to 8 / 15.

[0085] In summary, it can be understood that the value of the training factor of the positive sample / negative sample is between 0 and 1 (including 0 and 1).

[0086] In an illustrated embodiment, the calculation of the training factor of the positive sample / negative sample can be shown in the following formula (1):

[0087]

[0088] Among them, γ unified (y) is the training factor of positive samples / negative samples. Specifically, the positive samples (i.e., the preferred data) can be recorded as y w , correspondingly, the training factor of the positive sample can be recorded as γ unified (y w ), negative samples (i.e. non-preferred data) can be recorded as y l , correspondingly, the training factor of negative samples can be recorded as γ unified (y l ). M is the mask vector, M i is the i-th element in the mask vector, and N is the length of the mask vector (or the total number of elements contained), that is, the above target length.

[0089] Alternatively, in some possible implementations, the training factor of the positive sample / negative sample may be directly determined according to the number of elements with a value of 1 in the mask vector of the positive sample / negative sample. For example, the training factor of the positive sample "Your symptoms may be the early stage of a cold" may be directly equal to 11, and the training factor of the negative sample "I don't know your situation" may be directly equal to 8, etc. This specification does not make specific limitations on this.

[0090] Below, we will take the fine-tuning method adopted in the fine-tuning stage based on reinforcement learning as an example, specifically DPO, and use the formula to explain how the fine-tuning method for the LLM model provided in this application aligns SFT and DPO, and reason to prove its feasibility.

[0091] First, the optimization objective of the traditional RLHF, namely maximizing the reward and minimizing the KL divergence of the reference strategy, can be expressed as follows:

[0092]

[0093] Furthermore, DPO simplifies this process and relies on theoretical preference models such as the Bradley-Terry model to measure the alignment between the reward function and the empirical preference data. The loss function L of DPO is DPO (π θ ; π ref ) can be expressed as follows:

[0094]

[0095] Among them, π θis the model under training, π ref is the reference model. In DPO implementation, the reference model π ref It is often set as the initial model π init , which is the pre-trained LLM basic model. The optimization objective shown in formula (3) adds the preference data y w The possibility of and reduce the non-preferred data y l The examples are weighted by their ratings from the implicit reward model, scaled by β.

[0096] Secondly, the Loss of SFT is generally believed to enable the model to adjust parameters by minimizing the cross entropy loss, using only positive labels (i.e., preference data y w )’s SFT loss function L SFT (π θ ) can be expressed as shown in the following formula (4):

[0097]

[0098] Furthermore, the present application takes into account that during the training process of SFT, it is not desirable for the LLM model to lose its general language ability due to overfitting. On the other hand, objectively, since the Loss of SFT and Pair DPO Loss are not in the same order of magnitude (for example, the Loss of SFT is often more than ten or twenty, but the Loss of DPO is only a few tenths), the LLM model can be overfitted. SFT (π θ ) to make corrections, specifically using the initial model π init Make a preference correction on the result of SFT. For example, the loss function L of the modified SFT is SFT_Modify (π θ ) can be expressed as follows:

[0099]

[0100] As shown in formula (5) above, by introducing the initial model π init The response output (that is, the output of the initial model will be subtracted during each training, and the final trained model will not be too different from the initial model) can balance the generalization ability of the LLM model and its adaptability to preferred data, prevent overfitting from causing loss of general language ability, and thus ensure the generation quality of the LLM model.

[0101] Further, as mentioned above, considering that in the DPO implementation, the reference model π ref It is often set as the initial model π init , then the loss function of SFT is L SFT_Modify (π θ) can be further expressed as shown in the following formula (6):

[0102]

[0103] As shown in formula (6), the rewritten SFT loss function L SFT_Modify (π θ ) structure, and the loss function L of DPO shown in the above formula (3) DPO (π θ ; π ref ) in the preference data y w The partial structures of are very similar, and the optimization objectives can be considered consistent. Based on this, combined with the loss function L of DPO shown in formula (3) DPO (π θ ; π ref ), as well as the format conversion processing of the fine-tuning data and the calculation of the training factor described in the above steps S202 and S203, the loss function L of the unified alignment training method provided by the present application can be obtained. Unified (π θ ; π ref ), as shown in the following formula (7):

[0104]

[0105] In summary, the loss function L obtained after optimization in this application is Unified (π θ ; π ref ), which is equivalent to adapting the modified SFT on the basis of the original DPO, and adapting the SFT fine-tuning data after format conversion, so as to realize the unified alignment of the fine-tuning of the two stages of SFT and DPO, with only one unified optimization goal. Compared with the traditional fine-tuning method of fine-tuning in multiple stages, the present application simplifies the fine-tuning process and greatly improves the fine-tuning efficiency of the LLM model. Moreover, fine-tuning data of all formats can be applied to the fine-tuning method of the present application after format unification, so the demand for fine-tuning data is reduced. For example, a large amount of labeled data and a large amount of preference pair data are no longer strictly required. The two can be mixed and used, which reduces the overall cost of obtaining fine-tuning data.

[0106] Corresponding to the implementation of the above method flow, the embodiment of this specification also provides a fine-tuning device for the LLM model. The fine-tuning of the LLM model includes multiple fine-tuning stages performed in sequence, and the multiple fine-tuning stages include a fine-tuning stage based on reinforcement learning. Figure 3 , Figure 3 FIG. 3 is a schematic diagram of a fine-tuning device for an LLM model provided by an exemplary embodiment. The device 30 can be applied to Figure 1 The system architecture shown can be applied to Figure 1 The computing devices in the system architecture shown. Figure 3 As shown, the device 30 comprises:

[0107] An acquiring unit 301 is configured to acquire a fine-tuning data set, wherein the fine-tuning data set includes first fine-tuning data in multiple formats corresponding to the multiple fine-tuning stages;

[0108] A format conversion unit 302 is used to perform format conversion processing on the first fine-tuning data in the multiple formats respectively based on a preset target format to obtain second fine-tuning data in a target format corresponding to the first fine-tuning data; wherein the target format is the format of the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning;

[0109] The fine-tuning unit 303 is used to further perform fine-tuning training on the pre-trained LLM model based on the second fine-tuning data in the target format.

[0110] In an illustrated embodiment, the second fine-tuning data in the target format includes a question sample and a positive-negative sample pair corresponding to the answer to the question sample; the positive sample in the positive-negative sample pair includes an answer that meets the user's preference, and the negative sample in the positive-negative sample pair includes an answer that does not meet the user's preference.

[0111] In an illustrated embodiment, the fine-tuning unit 303 is specifically configured to:

[0112] Determine, according to the length of the first fine-tuning data, the training factors of the positive samples and the negative samples contained in the second fine-tuning data corresponding to the first fine-tuning data; wherein the training factors are used to represent the weights of the positive samples / negative samples corresponding thereto in the fine-tuning training;

[0113] Based on the second fine-tuning data in the target format and the training factors of the positive samples and negative samples contained in the second fine-tuning data, further fine-tuning training is performed on the pre-trained LLM model.

[0114] In an illustrated embodiment, the fine-tuning unit 303 is specifically configured to:

[0115] Determine a target length of the first fine-tuning data having the longest length among the first fine-tuning data in the plurality of formats;

[0116] And according to the ratio of the length of each first fine-tuning data to the target length, the training factors of the positive samples and the negative samples contained in the second fine-tuning data corresponding to each first fine-tuning data are determined.

[0117] In an illustrated embodiment, the multiple fine-tuning stages also include a fine-tuning stage based on supervised learning performed before the fine-tuning stage based on reinforcement learning; the first fine-tuning data in the multiple formats include supervised fine-tuning data corresponding to the fine-tuning stage based on supervised learning, and preference pair data corresponding to the fine-tuning stage based on reinforcement learning.

[0118] In one illustrated embodiment, the supervised fine-tuning data includes a question sample and an answer sample corresponding to the question sample as a positive label; in the second fine-tuning data in a target format corresponding to the supervised fine-tuning data, the positive sample is the answer sample as a positive label in the supervised fine-tuning data, and the negative sample is composed of a placeholder that does not contain semantic information.

[0119] In an illustrated embodiment, the training factor of the positive sample contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length of the answer sample as the positive label in the supervised fine-tuning data to the target length, and the training factor of the negative sample is 0.

[0120] In one illustrated embodiment, the supervised fine-tuning data includes a question sample and an answer sample corresponding to the question sample as a negative label; in the second fine-tuning data in a target format corresponding to the supervised fine-tuning data, the positive sample is composed of a placeholder that does not contain semantic information, and the negative sample is the answer sample as a negative label in the supervised fine-tuning data.

[0121] In an illustrated embodiment, the training factor of the positive sample contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is 0, and the training factor of the negative sample is the ratio of the length of the answer sample as a negative label in the supervised fine-tuning data to the target length.

[0122] In one illustrated embodiment, the supervised fine-tuning data includes a question sample, and an answer sample as a positive label and an answer sample as a negative label corresponding to the question sample; in the second fine-tuning data in a target format corresponding to the supervised fine-tuning data, the positive sample is the answer sample as a positive label in the supervised fine-tuning data, and the negative sample is the answer sample as a negative label in the supervised fine-tuning data.

[0123] In an illustrated embodiment, the training factor of the positive sample contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length of the answer sample as the positive label in the supervised fine-tuning data to the target length, and the training factor of the negative sample is the ratio of the length of the answer sample as the negative label in the supervised fine-tuning data to the target length.

[0124] In one illustrated embodiment, the preference pair data includes a question sample, and an answer sample that meets the user's preference and an answer sample that does not meet the user's preference corresponding to the question sample; in the second fine-tuning data in the target format corresponding to the preference pair data, the positive sample is the answer sample in the preference pair data that meets the user's preference, and the negative sample is the answer sample in the preference pair data that does not meet the user's preference.

[0125] In an illustrated embodiment, the training factor of the positive sample contained in the second fine-tuning data of the target format corresponding to the preference pair data is the ratio of the length of the answer sample in the preference pair data that meets the user preference to the target length, and the training factor of the negative sample is the ratio of the length of the answer sample in the preference pair data that does not meet the user preference to the target length.

[0126] In an illustrated embodiment, the fine-tuning method adopted in the reinforcement learning-based fine-tuning stage includes any one of reinforcement learning based on human feedback (RLHF) and direct preference optimization (DPO).

[0127] In an illustrated embodiment, the fine-tuning dataset includes a fine-tuning dataset related to a medical scenario.

[0128] The implementation process of the functions and effects of each unit in the above-mentioned device 30 is specifically described in the above-mentioned embodiment, and will not be repeated here. It should be understood that the above-mentioned device 30 can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor (CPU) of the device where it is located reading the corresponding computer program instructions into the memory and running them. From the hardware level, in addition to the CPU and memory, the device where the above-mentioned device is located usually also includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards and cards for realizing network communication functions.

[0129] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art may understand and implement it without creative work.

[0130] The devices, units, and modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, a vehicle-mounted computer, or a combination of any of these devices.

[0131] Corresponding to the above method embodiment, the embodiment of this specification also provides a computing device. Figure 4 , Figure 4 It is a structural diagram of a computing device provided by an exemplary embodiment. Figure 4 The computing device shown may be Figure 1 The computing devices in the system architecture shown. Figure 4 As shown, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004, and the output device 1005 may be connected via a bus or other means. Figure 4 As shown, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program that can be run by the processor 1001. The processor 1001 can be a CPU, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiment. When the processor 1001 runs the stored computer program, it can execute the various steps of the fine-tuning method for the LLM model in the embodiment of this specification, including: obtaining a fine-tuning data set, the fine-tuning data set contains first fine-tuning data in multiple formats corresponding to multiple fine-tuning stages of the LLM model; based on a preset target format, the first fine-tuning data in the multiple formats are format-converted to obtain second fine-tuning data in a target format corresponding to the first fine-tuning data; wherein the target format is the format of the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning included in the multiple fine-tuning stages; based on the second fine-tuning data in the target format, further fine-tuning training is performed on the pre-trained LLM model, and so on.

[0132] For a detailed description of each step of the above-mentioned fine-tuning method for the LLM model, please refer to the previous content and will not be repeated here.

[0133] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored. When these computer programs are executed by a processor, they execute the various steps of the fine-tuning method for the LLM model in the embodiment of this specification. Please refer to the description of the above embodiment for details, which will not be repeated here.

[0134] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

[0135] In a typical configuration, a terminal device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0136] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0137] Computer readable media include permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0138] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0139] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0140] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A fine-tuning method for an LLM model, characterized in that: The fine-tuning of the LLM model includes a plurality of fine-tuning stages performed sequentially, wherein the plurality of fine-tuning stages include a fine-tuning stage based on reinforcement learning; the method includes: Acquire a fine-tuning data set, wherein the fine-tuning data set includes first fine-tuning data in multiple formats corresponding to the multiple fine-tuning stages; Based on a preset target format, format conversion processing is performed on the first fine-tuning data in the multiple formats respectively to obtain second fine-tuning data in a target format corresponding to the first fine-tuning data; wherein the target format is the format of the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning; Based on the second fine-tuning data in the target format, further fine-tuning training is performed on the pre-trained LLM model.

2. The method according to claim 1, characterized in that The second fine-tuning data in the target format includes a question sample and a positive-negative sample pair corresponding to the answer to the question sample; the positive sample in the positive-negative sample pair includes an answer that meets the user's preference, and the negative sample in the positive-negative sample pair includes an answer that does not meet the user's preference.

3. The method according to claim 2, characterized in that The further performing fine-tuning training on the pre-trained LLM model based on the second fine-tuning data in the target format includes: Determine, according to the length of the first fine-tuning data, the training factors of the positive samples and the negative samples contained in the second fine-tuning data corresponding to the first fine-tuning data; wherein the training factors are used to represent the weights of the positive samples / negative samples corresponding thereto in the fine-tuning training; Based on the second fine-tuning data in the target format and the training factors of the positive samples and negative samples contained in the second fine-tuning data, further fine-tuning training is performed on the pre-trained LLM model.

4. The method according to claim 3, characterized in that The determining, according to the length of the first fine-tuning data, the training factors of the positive samples and the negative samples contained in the second fine-tuning data corresponding to the first fine-tuning data comprises: Determine a target length of the first fine-tuning data having the longest length among the first fine-tuning data in the plurality of formats; And according to the ratio of the length of each first fine-tuning data to the target length, the training factors of the positive samples and the negative samples contained in the second fine-tuning data corresponding to each first fine-tuning data are determined.

5. The method according to claim 4, characterized in that The multiple fine-tuning stages also include a fine-tuning stage based on supervised learning performed before the fine-tuning stage based on reinforcement learning; the first fine-tuning data in multiple formats include supervised fine-tuning data corresponding to the fine-tuning stage based on supervised learning, and preference pair data corresponding to the fine-tuning stage based on reinforcement learning.

6. The method according to claim 5, characterized in that The supervised fine-tuning data includes a question sample and an answer sample corresponding to the question sample as a positive label; in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data, the positive sample is the answer sample as a positive label in the supervised fine-tuning data, and the negative sample is composed of a placeholder that does not contain semantic information.

7. The method according to claim 6, characterized in that The training factor of the positive sample contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length of the answer sample as the positive label in the supervised fine-tuning data to the target length, and the training factor of the negative sample is 0.

8. The method according to claim 5, characterized in that The supervised fine-tuning data includes a question sample and an answer sample corresponding to the question sample as a negative label; in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data, the positive sample is composed of a placeholder that does not contain semantic information, and the negative sample is the answer sample as a negative label in the supervised fine-tuning data.

9. The method according to claim 8, characterized in that The training factor of the positive sample contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is 0, and the training factor of the negative sample is the ratio of the length of the answer sample as a negative label in the supervised fine-tuning data to the target length.

10. The method according to claim 5, characterized in that The supervised fine-tuning data includes a question sample, and an answer sample as a positive label and an answer sample as a negative label corresponding to the question sample; in the second fine-tuning data in a target format corresponding to the supervised fine-tuning data, the positive sample is the answer sample as a positive label in the supervised fine-tuning data, and the negative sample is the answer sample as a negative label in the supervised fine-tuning data.

11. The method according to claim 10, characterized in that The training factor of the positive sample contained in the second fine-tuning data in the target format corresponding to the supervised fine-tuning data is the ratio of the length of the answer sample as the positive label in the supervised fine-tuning data to the target length, and the training factor of the negative sample is the ratio of the length of the answer sample as the negative label in the supervised fine-tuning data to the target length.

12. The method according to claim 5, characterized in that The preference pair data includes question samples, and answer samples that meet user preferences and answer samples that do not meet user preferences corresponding to the question samples; in the second fine-tuning data in the target format corresponding to the preference pair data, the positive samples are the answer samples in the preference pair data that meet user preferences, and the negative samples are the answer samples in the preference pair data that do not meet user preferences.

13. The method according to claim 12, characterized in that The training factor of the positive sample contained in the second fine-tuning data in the target format corresponding to the preference pair data is the ratio of the length of the answer sample in the preference pair data that meets the user preference to the target length, and the training factor of the negative sample is the ratio of the length of the answer sample in the preference pair data that does not meet the user preference to the target length.

14. The method according to any one of claims 1 to 13, characterized in that: The fine-tuning method adopted in the reinforcement learning-based fine-tuning stage includes any one of reinforcement learning RLHF based on human feedback and direct preference optimization DPO.

15. The method according to any one of claims 1 to 13, characterized in that: The fine-tuning dataset includes a fine-tuning dataset related to medical scenarios.

16. A fine-tuning device for an LLM model, characterized in that: The fine-tuning of the LLM model includes a plurality of fine-tuning stages performed sequentially, wherein the plurality of fine-tuning stages include a fine-tuning stage based on reinforcement learning; the device includes: An acquiring unit, configured to acquire a fine-tuning data set, wherein the fine-tuning data set includes first fine-tuning data in multiple formats corresponding to the multiple fine-tuning stages; a format conversion unit, configured to perform format conversion processing on the first fine-tuning data in the plurality of formats respectively based on a preset target format, so as to obtain second fine-tuning data in a target format corresponding to the first fine-tuning data; wherein the target format is a format of the first fine-tuning data corresponding to the fine-tuning stage based on reinforcement learning; A fine-tuning unit is used to further perform fine-tuning training on the pre-trained LLM model based on the second fine-tuning data in the target format.

17. A computing device, characterized in that include: Memory and processor; The memory has stored thereon computer programs / instructions executable by the processor; When the processor runs the computer program / instructions, it performs the method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that: A computer program / instruction is stored thereon, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

19. A computer program product, characterized in that The computer program product comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images