Training method and device of large language model, electronic equipment and storage medium

Abstract

The application provides a large language model training method and device, electronic equipment and storage medium, and relates to the technical field of artificial intelligence. The large language model training method comprises: obtaining an instruction response data pair; the instruction response data pair is composed of an instruction and a response generated by a large language model according to the instruction, and the response includes text in a preset field; constraint information contained in the response is extracted from the response contained in the instruction response data pair; a training data set is generated according to the response, the constraint information contained in the response and the instruction corresponding to the response; the pre-trained large language model is further trained by using the training data set to obtain a target large language model, so that the target large language model can be used to generate a response according to an instruction. The application can efficiently train a large language model that can accurately execute instructions with complex constraints, thereby effectively improving the utilization rate of computing resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a large language model training method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Large language models (LLMs) have shown excellent performance in natural language processing tasks. In processing complex instructions with multiple constraints, existing methods usually rely on adding these constraints to the instructions and using advanced LLMs such as GPT-4 to generate responses. However, when faced with instructions containing complex constraints such as format, length, etc., these models often perform poorly, affecting their effectiveness and range of application.

[0003] To improve the instruction execution capability of LLMs, a post-training strategy is usually used, i.e., training with a large amount of instruction-response data pairs. Although this method is simple and widely used, the training effect is not good, even the most advanced LLMs still struggle to execute complex instructions, resulting in limited data quality and the need to spend a lot of effort to filter training data.

[0004] Therefore, how to efficiently train a large language model that can accurately execute instructions with complex constraints and effectively improve the utilization of computing resources is a technical problem that needs to be solved. SUMMARY

[0005] To solve the above problems in the prior art, the present application provides a large language model training method, device, electronic equipment and storage medium to efficiently train a large language model that can accurately execute instructions with complex constraints and effectively improve the utilization of computing resources.

[0006] The present application provides a large language model training method, comprising the following steps.

[0007] Obtain an instruction-response data pair; wherein the instruction-response data pair is composed of an instruction and a response generated by a large language model according to the instruction, the response including text in a predetermined field; extract constraint information contained in the response from the response contained in the instruction-response data pair; wherein the constraint information is used to describe the conditions or standards that the response meets; generate a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response; use the training data set to continue training a pre-trained large language model to obtain a target large language model, so as to generate a response according to an instruction using the target large language model.

[0008] According to the method for training a large language model provided by the application, the constraint information comprises a first type of constraint; wherein the first type of constraint is a constraint that can be extracted using a script; and the extracting of the constraint information contained in the response from the instruction-response data pair comprises: using a script implemented by a preset language to extract the constraint contained in the response from the response; and filling the extracted constraint contained in the response into a generated constraint template to obtain a textual representation of the constraint contained in the response.

[0009] According to the method for training a large language model provided by the application, the constraint information comprises a second type of constraint; wherein the second type of constraint is a constraint that can be extracted from a response by a large language model; and the extracting of the constraint information contained in the response from the instruction-response data pair comprises: generating a prompt word according to the response, the instruction corresponding to the response, and a textual description of the second type of constraint; and inputting the prompt word into a large language model to obtain the constraint contained in the response output by the large language model.

[0010] According to the method for training a large language model provided by the application, the constraint information comprises a plurality of constraints; and the generating of a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response comprises: randomly combining the plurality of constraints to obtain a plurality of constraint sets; wherein each constraint set comprises at least one constraint; and the training data set is obtained according to each constraint set, the instruction corresponding to the constraint set, and the response corresponding to the constraint set.

[0011] According to the method for training a large language model provided by the application, the obtaining of the training data set according to each constraint set, the instruction corresponding to the constraint set, and the response corresponding to the constraint set comprises:

[0012] According to the method for training a large language model provided by the application, the obtaining of the training data set according to each constraint set, the instruction corresponding to the constraint set, and the response corresponding to the constraint set comprises:

[0013] According to the method for training a large language model provided in the application, the training data set comprises a first training subset and a second training subset; the training data set is obtained according to each set of constraint sets and corresponding instructions and responses, which comprises: obtaining a set of first sample data in the first training subset according to the corresponding response and instruction of each set of constraint sets, and taking the constraint set as the label of the first sample data; obtaining the first training subset according to all the first sample data; obtaining a set of second sample data in the second training subset according to each set of constraint sets and corresponding instructions, and taking the corresponding response of the constraint set as the label of the second sample data; obtaining the second training subset according to all the second sample data; the pre-trained large language model is further trained by using the training data set, which comprises: jointly training the pre-trained large language model by using the first training subset and a first loss function, and the second training subset and a second loss function; wherein the first loss function is constructed based on the first sample data in the first training subset, and the probability of generating the label of the first sample data by the large language model; and the second loss function is constructed based on the second sample data in the second training subset, and the probability of generating the label of the second sample data by the large language model.

[0014] The application further provides a training device for a large language model, comprising the following modules:

[0015] The acquisition module is configured to acquire instruction-response data pairs, wherein each instruction-response data pair comprises an instruction and a response generated by a large language model based on the instruction, and the response comprises text in a preset field; the extraction module is configured to extract constraint information contained in the response from the instruction-response data pairs; wherein the constraint information is used to describe conditions or standards met by the response; the generation module is configured to generate a training data set based on the response, the constraint information contained in the response, and the instruction corresponding to the response; and the training module is configured to further train a pre-trained large language model by using the training data set to obtain a target large language model, so as to generate a response based on an instruction by using the target large language model.

[0016] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for training a large language model according to any one of the above embodiments when executing the computer program.

[0017] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the method for training a large language model according to any one of the above embodiments.

[0018] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the training method of the large language model according to any one of the above.

[0019] The application provides a training method and device of a large language model, an electronic device and a storage medium. The constraint information is extracted from the response contained in the instruction response data, so the large language model can accurately understand and execute the constraint information, and the training data set constructed by using the constraint information can be used to train the large language model that can accurately execute instructions with complex constraints. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 is one of the flowcharts of the training method of the large language model provided by the application.

[0022] Figure 2 is the second flowchart of the training method of the large language model provided by the application.

[0023] Figure 3 is the third flowchart of the training method of the large language model provided by the application.

[0024] Figure 4 is a schematic diagram of the constraint information contained in the text generated by using the large language model.

[0025] Figure 5 is a schematic diagram of the framework of constructing the training data set provided by the application.

[0026] Figure 6 is a schematic diagram of the examples of the response generated by the large language model based on the instruction carrying the constraint and the response generated by the instruction not carrying the constraint.

[0027] Figure 7 is a structural schematic diagram of the training device of the large language model provided by the application.

[0028] Figure 8 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0030] The inventors found an important fact in the process of experimental research: the existing data set actually contains implicit complex constraints. For example, as shown in Figure 4 , although the original short instruction does not explicitly specify specific requirements such as writing style and length, the response content has met these constraint conditions at multiple levels.

[0031] The inventors further found through case studies that, as shown in Figure 6 , when a large language model generates a response according to an instruction, if the instruction contains inappropriate constraint conditions, the generated response may contain ambiguous terms such as '[database name]', and lack necessary details and depth, thereby causing the content quality of the final response to decrease.

[0032] Therefore, the constraints that can be understood and executed by a large language model can be extracted from the existing high-quality instruction-response data pairs and integrated into the instruction to construct training data containing effective constraints without noise, and a large language model that can efficiently and accurately execute complex instructions is obtained by training using the training data.

[0033] For example, given an instruction x, define a specific task such as 'write a blog about French cuisine'; a constraint set c that specifies the conditions for the response, such as length restrictions; and a response y that meets both the constraint c and the instruction x, the goal is to construct a high-quality (x, c, y) triple data set.

[0034] The training method of the large language model of the present application will be described below. Figures 1-6

[0035] Figure 1 is one of the flowcharts of the training method of the large language model provided by the present application, as shown in Figure 1 , the method comprises the following steps:

[0036] Step 101, obtaining instruction-response data pairs.

[0037] ​The instruction response data pair consists of an instruction and a response generated by the large language model according to the instruction. The response includes text in a preset field. For example, as shown in Figure 4 the instruction is "write a blog about French cuisine", and the response is:

[0038] "Title: The Charm of French Cuisine: A Journey Through Time and Taste

[0039] Content: French cuisine is often synonymous with elegance, refinement, and an unparalleled pursuit of culinary art. Whether in the bustling cafes of Paris or the quiet countryside kitchens of Provence, French cooking is not just a meal, but an artistic expression....".

[0040] In the specific implementation process, high-quality instruction response data pairs can be collected from a data set generated by using a large language model. The quality score of the response is greater than a preset threshold, the response meets multiple constraint conditions, and there is a dependency relationship between the multiple constraint conditions.

[0041] For example only, as shown in Figure 5 , a large number of instruction response data pairs can be collected from four supervised fine-tuning data sets: Alpaca GPT4, OrcaChat, Evol Instruct, and OpenAssistant. For Alpaca GPT4 and OpenAssistant data sets, human-labeled instances with the highest quality scores can be specifically selected to ensure that the collected instruction response data pairs have high quality. To further enhance the diversity and challenge of the data, instances with response lengths exceeding 300 words can be selected from the above data sets to ensure that these instances can implicitly cover more complex constraint conditions. Finally, from the instruction response data pairs selected from the four data sets, random sampling is performed, and the proportion of each data set instance is mixed to finally obtain a high-quality data set containing 4,500 original instances (instruction response data pairs).

[0042] Step 102, extracting constraint information contained in the response from the instruction response data pair.

[0043] Constraint information is used to describe conditions or standards that the response meets, such as writing style, hierarchical instructions, and response length information.

[0044] In some embodiments, the constraint information includes a first type of constraint; wherein the first type of constraint is a constraint that can be extracted using a script, such as length, keyword, and punctuation constraints.

[0045] In specific implementation, a preset language-implemented script (for example, a Python script) can be used to extract the constraints contained in the response from the response; and the extracted constraints contained in the response are filled into the generated constraint template to obtain a literal representation of the constraints contained in the response.

[0046] In some embodiments, the constraint information contains constraints of a second type; wherein the constraints of the second type are constraints that can be extracted from the response by the large language model, for example, style constraints and the like.

[0047] In specific implementation, a prompt can be generated according to the response, the instructions corresponding to the response, and the literal description of the constraints of the second type; and the prompt is input into the large language model to obtain the constraints contained in the response output by the large language model.

[0048] For example, a set of 19 common constraint types is defined, including length and style, etc. Six of them are the first type of constraints, and the relevant values in the response are automatically identified using a Python script, and the template is filled to construct the constraints.

[0049] For example, for the length constraint, the generated constraint template is: “Please generate a response that is less than <placeholder> words but more than <placeholder> words”. For example, for the keyword and punctuation constraint, the corresponding constraint template can be filled according to a preset rule (for example, extracting keywords according to word frequency) or randomly selecting the corresponding items existing in the response.

[0050] The other 13 constraints of the second type are processed using a large language model (for example, Llama3-70B-Instruct) to generate constraints that are implicitly satisfied in the response. For example, the following prompt can be generated: “Please enrich the given instructions: … according to the provided output: … and comply with the requirements of the constraint set: …”. The prompt is input into the large language model, and the large language model extracts the constraints satisfied in the given output.

[0051] After extracting the constraints in the response using the large language model, the large language model can be used again to verify whether the constraints extracted by it can be understood and executed by it. For example, the extracted constraints can be added to the instructions, and the instructions carrying the constraint information are input into the large language model. If the large language model can generate a correct response, it can be determined that the extracted constraints can be understood and executed by the large language model; if the large language model does not generate a correct response, it can be determined that the extracted constraints cannot be understood and executed by the large language model, and the constraint can be excluded.

[0052] In specific implementation, a preset similarity algorithm (for example, using the ROUGE-L algorithm, setting a similarity threshold of 0.6) can also be used to exclude constraints similar to the instructions and other constraints.

[0053] Step 103, generating a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response.

[0054] In the specific implementation process, the constraint information contains multiple constraints, and the multiple constraints can be randomly combined to obtain multiple constraint sets; each constraint set contains at least one constraint; and the training data set is obtained according to each constraint set and the instruction and response corresponding to the constraint set.

[0055] For example, as shown in Figure 5 , 6 to 8 constraints can be randomly selected from the multiple constraints corresponding to each instruction, the order of the constraints can be shuffled, and the constraints can be recombined into the final constraint c.

[0056] In the specific implementation process, the training data set can be obtained according to each constraint set and the instruction and response corresponding to the constraint set by various methods, and is not limited by the description in the specification.

[0057] For two embodiments of obtaining the training data set according to each constraint set and the instruction and response corresponding to the constraint set, please refer to the related content in Figure 2 and Figure 3 , which will not be repeated here.

[0058] In some embodiments, in order to obtain more abundant training data, at least one (for example, 1 to 3) in-context demonstration can be added to a preset proportion (for example, 50%) of data in the training data set generated above.

[0059] Step 104, using the training data set to continue training the pre-trained large language model to obtain a target large language model, so as to use the target large language model to generate a response according to an instruction.

[0060] In the specific implementation process, the target large language model is applied to various fields. For example, in the field of content creation and writing assistance, the target large language model can help generate news reports, articles, blogs, etc., and provide inspiration or reduce the workload for writers; in the field of dialogue systems, the target large language model can be used as the core technology of chat robots (such as intelligent customer service, companion robots), so that the robot can understand and generate natural language dialogue; in the field of education, the target large language model can generate exercises, test questions or answers, and assist teaching and learning.

[0061] In the specific implementation process, the training data set can be used to continue training the pre-trained large language model to obtain the target large language model by various methods, and is not limited by the description in the specification.

[0062] Two embodiments of using the training data set to continue training the pre-trained large language model to obtain the target large language model are described in detail in Figure 2 and Figure 3 , which will not be repeated here.

[0063] Figure 2 is a flowchart of the second embodiment of the training method of the large language model provided by the present application, as shown in Figure 2 , the method comprises the following steps:

[0064] Step 201: According to each constraint set and its corresponding instruction, a set of sample data in the training data set is obtained, and the response corresponding to the constraint set is taken as the label of the sample data.

[0065] In the specific implementation process, each constraint set can be added to the corresponding instruction to obtain a set of sample data, and the response corresponding to the constraint set is taken as the label of the sample data.

[0066] All sample data and its corresponding labels constitute the training data set.

[0067] Step 202: Using the training data set, the pre-trained large language model is supervised fine-tuned.

[0068] In the specific implementation process, the pre-trained large language model can be supervised fine-tuned using various ways using the training data set, which is not limited by the description in the specification.

[0069] Figure 3 is a flowchart of the third embodiment of the training method of the large language model provided by the present application, as shown in Figure 3 , the method comprises the following steps:

[0070] Step 301: According to the response corresponding to each constraint set and the instruction, a set of first sample data in the first training subset is obtained, and the constraint set is taken as the label of the first sample data.

[0071] Step 302: According to all first sample data, the first training subset is obtained.

[0072] Since the large language model generates the correct response according to the instruction and the constraint, it is necessary to have a correct understanding of the constraint first. In order to enhance the understanding of the language model to the complex constraint, the present application proposes a reverse training method, which takes the instruction and response data (x, y) in the instruction-response data pair collected in step 101 as the input of the large language model, and trains the large language model to correctly generate the constraint c extracted in step 102.

[0073] In this embodiment, the training data set includes a first training subset and a second training subset. The first training subset is used for the reverse training described above.

[0074] Step 303, according to each set of constraint set and its corresponding instruction, a set of second sample data in the second training subset is obtained, and the response corresponding to the constraint set is taken as the label of the second sample data.

[0075] Step 304, according to all second sample data, the second training subset is obtained.

[0076] In the present application, a standard supervised fine-tuning (SFT) is also used to continue training the large language model, which is called forward training. In the forward training, the instruction in the instruction-response data pair collected in step 101 and the constraint corresponding thereto extracted in step 102 are taken as the input of the large language model, and the large language model is trained to correctly generate the response in the instruction-response data pair.

[0077] In the present embodiment, the second training subset is used for the above-mentioned forward training.

[0078] Step 305, the pre-trained large language model is jointly trained by using the first training subset and the first loss function, and the second training subset and the second loss function.

[0079] The first loss function is constructed based on the first sample data in the first training subset, and the probability of the large language model generating the label of the first sample data. As an example, the first loss function is a logarithmic loss function as follows:

[0080] L r =−logP θ (c∣x,y) (1)

[0081] Wherein, L r is the value of the first loss function, one of the training goals is to minimize the value; the large language model is represented by parameter θ; x is the instruction, y is the response, and c is the constraint; P θ (c∣x, y) represents the probability of the large language model generating the constraint c given the instruction x and the response y.

[0082] The second loss function is constructed based on the second sample data in the second training subset, and the probability of the large language model generating the label of the second sample data. As an example, the second loss function is a logarithmic loss function as follows:

[0083] L f =−logP θ (y∣x,c) (2)

[0084] Wherein, L fFor the value of the second loss function, one of the goals of training is to minimize the value; the large language model is represented by parameters θ; x is the instruction, y is the response, and c is the constraint; P θ (y∣x,c) represents the probability that the large language model generates the response y given the instruction x and the constraint c.

[0085] For example, the loss function of joint supervised fine-tuning is as follows:

[0086] L=αL f +(1−α)L r (3)

[0087] During the joint supervised fine-tuning process, α can be set to 0 first, and the large language model is trained using the first training subset for 70% of the training time; then α is set to 1, and the large language model is trained using the second training subset for the remaining time.

[0088] After the above joint training, a supervised fine-tuning (SFT) version of the large language model is obtained.

[0089] In some embodiments, after obtaining the SFT version of the language model, the SFT version of the language model can also be trained using a direct preference optimization (DPO) objective to obtain the best version.

[0090] In order to verify the effect of the target large language model trained by the method provided by the present application, two widely used open source base models, Mistral7B and Llama38B, are used as the main models of the experiment: llama3CRAB and MistralCRAB. In the SFT stage, a learning rate of 5×10−6, a batch size of 256, and training of the Mistral model for 4 epochs and the Llama3 model for 3 epochs are used. In the DPO optimization stage, a learning rate of 5×10−7, a batch size of 64, and 1 training epoch are used.

[0091] Popular open source and proprietary large language models are selected as baseline models and are divided into three categories for comparison:

[0092] (1) Proprietary LLMs: including GPT-3.5 and GPT-4.

[0093] (2) General Instruction Fine-Tuned LLMs: including Vicuna-V1.5 13B trained on 125k ShareGPT dataset, WizardLM-V1.2 13B trained on 196k Evol-Instruct dataset, and Zephyr beta7B trained on UltraFeedback dataset using DPO objective, which achieved state-of-the-art performance on Mistral7B-based chat benchmarks.

[0094] (3) Models optimized specifically for complex instruction execution tasks: Conifer series of models were used, which first generate constraints based on seed instructions, and then use GPT-4 to generate corresponding responses based on instructions and constraints, achieving state-of-the-art performance in complex instruction execution with 7B model size.

[0095] Two widely used and challenging complex instruction execution datasets, IFEval and FollowBench, were used for evaluation.

[0096] IFEval contains 541 instructions that can be automatically verified through Python scripts. Each instruction contains 1 to 3 constraints, mainly focusing on strict lexical and formatting constraints.

[0097] FollowBench is a fine-grained, multi-constraint instruction execution benchmark, with difficulty levels ranging from L1 to L5, based on the number of constraints in the instructions. L1 represents the simplest level, with only one constraint, while L5 is the most difficult, containing a combination of five constraints. It also includes five constraint categories: content, context, style, format, and example, as well as mixed constraint categories combining various constraint categories. FollowBench covers a total of 820 instructions for more than 50 different NLP tasks and can be automatically evaluated through Python scripts or GPT-4. For more details, please refer to the original paper.

[0098] The experimental results are shown in Table 1 below. The comparison results are as follows: (1) After training on the CRAB dataset, the models provided by the present application significantly outperform the corresponding base models and open-source models trained on general instruction execution datasets through SFT. The DPO version of the model provided by the present application performs best among all the compared models, demonstrating the effectiveness of the training data and training method constructed by the present application. (2) On IFEval, the models provided by the present application outperform the Conifer model trained specifically for complex instruction execution tasks. This indicates that the models provided by the present application perform better in following lexical and formatting constraints.

[0099] Table 1

[0100]

[0101] Complex instruction following ability not only involves following complex constraints, but also includes basic instruction execution ability, such as "write a blog about French cuisine", which is called general instruction execution. The general instruction execution ability of the model provided by the present application is further evaluated below. Since IFEval and FollowBench mainly focus on evaluating the ability to follow constraints, another widely used dataset AlpacaEval is adopted, which is an easy-to-use and high-quality automated instruction execution ability evaluation tool. Specifically, AlpacaEval2.0 is used, which contains 805 instructions, and gpt-4-1106-preview is used as the evaluator to obtain the final weighted win rate. The evaluation results are shown in Table 2 below, where "LC win rate" represents the length control win rate. The default reference model is gpt-4-1106-preview. It can be observed that the model provided by the present application is significantly better than the baseline model Conifer, and even exceeds the performance of the WizardLM70B version. The model provided by the present application is directly compared with Conifer. The LC win rate of the model provided by the present application reaches 60.6, which significantly exceeds Conifer, indicating that the model provided by the present application has stronger general instruction execution ability.

[0102] Table 2

[0103]

[0104] The training device of the large language model provided by the present application is described below. The training device of the large language model described below can be mutually corresponding with the training method of the large language model described above.

[0105] Figure 7 is a structural schematic diagram of the training device of the large language model provided by the present application. As Figure 7 shown, the device 700 includes the following modules.

[0106] The acquisition module 710 is configured to acquire an instruction-response data pair; wherein the instruction-response data pair is composed of an instruction and a response generated by a large language model according to the instruction, and the response includes text in a preset field.

[0107] The extraction module 720 is configured to extract constraint information contained in the response from the instruction-response data pair; wherein the constraint information is used to describe conditions or standards that the response meets.

[0108] The generation module 730 is configured to generate a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response.

[0109] The training module 740 is configured to perform fine-tuning on the pre-trained large language model by using the training data set to obtain a target large language model, so as to generate a response according to an instruction by using the target large language model.

[0110] Figure 8 An example of an entity structure diagram of an electronic device is shown in Figure 8 As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 can communicate with each other through the communications bus 840. The processor 810 can invoke a logical instruction in the memory 830 to execute a training method of a large language model, which includes: obtaining an instruction-response data pair; wherein the instruction-response data pair is composed of an instruction and a response generated by using a large language model according to the instruction, and the response includes text in a preset field; extracting constraint information contained in the response from the instruction-response data pair; wherein the constraint information is used to describe conditions or standards to which the response conforms; generating a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response; performing fine-tuning on a pre-trained large language model by using the training data set to obtain a target large language model, so as to generate a response according to an instruction by using the target large language model.

[0111] In addition, the logical instruction in the memory 830 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0112] In another aspect, the present application also provides a computer program product comprising a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to enable a computer to perform the training method of a large language model provided by the above-mentioned methods, the method comprising: obtaining an instruction-response data pair; wherein the instruction-response data pair is composed of an instruction and a response generated by a large language model according to the instruction, and the response comprises text in a preset field; extracting constraint information contained in the response from the instruction-response data pair; wherein the constraint information is used to describe conditions or standards to which the response conforms; generating a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response; and using the training data set to continue training a pre-trained large language model to obtain a target large language model, so as to generate a response according to an instruction by using the target large language model.

[0113] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program is executable by a processor to implement the training method of a large language model provided by the above-mentioned methods, the method comprising: obtaining an instruction-response data pair; wherein the instruction-response data pair is composed of an instruction and a response generated by a large language model according to the instruction, and the response comprises text in a preset field; extracting constraint information contained in the response from the instruction-response data pair; wherein the constraint information is used to describe conditions or standards to which the response conforms; generating a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response; and using the training data set to continue training a pre-trained large language model to obtain a target large language model, so as to generate a response according to an instruction by using the target large language model.

[0114] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0115] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0116] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a large language model, characterized in that, include: Acquire instruction response data pairs; wherein, the instruction response data pairs consist of an instruction and a response generated based on the instruction using a large language model, and the response includes text from a preset domain; From the responses contained in the instruction response data pair, the constraint information contained in the responses is extracted; wherein, the constraint information is used to describe the conditions or standards that the responses meet; Based on the response, the constraint information contained in the response, and the instruction corresponding to the response, a training dataset is generated, including: Multiple constraints are randomly combined to obtain multiple sets of constraints; wherein each set of constraints contains at least one constraint. The training dataset is obtained based on each set of constraints and its corresponding instructions and responses; Using the training dataset, the pre-trained large language model is further trained to obtain the target large language model, which is then used to generate a response based on the instruction. The training dataset includes a first training subset and a second training subset; The training dataset is obtained based on each set of constraints and its corresponding instructions and responses, including: Based on the response and instruction corresponding to each set of constraints, a set of first sample data in the first training subset is obtained, and the constraint set is used as the label of the first sample data. Based on all the first sample data, the first training subset is obtained; Based on each set of constraints and its corresponding instructions, a set of second sample data in the second training subset is obtained, and the response corresponding to the constraint set is used as the label of the second sample data. Based on all the second sample data, the second training subset is obtained; The step of further training the pre-trained large language model using the training dataset includes: The pre-trained large language model is jointly trained using the first training subset and the first loss function, as well as the second training subset and the second loss function; wherein, the first loss function is constructed based on the probability of the large language model generating the label of the first sample data from the first sample data in the first training subset, and the second loss function is constructed based on the probability of the large language model generating the label of the second sample data from the second sample data in the second training subset.

2. The training method for a large language model according to claim 1, characterized in that, The constraint information includes constraints of a first type; wherein, the constraints of the first type are constraints that can be extracted using a script; The step of extracting constraint information contained in the response from the response contained in the instruction response data pair includes: Using a script implemented in a preset language, extract the constraints contained in the response from the response; The constraints contained in the extracted response are filled into the generated constraint template to obtain the textual representation of the constraints contained in the response.

3. The training method for a large language model according to claim 1 or 2, characterized in that, The constraint information includes a second type of constraint; wherein the second type of constraint is a constraint that the large language model can extract from the response; The step of extracting constraint information contained in the response from the response contained in the instruction response data pair includes: Based on the response, the instruction corresponding to the response, and the textual description of the second type of constraint, generate prompt words; The prompt word is input into a large language model to obtain the constraints contained in the response output by the large language model.

4. The training method for a large language model according to claim 1, characterized in that, The training dataset is obtained based on each set of constraints and its corresponding instructions and responses, including: Based on each set of constraints and its corresponding instructions, a set of sample data in the training dataset is obtained, and the response corresponding to the constraint set is used as the label of the sample data. The step of further training the pre-trained large language model using the training dataset includes: The pre-trained large language model is then fine-tuned under supervision using the training dataset.

5. A training device for a large language model, characterized in that, include: An acquisition module is used to acquire instruction response data pairs; wherein, the instruction response data pair consists of an instruction and a response generated based on the instruction using a large language model, and the response includes text from a preset domain; An extraction module is used to extract constraint information contained in the response from the responses included in the instruction response data pair; wherein the constraint information is used to describe the conditions or standards that the response meets; The generation module is used to generate a training dataset based on the response, the constraint information contained in the response, and the instruction corresponding to the response, including: Multiple constraints are randomly combined to obtain multiple sets of constraints; wherein each set of constraints contains at least one constraint. The training dataset is obtained based on each set of constraints and its corresponding instructions and responses; The training module is used to further train the pre-trained large language model using the training dataset to obtain a target large language model, and to generate a response based on the instruction using the target large language model. The training dataset includes a first training subset and a second training subset; The training dataset is obtained based on each set of constraints and its corresponding instructions and responses, including: Based on the response and instruction corresponding to each set of constraints, a set of first sample data in the first training subset is obtained, and the constraint set is used as the label of the first sample data. Based on all the first sample data, the first training subset is obtained; Based on each set of constraints and its corresponding instructions, a set of second sample data in the second training subset is obtained, and the response corresponding to the constraint set is used as the label of the second sample data. Based on all the second sample data, the second training subset is obtained; The step of further training the pre-trained large language model using the training dataset includes: The pre-trained large language model is jointly trained using the first training subset and the first loss function, as well as the second training subset and the second loss function; wherein, the first loss function is constructed based on the probability of the large language model generating the label of the first sample data from the first sample data in the first training subset, and the second loss function is constructed based on the probability of the large language model generating the label of the second sample data from the second sample data in the second training subset.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method for the large language model as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method for the large language model as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method for the large language model as described in any one of claims 1 to 4.

Images