Large language model training method and device, electronic equipment and storage medium

Abstract

The invention provides a big language model training method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The training method of the large language model comprises the following steps: acquiring an instruction response data pair; wherein the instruction response data pair is composed of an instruction and a response generated by utilizing a large language model according to the instruction, and the response comprises a text of a preset field; extracting constraint information contained in the response from the response contained in the instruction response data pair; generating a training data set according to the response, constraint information contained in the response and an instruction corresponding to the response; and using the training data set to continuously train the pre-trained large language model to obtain a target large language model, and using the target large language model to generate a response according to the instruction. According to the method, the large language model capable of accurately executing the instruction with the complex constraint can be efficiently trained, so that the utilization rate of computing resources is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method, device, electronic device and storage medium for a large language model. Background Art

[0002] Large Language Models (LLMs) have demonstrated excellent performance in natural language processing tasks. When dealing with complex instructions with multiple constraints, existing methods usually rely on adding these constraints to the instructions and using advanced LLMs such as GPT-4 to generate responses. However, when faced with instructions containing complex constraints (such as format, length, etc.), these models often perform poorly, affecting their effectiveness and scope of application.

[0003] In order to improve the instruction execution capability of LLMs, a post-training strategy is usually adopted, that is, training with a large number of instruction-response data pairs. Although this method is simple and widely used, the training effect is not good. Even the most advanced LLMs are still unable to execute complex instructions, resulting in limited quality of generated data and requiring a lot of effort to screen the training data.

[0004] Therefore, how to efficiently train a large language model that can accurately execute instructions with complex constraints, thereby effectively improving the utilization of computing resources, is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above-mentioned problems existing in the prior art, the present invention provides a training method, device, electronic device and storage medium for a large language model, so as to efficiently train a large language model that can accurately execute instructions with complex constraints, thereby effectively improving the utilization rate of computing resources.

[0006] The present invention provides a large language model training method, comprising the following steps.

[0007] Obtain an instruction-response data pair; wherein the instruction-response data pair consists of an instruction and a response generated according to the instruction using a large language model, and the response includes text in a preset field; from the response contained in the instruction-response data pair, extract constraint information contained in the response; wherein the constraint information is used to describe the conditions or standards that the response meets; generate a training data set based on the response, the constraint information contained in the response, and the instruction corresponding to the response; use the training data set to continue training the pre-trained large language model to obtain a target large language model, so as to use the target large language model to generate a response according to the instruction.

[0008] According to a large language model training method provided by the present invention, the constraint information includes a first type of constraint; wherein the first type of constraint is a constraint that can be extracted using a script; extracting the constraint information contained in the response from the response contained in the instruction response data pair includes: extracting the constraints contained in the response from the response using a script implemented in a preset language; filling the extracted constraints contained in the response into a generated constraint template to obtain a textual representation of the constraints contained in the response.

[0009] According to a large language model training method provided by the present invention, the constraint information includes a second type of constraint; wherein the second type of constraint is a constraint that the large language model can extract from a response; extracting the constraint information contained in the response from the response contained in the instruction response data pair includes: generating a prompt word according to the response, the instruction corresponding to the response, and a text description of the second type of constraint; inputting the prompt word into the large language model to obtain the constraint contained in the response output by the large language model.

[0010] According to a large language model training method provided by the present invention, the constraint information includes multiple constraints; the training data set is generated according to the response, the constraint information included in the response and the instruction corresponding to the response, including: randomly combining the multiple constraints to obtain multiple groups of constraint sets; wherein each group of the constraint sets includes at least one constraint; and the training data set is obtained according to each group of the constraint sets and its corresponding instructions and responses.

[0011] According to a large language model training method provided by the present invention, the training data set is obtained according to each group of constraint sets and their corresponding instructions and responses, including: According to each group of the constraint sets and their corresponding instructions, a group of sample data in the training data set is obtained, and the responses corresponding to the constraint sets are used as labels of the sample data; and the use of the training data set to continue training the pre-trained large language model includes: using the training data set to supervise fine-tune the pre-trained large language model.

[0012] According to a large language model training method provided by the present invention, the training data set includes a first training subset and a second training subset; the training data set is obtained according to each group of the constraint sets and the corresponding instructions and responses, including: obtaining a group of first sample data in the first training subset according to the responses and instructions corresponding to each group of the constraint sets, and using the constraint set as a label of the first sample data; obtaining the first training subset according to all the first sample data; obtaining a group of second sample data in the second training subset according to each group of constraint sets and the corresponding instructions, and using the responses corresponding to the constraint set as the label of the second sample data; obtaining the second training subset according to all the second sample data; using the training data set to continue training the pre-trained large language model, including: using the first training subset and a first loss function, and the second training subset and a second loss function to jointly train the pre-trained large language model; wherein the first loss function is based on the first sample data in the first training subset, and the probability of the label of the first sample data generated by the large language model is constructed, and the second loss function is based on the second sample data in the second training subset, and the probability of the label of the second sample data generated by the large language model is constructed.

[0013] The present invention also provides a large language model training device, comprising the following modules: An acquisition module is used to acquire an instruction-response data pair; wherein the instruction-response data pair consists of an instruction and a response generated according to the instruction using a large language model, and the response includes text in a preset field; an extraction module is used to extract constraint information contained in the response from the response contained in the instruction-response data pair; wherein the constraint information is used to describe the conditions or standards that the response meets; a generation module is used to generate a training data set based on the response, the constraint information contained in the response, and the instruction corresponding to the response; a training module is used to continue training the pre-trained large language model using the training data set to obtain a target large language model, so as to generate a response according to the instruction using the target large language model.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method of the large language model as described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described large language model training methods.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned large language model training methods.

[0017] The training method, device, electronic device and storage medium of the large language model provided by the present invention extract the constraint information contained in the response from the response contained in the instruction response data pair; generate a training data set according to the response, the constraint information contained in the response and the instruction corresponding to the response; use the training data set to continue training the pre-trained large language model to obtain a target large language model. Since the constraint information is extracted from the response generated by the large language model, the large language model can accurately understand and execute the constraint information, so the training data set constructed using the constraint information can be used to train a large language model that can accurately execute instructions with complex constraints. Therefore, the present invention can efficiently train a large language model that can accurately execute instructions with complex constraints, thereby effectively improving the utilization rate of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the training method of the large language model provided by the present invention.

[0020] Figure 2 This is the second flow chart of the large language model training method provided by the present invention.

[0021] Figure 3 This is the third flow chart of the large language model training method provided by the present invention.

[0022] Figure 4 It is a schematic diagram of the constraint information contained in the text generated by the large language model.

[0023] Figure 5 It is a schematic diagram of a framework for constructing a training data set provided by the present invention.

[0024] Figure 6 A diagram illustrating examples of responses generated by a large language model based on instructions with constraints and responses generated based on instructions without constraints.

[0025] Figure 7 It is a structural schematic diagram of the large language model training device provided by the present invention.

[0026] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] During the experimental research, the inventor discovered an important fact: the existing data set actually contains implicit complex constraints. Figure 4 As shown, although the original brief instructions did not explicitly specify specific requirements such as writing style and length, the response content met these constraints on multiple levels.

[0029] The inventor further found through case studies that Figure 6 As shown in Figure 2, when a large language model generates a response based on an instruction, if the instruction contains inappropriate constraints, the generated response may contain vague terms such as '[database name]' and lack the necessary details and depth, resulting in a decrease in the content quality of the final response.

[0030] Therefore, we can use the existing high-quality instruction-response data pairs to extract constraints that can be understood and executed by the large language model, and integrate them into the instructions to construct training data containing effective constraints that do not carry noise. We can use this training data to train a large language model that can execute complex instructions efficiently and accurately.

[0031] For example, given an instruction x, defining a specific task such as “write a blog about French cuisine”; a constraint set c, specifying the conditions for the response, such as length limit; and a response y that satisfies both constraint c and instruction x, the goal is to build a high-quality dataset of (x, c, y) triplets.

[0032] Combine the following Figure 1-Figure 6 The present invention describes the training method of a large language model.

[0033] Figure 1 This is one of the flow charts of the training method of the large language model provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Obtain a command response data pair.

[0034] The instruction-response data pair consists of an instruction and a response generated based on the instruction using a large language model, and the response includes text in a preset domain. Figure 4 As shown in the example, the command is "write a blog about French cuisine" and the response is: “Title: The charm of French cuisine: a journey through time and taste

[0035] Content: French cuisine is often synonymous with elegance, sophistication and an unparalleled commitment to the culinary art. Whether in a bustling café in Paris or a quiet country kitchen in Provence, French cooking is more than just a meal; it is an expression of art. ..."

[0036] In a specific implementation process, high-quality instruction-response data pairs can be collected from a data set generated using a large language model, wherein the quality score of the response is greater than a preset threshold, the response satisfies multiple constraints, and there are dependencies between the multiple constraints.

[0037] Just as an example, Figure 5 As shown in the figure, a large number of instruction-response data pairs can be collected from four supervised fine-tuning datasets - Alpaca GPT4, OrcaChat, Evol Instruct and OpenAssistant. For the Alpaca GPT4 and OpenAssistant datasets, the instances with the highest human annotation quality scores can be specially selected to ensure that the collected instruction-response data pairs are of high quality. In order to further enhance the diversity and challenge of the data, instances with response lengths of more than 300 words can be selected from the above datasets to ensure that these instances can implicitly cover more complex constraints. Finally, random sampling is performed from the instruction-response data pairs selected from the four datasets, and they are mixed according to the proportion of instances in each dataset, and finally a high-quality dataset containing 4,500 original instances (instruction-response data pairs) is obtained.

[0038] Step 102: Extract constraint information contained in the response from the response contained in the instruction response data pair.

[0039] Constraint information is used to describe the conditions or standards that the response must meet, such as writing style, hierarchical instructions, and the length of the generated response.

[0040] In some embodiments, the constraint information includes a first type of constraint; wherein the first type of constraint is a constraint that can be extracted using a script, such as constraints such as length, keywords, and punctuation marks.

[0041] In the specific implementation process, a script implemented in a preset language (for example, a Python script) can be used to extract the constraints contained in the response from the response; the extracted constraints contained in the response are filled into the generated constraint template to obtain a textual representation of the constraints contained in the response.

[0042] In some embodiments, the constraint information includes a second type of constraint; wherein the second type of constraint is a constraint that a large language model can extract from the response, such as a style constraint.

[0043] In the specific implementation process, a prompt word can be generated according to the response, the instruction corresponding to the response and the text description of the second type of constraint; the prompt word is input into the large language model to obtain the constraints contained in the response output by the large language model.

[0044] As an example only, a set of 19 common constraint types are defined, including length and style, etc. Six of the first type of constraints are used to automatically identify relevant values ​​in the response using a Python script, filling in the template to construct the constraint.

[0045] For example, for the length constraint, the generated constraint template is: "Please generate a response with less than <placeholder> words but more than <placeholder> words." For another example, for keyword and punctuation constraints, the corresponding constraint template can be filled in according to preset rules (for example, extracting keywords by word frequency) or randomly selecting corresponding items in the response.

[0046] The other 13 second-type constraints are processed using a large language model (e.g., Llama3-70B-Instruct) to generate constraints that are implicitly satisfied in the response. As an example only, the following prompt can be generated: "Please enrich the given instructions: ... based on the provided output: ..., and comply with the requirements of the constraint set: ... ". This prompt is fed into the large language model, which extracts the constraints satisfied in the given output.

[0047] After using the large language model to extract the constraints in the response, the large language model can be used again to verify whether the extracted constraints can be understood and executed. For example, the extracted constraints can be added to the instruction, and the instruction carrying the constraint information can be input into the large language model. If the large language model can generate a correct response, it can be determined that the extracted constraints can be understood and executed by the large language model; if the large language model does not generate a correct response, it can be determined that the extracted constraints cannot be understood and executed by the large language model, and the constraints can be removed.

[0048] In a specific implementation process, a preset similarity algorithm (for example, using the ROUGE-L algorithm and setting a similarity threshold of 0.6) may also be used to exclude constraints that are similar to instructions and other constraints.

[0049] Step 103: Generate a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response.

[0050] In the specific implementation process, the constraint information includes multiple constraints, and the multiple constraints can be randomly combined to obtain multiple constraint sets; each constraint set includes at least one constraint; and a training data set is obtained based on each constraint set and its corresponding instructions and responses.

[0051] Just as an example, Figure 5 As shown, 6 to 8 constraints can be randomly selected from the multiple constraints corresponding to each instruction, their order can be disrupted, and they can be recombined into the final constraint c.

[0052] In the specific implementation process, a training data set can be obtained in a variety of ways according to each set of constraint sets and their corresponding instructions and responses, which are not limited to the description of this specification.

[0053] For two embodiments of obtaining training data sets according to each set of constraint sets and their corresponding instructions and responses, see Figure 2 and Figure 3 The relevant content in will not be repeated here.

[0054] In some embodiments, in order to obtain richer training data, at least one (eg, 1 to 3) in-context demonstration may be added to a preset proportion (eg, 50%) of the data in the above-generated training data set.

[0055] Step 104: Use the training data set to continue training the pre-trained large language model to obtain a target large language model, so as to use the target large language model to generate a response according to the instruction.

[0056] In the specific implementation process, the target large language model is applied in many fields. For example, in the field of content creation and writing assistance, the target large language model can help generate news reports, articles, blogs, etc., provide inspiration for writers or reduce their workload; in the field of dialogue systems, the target large language model can be used as the core technology of chatbots (such as intelligent customer service, companion robots), enabling robots to understand and generate natural language dialogues; in the field of education, the target large language model can generate exercises, test questions or answers to assist teaching and learning.

[0057] In the specific implementation process, the pre-trained large language model can be further trained using the training data set in a variety of ways to obtain a target large language model, which is not limited to the description of this specification.

[0058] Regarding two embodiments of using the training data set to continue training the pre-trained large language model to obtain the target large language model, see Figure 2 and Figure 3 The relevant content in will not be repeated here.

[0059] Figure 2 This is the second flow chart of the training method of the large language model provided by the present invention. Figure 2 As shown, the method includes the following: Step 201: According to each set of constraint sets and its corresponding instructions, a set of sample data in the training data set is obtained, and the response corresponding to the constraint set is used as a label of the sample data.

[0060] In a specific implementation process, each set of constraint sets may be added to a corresponding instruction to obtain a set of sample data, and the response corresponding to the constraint set may be used as a label of the sample data.

[0061] All sample data and their corresponding labels constitute the training data set.

[0062] Step 202: Use the training data set to perform supervised fine-tuning on the pre-trained large language model.

[0063] In the specific implementation process, a variety of methods can be used to supervise and fine-tune the pre-trained large language model using training data sets, which is not limited to the description of this specification.

[0064] Figure 3 FIG. 3 is a flow chart of the training method of the large language model provided by the present invention. Figure 3 As shown, the method includes the following: Step 301: Obtain a set of first sample data in a first training subset according to the responses and instructions corresponding to each set of constraint sets, and use the constraint sets as labels for the first sample data.

[0065] Step 302: Obtain a first training subset based on all first sample data.

[0066] Since the large language model generates correct responses based on instructions and constraints, it is necessary to first have a correct understanding of the constraints. In order to enhance the language model's understanding of complex constraints, the present invention proposes a reverse training method, which uses the instructions and response data (x, y) in the instruction-response data pairs collected in step 101 as inputs to the large language model, and trains the large language model to correctly generate the constraint c extracted in step 102.

[0067] In this embodiment, the training data set includes a first training subset and a second training subset. The first training subset is used for the above-mentioned reverse training.

[0068] Step 303: According to each set of constraint sets and its corresponding instructions, a set of second sample data in the second training subset is obtained, and the response corresponding to the constraint set is used as a label of the second sample data.

[0069] Step 304: Obtain a second training subset based on all second sample data.

[0070] In the present invention, standard supervised fine-tuning (SFT) is also used to continue training the large language model, which is called forward training. Forward training uses the instructions in the instruction-response data pair collected in step 101 and the corresponding constraints extracted in step 102 as inputs to the large language model, and trains the large language model to correctly generate the responses in the instruction-response data pair.

[0071] In this embodiment, the second training subset is used for the above-mentioned forward training.

[0072] Step 305: jointly train the pre-trained large language model using the first training subset and the first loss function, and the second training subset and the second loss function.

[0073] The first loss function is constructed based on the first sample data in the first training subset and the probability of the label of the first sample data generated by the large language model. As an example only, the first loss function is a logarithmic loss function as shown below: L r = −logP θ (c|x,y) (1)

[0074] Among them, L r is the value of the first loss function, one of the training goals is to minimize this value; the large language model is represented by the parameter θ; x is the instruction, y is the response, and c is the constraint; P θ (c|x, y) represents the probability that the large language model generates constraint c given instruction x and response y.

[0075] The second loss function is constructed based on the second sample data in the second training subset and the probability of the label of the second sample data generated by the large language model. As an example only, the second loss function is a logarithmic loss function as shown below: L f = −logP θ (y|x,c) (2)

[0076] Among them, L f is the value of the second loss function, one of the training goals is to minimize this value; the large language model is represented by parameter θ; x is the instruction, y is the response, and c is the constraint; P θ(y|x,c) represents the probability of the large language model generating response y given the instruction x and constraint c.

[0077] Just as an example, the loss function for joint supervised fine-tuning looks like this: L = αL f +(1−α)L r (3)

[0078] During the joint supervised fine-tuning process, you can first set α to 0, and use the first training subset to train the large language model for 70% of the training time; then set α to 1, and use the second training subset to train the large language model for the remaining time.

[0079] After the above joint training, the supervised fine-tuning (SFT) version of the large language model is obtained.

[0080] In some embodiments, after obtaining the SFT version of the language model, the Direct Preference Optimization (DPO) objective may be used to train the SFT version of the language model to obtain the version with the best effect.

[0081] In order to verify the effect of the target large language model trained by the method provided by the present invention, two widely used open source basic models, Mistral7B and Llama38B, were used as the backbone models of the experiment: llama3CRAB and MistralCRAB. In the SFT stage, a learning rate of 5×10−6 and a batch size of 256 were used, and the Mistral model was trained for 4 epochs and the Llama3 model for 3 epochs. In the DPO optimization stage, a learning rate of 5×10−7 and a batch size of 64 were used, and 1 training epoch was performed.

[0082] We select popular open source and proprietary large language models as baseline models and divide them into three categories for comparison: (1) Proprietary LLMs: including GPT-3.5 and GPT-4.

[0083] (2) General instruction fine-tuning LLMs: including Vicuna-V1.5 13B trained on the 125kShareGPT dataset, WizardLM-V1.2 13B trained on the 196kEvol-Instruct dataset, and Zephyrbeta7B trained on the UltraFeedback dataset using the DPO objective, which achieves leading performance on the Mistral7B-based chat benchmark.

[0084] (3) A model specifically optimized for complex instruction execution tasks: The Conifer series model was used, which first generates constraints based on seed instructions, and then uses GPT-4 to generate corresponding responses based on the instructions and constraints, achieving state-of-the-art performance in complex instruction execution at a 7B model scale.

[0085] Two widely used and challenging complex instruction execution datasets, IFEval and FollowBench, are used for evaluation.

[0086] IFEval contains 541 instructions that can be automatically verified by Python scripts. Each instruction contains 1 to 3 constraints, mainly focusing on strict vocabulary and format constraints.

[0087] FollowBench is a fine-grained, multi-constrained instruction execution benchmark with five levels of difficulty (L1 to L5), divided according to the number of constraints in the instruction, L1 represents the simplest level with only one constraint, and L5 is the most difficult, containing a combination of five constraints. It also includes five constraint categories, namely content, context, style, format, and examples, as well as a mixed constraint category that combines various constraint categories. FollowBench covers a total of 820 instructions for more than 50 different NLP tasks and can be automatically evaluated by Python scripts or GPT-4. See the original paper for more details.

[0088] The experimental results are shown in Table 1 below. The comparison results are as follows: (1) After training on the CRAB dataset, the model provided by the present invention is significantly better than the corresponding basic model and the open source model trained on the general instruction execution dataset through SFT. The DPO version model provided by the present invention performs best among all compared models, proving the effectiveness of the training data and training method constructed by the present invention. (2) On IFEval, the model provided by the present invention exceeds the Conifer model specially trained for complex instruction execution tasks. This shows that the model provided by the present invention performs better in complying with vocabulary and format constraints.

[0089] Table 1

[0090] Complex instruction execution capabilities not only involve following complex constraints, but also basic instruction execution capabilities, such as "write a blog about French cuisine", which is called general instruction execution. The general instruction execution capabilities of the model provided by the present invention are further evaluated below. Given that IFEval and FollowBench mainly focus on evaluating the ability to follow constraints, another widely used dataset, AlpacaEval, is used, which is an easy-to-use and high-quality automated instruction execution capability evaluation tool. Specifically, AlpacaEval2.0 was used, which contains 805 instructions, and gpt-4-1106-preview was used as an evaluator to obtain the final weighted win rate. The evaluation results are shown in Table 2 below, where "LC win rate" means the length control win rate. The default reference model is gpt-4-1106-preview. It can be observed that the model provided by the present invention is significantly better than the baseline model Conifer, and even exceeds the performance of the WizardLM70B version. By directly comparing the model provided by the present invention with Conifer. The LC win rate of the model provided by the present invention reached 60.6, significantly exceeding Conifer, indicating that the model provided by the present invention has stronger general instruction execution capabilities.

[0091] Table 2

[0092] The training device for a large language model provided by the present invention is described below. The training device for a large language model described below and the training method for a large language model described above can refer to each other.

[0093] Figure 7 Schematic diagram of the structure of the training device of the large language model provided by the present invention. Figure 7 As shown, the device 700 includes the following modules.

[0094] The acquisition module 710 is used to acquire an instruction-response data pair; wherein the instruction-response data pair consists of an instruction and a response generated according to the instruction using a large language model, and the response includes text in a preset domain.

[0095] The extraction module 720 is used to extract the constraint information contained in the response from the response contained in the instruction response data pair; wherein the constraint information is used to describe the conditions or standards that the response complies with.

[0096] The generating module 730 is used to generate a training data set according to the response, the constraint information contained in the response and the instruction corresponding to the response.

[0097] The training module 740 is used to continue training the pre-trained large language model using the training data set to obtain a target large language model, so as to generate a response according to the instruction using the target large language model.

[0098] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the training method of the large language model, the method comprising: obtaining an instruction response data pair; wherein the instruction response data pair is composed of an instruction and a response generated according to the instruction using the large language model, and the response includes a text of a preset domain; from the response included in the instruction response data pair, extracting the constraint information included in the response; wherein the constraint information is used to describe the condition or standard that the response meets; generating a training data set according to the response, the constraint information included in the response and the instruction corresponding to the response; using the training data set, continuing to train the pre-trained large language model to obtain a target large language model, so as to generate a response according to the instruction using the target large language model.

[0099] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0100] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large language model training method provided by the above methods, the method including: obtaining an instruction-response data pair; wherein the instruction-response data pair consists of an instruction and a response generated according to the instruction using a large language model, and the response includes a text in a preset field; from the response contained in the instruction-response data pair, extracting the constraint information contained in the response; wherein the constraint information is used to describe the conditions or standards that the response meets; generating a training data set based on the response, the constraint information contained in the response, and the instruction corresponding to the response; using the training data set, continuing to train the pre-trained large language model to obtain a target large language model, so as to generate a response according to the instruction using the target large language model.

[0101] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the large language model training method provided by the above-mentioned methods, the method comprising: obtaining an instruction-response data pair; wherein the instruction-response data pair consists of an instruction and a response generated according to the instruction using a large language model, and the response includes a text of a preset field; from the response contained in the instruction-response data pair, extracting constraint information contained in the response; wherein the constraint information is used to describe the conditions or standards that the response meets; generating a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response; using the training data set, continuing to train the pre-trained large language model to obtain a target large language model, so as to generate a response according to the instruction using the target large language model.

[0102] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a large language model, characterized in that: include: Acquire an instruction response data pair; wherein the instruction response data pair consists of an instruction and a response generated according to the instruction using a large language model, and the response includes text in a preset field; Extracting constraint information contained in the response from the response contained in the instruction response data pair; wherein the constraint information is used to describe the conditions or standards that the response complies with; generating a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response; The pre-trained large language model is further trained using the training data set to obtain a target large language model, so as to generate a response according to the instruction using the target large language model.

2. The method for training a large language model according to claim 1, characterized in that: The constraint information includes a first type of constraint; wherein the first type of constraint is a constraint that can be extracted using a script; The step of extracting constraint information contained in a response included in the instruction response data pair includes: Extracting the constraints contained in the response from the response using a script implemented in a preset language; The constraints contained in the extracted response are filled into the generated constraint template to obtain a text representation of the constraints contained in the response.

3. The method for training a large language model according to claim 1 or 2, characterized in that: The constraint information includes a second type of constraint; wherein the second type of constraint is a constraint that can be extracted from the response by a large language model; The step of extracting constraint information contained in a response included in the instruction response data pair includes: generating a prompt word according to the response, the instruction corresponding to the response, and the text description of the constraint of the second type; The prompt word is input into a large language model to obtain constraints contained in the response output by the large language model.

4. The method for training a large language model according to claim 1, characterized in that: The constraint information includes multiple constraints; The generating a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response includes: Randomly combining the plurality of constraints to obtain a plurality of constraint sets; wherein each constraint set includes at least one constraint; The training data set is obtained according to each group of the constraint sets and its corresponding instructions and responses.

5. The method for training a large language model according to claim 4, characterized in that: The step of obtaining the training data set according to each set of the constraint set and its corresponding instruction and response includes: According to each set of the constraint sets and the corresponding instructions, a set of sample data in the training data set is obtained, and the responses corresponding to the constraint sets are used as labels of the sample data; The method of continuing to train the pre-trained large language model using the training data set includes: The pre-trained large language model is fine-tuned using the training dataset.

6. The method for training a large language model according to claim 4, characterized in that: The training data set includes a first training subset and a second training subset; The step of obtaining the training data set according to each set of the constraint set and its corresponding instruction and response includes: According to the responses and instructions corresponding to each group of the constraint sets, a group of first sample data in the first training subset is obtained, and the constraint sets are used as labels of the first sample data; Obtaining the first training subset according to all the first sample data; According to each set of constraint sets and the corresponding instructions, a set of second sample data in the second training subset is obtained, and the response corresponding to the constraint set is used as a label of the second sample data; Obtaining the second training subset according to all the second sample data; The method of continuing to train the pre-trained large language model using the training data set includes: The pre-trained large language model is jointly trained using the first training subset and the first loss function, as well as the second training subset and the second loss function; wherein the first loss function is constructed based on the first sample data in the first training subset, and the probability of the label of the first sample data being generated by the large language model; and the second loss function is constructed based on the second sample data in the second training subset, and the probability of the label of the second sample data being generated by the large language model.

7. A large language model training device, characterized in that: include: An acquisition module, used to acquire an instruction response data pair; wherein the instruction response data pair consists of an instruction and a response generated according to the instruction using a large language model, and the response includes text in a preset field; An extraction module, used to extract constraint information contained in the response from the response contained in the instruction response data pair; wherein the constraint information is used to describe the conditions or standards that the response complies with; A generating module, configured to generate a training data set according to the response, the constraint information contained in the response, and the instruction corresponding to the response; The training module is used to continue training the pre-trained large language model using the training data set to obtain a target large language model, so as to generate a response according to the instruction using the target large language model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the training method of the large language model according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for training a large language model as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for training a large language model as claimed in any one of claims 1 to 6 is implemented.

Images