Optimized training, data processing methods, devices, equipment and media for large models

Abstract

The present application relates to the field of data processing technology, and in particular provides an optimization training, data processing method, device, equipment and storage medium for a large model. First, for each sample instruction, the sample instruction is input into a basic model to obtain N candidate answers for the sample instruction, and the candidate answers are scored using M different reward models to obtain M scoring results for the candidate answers; then the M scoring results of the candidate answers are weighted to obtain the preference value of the candidate answer; the first answer and the second answer are selected from the N candidate answers based on the preference value of the candidate answer; finally, a preference sample data set is constructed, the preference sample data set includes a preference sample, the preference sample includes a sample instruction and the first answer and the second answer corresponding to the sample instruction, and the preference sample data set is used to optimize the basic model in an offline manner. This reduces the consumption of massive computing resources and makes the large model preference alignment training task easier to implement.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a large model optimization training, data processing method, device, equipment and medium. Background Art

[0002] Large pre-trained language models, or big models for short, refer to deep learning models trained with large amounts of text data that can generate natural language text or understand the meaning of natural language text. The training of big models mainly consists of three stages: unsupervised pre-training, supervised fine-tuning, and preference alignment. Preference alignment refers to adjusting the behavior of big models so that their output is more consistent with human values ​​and subjective opinions, in order to reduce harmful, biased, or inaccurate outputs, increase beneficial, neutral, or accurate outputs, improve the performance of big models in aligning user intent, and generate outputs with greater accuracy, logical coherence, and readability.

[0003] Traditional preference alignment training is also called reinforcement learning from human feedback (RLHF), which includes the following process: select a supervised fine-tuned model as the base model, interact with the base model through a large number of human instructions, collect different answers generated by the base model under the same instruction, and then manually annotate the quality of multiple answers by human experts. Train a reward model (RM) based on the annotated preference data. The reward model receives an instruction and an answer as input and outputs a scalar value score for the answer. The model can reflect the subjective preferences of human experts. Through a large number of human instructions, the reward model is used to interact with the base model, and the output of the base model is fed back with a scalar value, and the preference alignment ability of the base model is trained within the policy optimization framework of the reinforcement learning algorithm.

[0004] Traditional RLHF relies on online reinforcement learning and completes training through iterative interactive trial and error and labeling. It requires alternating reasoning and parameter optimization of large models, which consumes a lot of computing resources. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a large model optimization training, data processing method, device, equipment and storage medium, which reduces the consumption of massive computing resources compared to traditional online preference alignment training and makes large model preference alignment training tasks easier to implement.

[0006] In a first aspect, the present application provides an optimization training method for a large model, the method comprising: obtaining K sample instructions and N groups of hyperparameters, K being greater than or equal to 1, and N being greater than or equal to 1; for each sample instruction, inputting the sample instruction into a basic model, using N groups of hyperparameters to control the basic model to process the sample instruction, and obtaining N candidate answers for the sample instruction, wherein the basic model is a large model that has undergone unsupervised pre-training and supervised fine-tuning; for each candidate answer, using M different reward models to score the candidate answers respectively, and obtaining M scoring results for the candidate answers, wherein M is greater than or equal to 1; performing weighted calculation on the M scoring results of the candidate answers to obtain a preference value for the candidate answer; for each sample instruction, selecting a first answer and a second answer from the N candidate answers based on the preference value of the candidate answer, the preference value of the first answer being greater than the preference value of the second answer; constructing a preference sample data set, the preference sample data set comprising multiple preference samples, the preference samples comprising the sample instruction and the first answer and the second answer corresponding to the sample instruction, and the preference sample data set being used to optimize the training of the basic model in an offline manner.

[0007] In a second aspect, the present application provides a data processing method, which includes: obtaining user instructions; inputting the user instructions into a large model so that the large model executes the user instructions, and the large model is obtained through the optimization training method of the large model in the first aspect mentioned above; and receiving the answer output by the large model.

[0008] In a third aspect, the present application provides an optimization training device for a large model, the device comprising: a data acquisition module, used to obtain K sample instructions and N groups of hyperparameters, K is greater than or equal to 1, and N is greater than or equal to 1; an answer generation module, used to input the sample instruction into a basic model for each sample instruction, and use N groups of hyperparameters to control the basic model to process the sample instruction to obtain N candidate answers for the sample instruction, and the basic model is a large model that has undergone unsupervised pre-training and supervised fine-tuning; a scoring module, used to score each candidate answer using M different reward models, and obtain M scores for the candidate answer. The preferred answer selection module is used to select the first answer and the second answer from N candidate answers based on the preference value of the candidate answer for each sample instruction, and the preference value of the first answer is greater than the preference value of the second answer; the data set construction module is used to construct a preference sample data set, and the preference sample data set includes multiple preference samples, and the preference samples include sample instructions and the first answer and the second answer corresponding to the sample instructions. The preference sample data set is used to optimize the basic model in an offline manner.

[0009] In a fourth aspect, the present application provides a data processing device, which includes: a user instruction acquisition module for acquiring user instructions; a data processing module for inputting user instructions into a large model so that the large model executes the user instructions, and the large model is obtained through the optimization training method of the large model in the first aspect mentioned above; an answer receiving module for receiving answers output by the large model.

[0010] In a fifth aspect, the present application provides an optimization training device for a large model, the device comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the optimization training method for the large model as in the first aspect above or the data processing method as in the second aspect above.

[0011] In a sixth aspect, the present application provides a storage medium, which may be a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the program implements the optimization training method for a large model as in the first aspect above or the data processing method as in the second aspect above.

[0012] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions, which, when executed by a processor, implements the optimization training method of a large model as described in any one of the first aspects or the data processing method as described in the second aspect.

[0013] The embodiment of the present application provides an optimization training, data processing method, device, equipment and storage medium for a large model. First, a large model that has undergone unsupervised pre-training and supervised fine-tuning is selected as the basic model, and then K diversified sample instructions covering multiple tasks are sampled in the labeled fine-tuning data set of the basic model to ensure a reasonable ratio between multiple task instructions. Then, for each sample instruction, N groups of random hyperparameters are used to repeatedly drive the basic model to generate N candidate answers to form a candidate answer set for the sample instruction, thereby constructing an unbiased sampling subset of the input and output data of the basic model. M reward models are selected to score each sample instruction and its N candidate answers, and each candidate answer corresponds to M scores. Based on RewardBench, the M reward models are evaluated separately to obtain the weight value of each reward model. Then the score value generated by each reward model is normalized. Finally, the M scores under each answer are weighted and fused. At this time, the preference value of each candidate answer. The N candidate answers under each sample instruction are screened, and the candidate answer with the highest preference value is selected as the better answer, and the candidate answer with the lowest preference value is selected as the worse answer. Together with the sample instruction, a preference sample is formed, and then a preference data set with a total of K is constructed. Based on the offline preference data set, a direct strategy optimization algorithm is used to train the model for preference alignment according to the minimized loss function, and a large model with preference alignment capability is obtained.

[0014] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0015] 1. Using multiple reward models to complete the preference data scoring and labeling can fully automate the preference alignment training of large models, avoid the human resource dependence in the traditional preference alignment training process, and reduce the cost of preference alignment of large models.

[0016] 2. The fusion scores of multiple reward models are used as the preference benchmark, which makes the scoring system have better generalization performance and makes the whole process suitable for widely used preference alignment task training.

[0017] 3. An offline approach is used to sequentially complete the two processes of preference data set construction and preference alignment training. Compared with traditional online preference alignment training, this reduces the consumption of massive computing resources and makes large-model preference alignment training tasks easier to implement.

[0018] 4. Through statistical sampling, the scores output by different reward models are normalized to the same scoring scale range, thereby supporting the weighted fusion of scores from multiple reward models. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A schematic diagram of a flow chart of an optimization training method for a large model provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of scoring candidate answers by multiple reward models provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a flow chart of an optimized training method for an optimized large model provided in an embodiment of the present application;

[0024] Figure 4 A flowchart of a data processing method provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of a large model optimization training device provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0029] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0030] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0031] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0032] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0033] The training of large models mainly includes three stages: unsupervised pre-training, supervised fine-tuning, and preference alignment. Unsupervised pre-training is the first step in the training process of large models. In the unsupervised pre-training stage, the model performs unsupervised learning through a large amount of text data to learn the structure of the language, grammar, and the relationship between vocabulary. The goal of unsupervised pre-training is to enable the model to acquire extensive language understanding and generation capabilities. Unsupervised pre-training is usually based on the Transformer architecture, which can capture long-distance dependencies and understand the relationship between each word in a sentence and other words through the self-attention mechanism. By performing unsupervised pre-training on large-scale unlabeled text data, the large model can tap into rich language knowledge and patterns, learn the common features of the language, and thus show excellent performance in various downstream tasks.

[0034] After pre-training, the large model may go through two stages of model optimization to optimize the performance of the large model on specific tasks, such as multi-round dialogue, knowledge question answering, etc. The two stages of model optimization mainly include the supervised fine-tuning stage and the preference alignment stage.

[0035] The supervised fine-tuning stage uses task-specific or domain-specific datasets to further optimize the large model, so that the large model can better comply with specific instructions and strengthen the large model's question-answering ability on domain tasks. The supervised fine-tuning stage uses domain data in the form of <instructions, answers> for training, so that the large model can better comply with specific instructions and strengthen the large model's question-answering ability on domain tasks. However, relying solely on supervised fine-tuning may have the following problems: (1) The difference in the distribution of the supervisory signal in the supervised fine-tuning data and the data in the pre-training stage may introduce erroneous cognition into the model, causing the model to hallucinate; (2) Supervised fine-tuning lacks a negative feedback mechanism and can only teach the model what the correct answer is, but not what the wrong answer is, resulting in insufficient generalization of the model and inability to better align human values ​​and intentions; (3) Supervised fine-tuning is not sufficient to optimize complex sequence-level objectives (e.g., the usefulness, harmlessness, safety, and repeatability of the model's responses), and is limited in generating more flexible and diverse outputs.

[0036] Therefore, preference alignment training needs to be further implemented after supervised fine-tuning. Preference alignment refers to the process of matching the output of a large model with human preferences. Preference alignment is particularly important in large models because the output of the large model needs to meet user expectations and needs. Through preference alignment training, human preferences and subjective opinions can be explicitly introduced to improve the model's performance in aligning user intent and generate outputs with greater accuracy, logical coherence, and readability. Traditional preference alignment is also often referred to as reinforcement learning from human feedback (RLHF), which includes the following processes:

[0037] (1) A supervised fine-tuned model is selected as the base model. A large number of input instructions are used to interact with the base model. Multiple different answers generated by the base model under the same input instruction are collected. Human experts then manually label the quality of multiple different answers and sort the answers from good to bad. The sorting results are then combined in pairs to form a data format of <instruction, better answer, worse answer> as training samples.

[0038] A reward model (RM) is trained based on the labeled preference data. The reward model receives data in the form of <instruction, better answer, worse answer> as input, and outputs the scalar value scores of the better answer and the worse answer. The scalar value scores of each answer are used to adjust the model parameters of the basic model. The reward model can reflect the subjective preferences of human experts.

[0039] By interacting with the basic model through a large number of input instructions, the reward model is used to provide scalar value feedback to the output of the basic model, and the preference alignment ability of the basic model is trained within the policy optimization framework of the reinforcement learning algorithm.

[0040] Among them, when human experts mark preference data, they generally use a ranking method to evaluate the quality of multiple answers. This is because human experts have individual differences in their values ​​and it is impossible to establish a unified and objective scoring standard. The absolute score values ​​given directly by experts may have large differences and uncertainties due to personal subjective factors, different evaluation criteria, or factors that are difficult to measure. Based on the combination of the sorting results in pairs, a data form of <instruction, better answer, worse answer> is formed. Then, in the training of the reward model, the Bradley-Terry model is used to convert the results of the pairwise comparison into scalar values.

[0041] The traditional RLHF preference alignment technique has the following defects.

[0042] First, traditional RLHF needs to rely on online reinforcement learning and complete training through iterative interactive trial and error and labeling, which requires alternating between reasoning and parameter optimization of large models. In other words, the above-mentioned basic model needs to output the answer, the reward model needs to score the answer, and then the model parameters are optimized based on the score. Finally, the answer is output based on the basic model, which requires multiple online cycle training, so it consumes a lot of resources and time.

[0043] Second, traditional RLHF relies on human experts to manually annotate a large amount of high-quality preference feedback data, which has three problems: (1) high time and human resource costs; (2) in order to make the annotation representative, it cannot rely solely on a single human expert. However, faced with large-scale, multi-field data annotation work, it is usually difficult to establish a professional, high-level group of human experts; (3) there are systematic individual differences in the subjective preference intentions and values ​​of different human experts, which makes it difficult to establish a unified preference scoring standard.

[0044] Third, since the training data is labeled by human experts, the amount of preference data for input instructions and answer annotations is limited, and the generalization and effectiveness of the reward model are difficult to guarantee, resulting in the preference alignment capability of large models being difficult to adapt to out-of-distribution situations.

[0045] In order to solve the above technical problems, the embodiment of the present application provides an optimization training method for a large model, which can support the fully automated construction of high-quality and diverse preference sample data sets, avoid the manual labeling costs of human experts, and improve the generalization performance of the preference scoring system by integrating the scores of multiple reward models, so that the whole process is applicable to the training of widely used preference alignment tasks. In addition, the two processes of preference data set construction and preference alignment training are completed in sequence in an offline manner, which reduces the consumption of huge computing resources by online RLHF-based preference alignment training, making large model preference alignment training tasks easier to implement.

[0046] The optimization training method for the large model provided by the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0047] Figure 1 This is a flow chart of a large model optimization training method in an embodiment of the present application. This embodiment is applicable to the situation of optimizing and training a large language model. The method can be executed by an optimization training device for a large model. The optimization training device for a large model can be implemented in software and / or hardware. The optimization training device for a large model can be configured in an electronic device.

[0048] like Figure 1 As shown, the optimization training method for a large model provided in an embodiment of the present application mainly includes steps S101-S106.

[0049] S101. Obtain K sample instructions and N sets of hyperparameters, where K is greater than or equal to 1, and N is greater than or equal to 1.

[0050] Instructions refer to specific requirements or questions that users ask the big model. Instructions are the main way for users to interact with the big model. Through instructions, users can guide the model to generate the required information, text, images and other content. Instructions can be very diverse. For example, instructions can be simple query instructions, such as "What is the weather in Beijing?" Instructions can also be complex task instructions, such as: "Please help me write an article about the development of artificial intelligence." Furthermore, depending on the application scenario of the big model, the form and complexity of instructions vary. For example, in the field of natural language processing, an instruction may be a paragraph of text; in the field of image generation, an instruction may be a text description plus specific style requirements.

[0051] The sample instruction is an instruction for optimizing the training of the large model during the training of the large model. Obtaining the sample instruction may include obtaining a fine-tuning sample instruction from a fine-tuning dataset of the large model, wherein the fine-tuning dataset is a dataset used for supervised fine-tuning training of the large model.

[0052] In the embodiment of the present application, the large model that has undergone unsupervised pre-training and supervised fine-tuning is used as the base model, and the base model is subjected to preference alignment training. In other words, the optimization training method of the large model provided in the embodiment of the present application is mainly used for optimization training of preference alignment of the large model that has undergone unsupervised pre-training and supervised fine-tuning.

[0053] Get K sample instructions, where K is an integer and K is greater than or equal to 1. Get all fine-tuning sample instructions from the fine-tuning dataset of the large model as training sample instructions for preference alignment. Or get a portion of fine-tuning sample instructions from the fine-tuning dataset of the large model as training sample instructions for preference alignment.

[0054] In this way, the sample instructions for preference alignment and the sample instructions for supervised fine-tuning training can be identically distributed, and because the sample instructions of the fine-tuning dataset for supervised fine-tuning cover a variety of tasks, the sample instructions for preference alignment can also cover a variety of tasks. In this way, it can prevent the large model from overfitting on a single task and improve the generalization ability of the model.

[0055] In an embodiment of the present application, a large model that has been fine-tuned under supervision is selected as the base model, and then K diverse sample instructions covering multiple tasks are sampled from the labeled fine-tuning data set of the base model to ensure a reasonable ratio between the instructions of multiple tasks.

[0056] The above hyperparameters can also be called inference hyperparameters. Hyperparameters are parameters used by the large model to generate answers, and are mainly used to adjust the behavior of the large model during the inference phase. Specifically, the above hyperparameters are mainly used to drive the basic model to generate diverse answers through random inference parameter strategies.

[0057] The above set of hyperparameters mainly include: sampling temperature T, maximum cumulative probability top p , the maximum number of candidate words top k , the maximum number of tokens generated max .

[0058] The sampling temperature T is used to control the randomness of the output of the large model. This is achieved by adjusting the probability distribution of the next word predicted by the large model. A higher temperature will make the probability distribution flatter, increasing the diversity and randomness of the generated text; a lower temperature will make the probability distribution sharper, and the generated text more certain and predictable.

[0059] The highest cumulative probability top p Sampling, also known as kernel sampling, is sampling from the smallest set of tokens whose cumulative probability reaches p. pControls how many words enter the candidate set in each step of the large model. By setting the maximum cumulative probability top p The number of candidate words can be adjusted dynamically to strike a balance between diversity and plausibility. A lower p-value will reduce the number of candidate words, and the generated text will be more focused and predictable; a higher p-value will increase the number of candidate words, and the generated text will be more diverse and random.

[0060] Maximum number of candidate words top k Sampling is to select the k words with the highest probability from the probability distribution of the next word predicted by the model, and then randomly select one of these k words as the output answer. By setting k, you can limit the creativity of the model and make its output more reasonable and controllable. A smaller k value will reduce the number of candidate words, and the generated text will be more focused and predictable; a larger k value will increase the number of candidate words, and the generated text will be more diverse and random.

[0061] The maximum number of tokens generated max Defines the maximum length of text generated by the large model, that is, the maximum number of tokens that the large model can generate during the inference process. max , you can control the length of the generated text and prevent large models from generating too long text.

[0062] Obtaining N sets of hyperparameters may include: obtaining N sets of data output by the random generation model as N sets of hyperparameters. Further, each different sample instruction may use the same N sets of hyperparameters, so that the workload of the random generation model can be reduced. Alternatively, each different sample instruction uses a different N sets of hyperparameters. In other words, for each sample instruction, the random generation model is used to generate N sets of hyperparameters. In this way, the diversity of the answers output by the large model can be further improved to improve the diversity of the training samples.

[0063] S102. For each sample instruction, input the sample instruction into a basic model, use N groups of hyperparameters to control the basic model to process the sample instruction, and obtain N candidate answers for the sample instruction.

[0064] The basic model is a large model that has undergone unsupervised pre-training and supervised fine-tuning. The unsupervised pre-training and supervised fine-tuning training can refer to the above description and will not be repeated in the embodiments of this application.

[0065] For each of the K sample instructions in S101 above, the random generation model generates N sets of hyperparameters. For each set of hyperparameters, the large model can be stimulated to generate an answer, which is mathematically expressed as formula (1).

[0066] response = LLM base(instruction|random(T,top p ,top k ,token max ))(1)

[0067] Among them, response represents an answer output by the large model, LLM base represents the supervised fine-tuned basic model, instruction represents an instruction, random represents random generation, and T represents the sampling temperature of large model reasoning, which can control the probability distribution difference of candidate words in each step of reasoning; top p Indicates the highest cumulative probability of large model reasoning, which can control how many words enter the candidate in each step of reasoning from the probability level; top k Indicates the maximum number of candidate tokens for each step of model reasoning; token max Indicates the maximum number of tokens generated by the large model, that is, the number of inference steps.

[0068] Through the above hyperparameter reasoning method, under the stimulation of N sets of hyperparameters, each sample instruction can generate N answers, forming a candidate answer set for each sample instruction. The candidate answer set includes N candidate answers.

[0069] The answer candidate set formed by multiple sampling of the model output answers is an unbiased sampling subset of the model input and output data, which ensures the consistency of data distribution in the data construction phase and preference alignment training, and establishes the data foundation for subsequent offline preference alignment training.

[0070] For each sample instruction, under random hyperparameters, the basic model is repeatedly driven to generate N answers according to formula (1) to form a candidate set of answers for the sample instruction, thereby constructing an unbiased sampling subset of the basic model input and output data.

[0071] S103. For each candidate answer, score the candidate answer using M different reward models to obtain M scoring results for the candidate answer, where M is greater than or equal to 1.

[0072] The above-mentioned reward model can learn to predict the scores of human evaluators on the answers generated by the model. In simple terms, the above-mentioned incentive model can identify which answers will be well-received by humans and which answers will be poorly rated by humans. The scoring result can be that the candidate answer is well-received by humans, or that the candidate answer is poorly rated by humans. Among them, the scoring result can also be a specific scoring value, where a larger scoring value indicates that the candidate answer is more likely to be well-received by humans, and a smaller scoring value indicates that the candidate answer is less likely to be well-received by humans.

[0073] The M incentive models can be incentive models of M different network architectures, or they can be reward models trained using preference data sets of different tasks. The M reward models can be obtained by self-training or by using open source reward models.

[0074] The training process of the reward model includes: first, collecting a large number of human experts' scores on the answers generated by the model. Use the human feedback data collected above to train a reward model. The goal of the reward model is to learn to predict the text or scores generated by human evaluators for the model. Once you have a reward model, you can use it to guide the training of a large model. At this stage, the model is not optimized through traditional supervised learning methods (i.e., minimizing the difference between the predicted output and the true label), but rather through reinforcement learning methods to adjust its parameters. Specifically, the model attempts to generate answers and evaluates its performance based on the reward signal given by the reward model. If the generated answer receives a high reward, it means that the output is closer to human preferences; otherwise, the strategy needs to be adjusted to improve future outputs.

[0075] M representative 7e 9 ~8e 9 Small open source reward models with different parameter sizes score candidate answers respectively. The open source reward models are trained on preference datasets of different tasks and have the ability to evaluate answer quality and distinguish good and bad answers to a certain extent. The advantage of selecting a small open source reward model is that a small-sized model can have a higher inference rate, thereby accelerating the labeling efficiency of preference data.

[0076] For each sample instruction, each reward model generates a scalar value score for each candidate answer, which is mathematically expressed as formula (2).

[0077]

[0078] Among them, x represents the sample instruction, y i represents the i-th candidate answer obtained by sampling the sample instruction x; RM j represents the jth reward model; Indicates that the j-th reward model is for <x,y i >The scoring result. The scoring result may be a specific scoring value.

[0079] Further, such as Figure 2As shown, under the stimulation of N sets of hyperparameters, sample instruction x can generate candidate answer 1, candidate answer 2, ..., candidate answer N, candidate answer 1 is input into M reward models to obtain M scoring results of candidate answer 1, candidate answer 2 is input into M reward models to obtain M scoring results of candidate answer 2, ..., candidate answer N is input into M reward models to obtain M scoring results of candidate answer N.

[0080] In the embodiment of the present application, preference scoring of candidate answers is completed through multiple open source reward models, and preference alignment training of large models can be completed fully automatically, avoiding human resource dependence in the preference alignment process and reducing the cost of preference alignment of large models.

[0081] S104: Perform weighted calculation on the M scoring results of the candidate answers to obtain a preference value of the candidate answer.

[0082] The preference value of a candidate answer is used to quantitatively represent the degree of preference of the candidate answer by humans. A larger preference value of a candidate answer indicates that the candidate answer is more likely to be favored by humans, and a smaller preference value of a candidate answer indicates that the candidate answer is more likely to be favored by humans.

[0083] Since the preference value of the candidate answer is weightedly integrated through M scoring results, the scoring system can be more generalized.

[0084] Since different reward models have different training strategies and produce different score ranges, they need to be normalized before weighted calculation. By means of statistical sampling, the score values ​​output by different reward models are normalized to the same score scale range, thus supporting the weighted fusion of scores from multiple reward models.

[0085] The M scoring results of the candidate answers are normalized, and the normalized M scoring values ​​are weighted to obtain the preference value of the candidate answer. The weighted calculation of the normalized M scoring values ​​may include: averaging the normalized M scoring values. The weighted calculation of the normalized M scoring values ​​may include: weighted calculation of the normalized M scoring values ​​and the weight values ​​corresponding to the M incentive models, and the weight values ​​corresponding to the M incentive models may be manually set.

[0086] In one possible implementation, a weighted calculation is performed on the M scoring results of the candidate answers to obtain a preference value of the candidate answers, including: for the i-th candidate answer, obtaining the weight values ​​of each scoring result of the candidate answer, the weight value of the scoring result is the score of the reward model that outputs the scoring result on the reward model evaluation benchmark RewardBench, and the score of the reward model is used to represent the ability to distinguish the quality of answers at the same spatial scale, where 1≤i≤N; weighting is performed based on the M scoring results of the candidate answers and the weight values ​​of each scoring result to obtain the preference value of the candidate answer.

[0087] Based on the ratings generated by the M reward models, in this embodiment, each reward model is comprehensively evaluated based on the reward model evaluation benchmark (RewardBench), and the ability of each reward model to distinguish good or bad answers is evaluated on the basis of a wide range of preference data covering diverse tasks. Then, the score of the reward model on RewardBench is used as its weight value, and the ratings given by the M reward models to the same candidate answer are weighted and fused to obtain the preference value of the candidate answer.

[0088] The reward model is obtained in mathematical form as formula (3).

[0089]

[0090] in, Represents the score of the j-th reward model on RewardBench; RM j represents the j-th reward model.

[0091] Then, the score values ​​of the candidate answers are normalized using the normalization formula (4).

[0092]

[0093] max(score j ) represents the maximum value of all the scoring results generated by the j-th reward model; min(score j ) represents the minimum value of all scoring results produced by the jth reward model.

[0094] Finally, the score values ​​of the candidate answers are weighted and fused using the weighted calculation formula (5).

[0095]

[0096] Among them, score i represents the preference value of the i-th candidate answer, represents the score of the j-th reward model on the reward model evaluation benchmark RewardBench, Represents the normalized scoring result of the j-th reward model for the i-th candidate answer.

[0097] In the embodiment of the present application, the weight value of the incentive model that represents the "authority" of the expert is introduced to make the fused preference value more representative. In principle, there are differences in the distribution of preference data trained by the M open source reward models, which leads to differences in their discrimination capabilities. Through the unified preference test benchmark of RewardBench, the quantitative capability representation of the M open source reward models at the same spatial scale is obtained, so that weighted fusion can be performed, making the integrated scoring system more generalizable.

[0098] By adopting an external reward model capability evaluation benchmark, we can obtain the quantitative capabilities of multiple open source reward models within a unified representation space scale, thereby supporting the weighted fusion of scores generated by multiple reward models, making the scoring system more generalizable.

[0099] S105. For each sample instruction, select a first answer and a second answer from N candidate answers based on the preference values ​​of the candidate answers, and the preference value of the first answer is greater than the preference value of the second answer.

[0100] In the preference alignment training of the large model, the data format of the training samples is <sample instruction, better answer, worse answer>, the first answer mentioned above can be used as the better answer in the data format, and the second answer mentioned above can be used as the worse answer in the data format.

[0101] In an embodiment of the present application, N candidate answers are combined in pairs to obtain multiple answer groups. For each answer group, the candidate answer with a larger preference value in the answer group is used as the first answer, and the candidate answer with a smaller preference value in the answer group is used as the second answer.

[0102] In this way, the answer groups after pairing are combined with sample instructions to increase the training data of the model.

[0103] In one possible implementation, a first answer and a second answer are selected from N candidate answers based on the preference values ​​of the candidate answers, including: selecting the candidate answer with the largest preference value from the N candidate answers as the first answer; selecting the candidate answer with the smallest preference value from the N candidate answers as the second answer.

[0104] Among the N candidate answers under one instruction, the answer with the highest fusion score is selected as the better answer, and the answer with the lowest fusion score is selected as the worse answer. This can maximize the preference difference of the data, give the model enough preference feedback, and speed up the training efficiency of preference alignment.

[0105] S106. Construct a preference sample data set, wherein the preference sample data set includes a plurality of preference samples, the preference samples include a sample instruction and a first answer and a second answer corresponding to the sample instruction, and the preference sample data set is used to optimize and train the basic model in an offline manner.

[0106] Based on the fusion score of the multi-reward model, a preference sample in the form of <sample instruction, first answer, second answer> can be constructed. Specifically, among the N candidate answers under a sample instruction, the candidate answer with the highest preference value is selected as the first answer, and the candidate answer with the lowest preference value is selected as the second answer. This can maximize the preference difference of the data, give the model sufficient preference feedback, and speed up the training efficiency of preference alignment. Finally, a preference sample can be expressed in the following mathematical form as formula (6).

[0107] <x,y better ,y worse > (6)

[0108] Among them, x represents the sample instruction, y better represents the first answer, that is, the candidate answer with the highest preference value; y worse Represents the first answer, which is the candidate answer with the lowest preference value.

[0109] In an embodiment of the present application, a large model that has been pre-trained and supervised fine-tuned is first selected as the basic model, and then K diverse sample instructions covering multiple tasks are sampled from the labeled fine-tuning data set of the basic model to ensure a reasonable ratio between the instructions of multiple tasks.

[0110] Then, for each sample instruction, under random hyperparameters, the basic model is repeatedly driven to generate N answers according to formula (1), forming N answer candidates for this sample instruction, thereby constructing an unbiased sampling subset of the basic model input and output data.

[0111] M small open source reward models are selected to score each sample instruction and its N candidate answers according to formula (2). The M score values ​​corresponding to each candidate answer are:

[0112] According to formula (3), M open source reward models are evaluated based on RewardBench to obtain the weight value of each reward model. Then, according to formula (4), the score value generated by each reward model is normalized. Finally, according to formula (5), the M score values ​​under each candidate answer are weighted and fused. At this time, each answer corresponds to a fused preference value.

[0113] According to formula (6), the N candidate answers under each sample instruction are screened, and the candidate answer with the highest preference value is selected as the better answer. The candidate answer with the lowest preference value is integrated as the worse answer, and together with the sample instruction, a preference sample is formed, thereby constructing a preference data set with a total of K.

[0114] In the embodiments of the present application, it is possible to support the fully automated construction of high-quality and diverse preference data sets, avoiding the manual labeling costs of human experts, and improving the generalization performance of the preference scoring system by integrating the preference values ​​of multiple reward model weights, making the entire process suitable for widely used preference alignment task training.

[0115] The technical solution provided by the embodiment of the present application has the following advantages:

[0116] 1. Using multiple reward models to complete the preference data scoring and labeling can fully automate the preference alignment training of large models, avoid the human resource dependence in the traditional preference alignment training process, and reduce the cost of preference alignment of large models.

[0117] 2. The fusion scores of multiple reward models are used as the preference benchmark, which makes the scoring system have better generalization performance and makes the whole process suitable for widely used preference alignment task training.

[0118] 3. An offline approach is used to complete the two processes of preference data set construction in sequence. Compared with traditional online preference alignment training, it reduces the consumption of massive computing resources and makes large-model preference alignment training tasks easier to implement.

[0119] 4. Through statistical sampling, the scores output by different reward models are normalized to the same scoring scale range, thereby supporting the weighted fusion of scores from multiple reward models.

[0120] On the basis of the above embodiments, the embodiments of the present application optimize the optimization training method of the above large model. The above embodiments provide an offline construction process of the preferred sample data set in the optimization training of the large model. The embodiments of the present application provide an offline training process in the optimization training of the large model, such as Figure 3 As shown, the optimization training process of the optimized large model provided in the embodiment of the present application includes steps S201-S202. It should be noted that the offline training process is performed after the offline construction process of the preferred sample data set.

[0121] S201. For any one of the preference samples in the preference sample data set, calculate the logarithmic probabilities of the first answer and the second answer in the preference sample based on a predefined loss function.

[0122] Model parameters in the base model are optimized based on the log probability.

[0123] Among them, the preference sample is usually composed of <sample instruction, first answer, second answer> data. The preference value of the first answer is greater than the preference value of the second answer. The basic model refers to a large model that has undergone unsupervised training and supervised fine-tuning, and has not yet undergone preference alignment training or is undergoing preference alignment training.

[0124] In this embodiment, a direct policy optimization (DPO) algorithm is used to perform offline preference alignment training.

[0125] The loss function of the direct strategy optimization algorithm is expressed as follows:

[0126]

[0127] Among them, θ represents the weight parameter of the basic model; E represents the expectation; is a preference sample data set, which contains K preference samples in total; σ represents the sigmoid function; π θ is the current generation strategy of the base model; π red is the original generation strategy of the base model; during training, the parameters of the base model are frozen and its generation strategy remains unchanged; β is a hyperparameter used to constrain the gap between the current generation strategy of the base model and the original generation strategy, so that the model does not deviate from the original strategy while aligning the preference, which can increase training stability.

[0128] S202. Optimize model parameters in the basic model based on logarithmic probability.

[0129] The calculation of the DPO loss function involves comparing the log probability of the first answer and the second answer. Then, an optimization algorithm such as gradient descent is used to directly minimize the loss function. By continuously adjusting the model parameters, the output generated by the model is more in line with the user's preferences.

[0130] Based on the offline preference dataset, a direct strategy optimization algorithm is used to train the model for preference alignment according to the above loss function to obtain a model with preference alignment capabilities.

[0131] In the embodiment of the present application, an offline method is adopted to sequentially complete the two processes of preference data set construction and preference alignment training. Compared with the traditional online preference alignment training, the consumption of huge computing resources is reduced, making the large model preference alignment training task easier to implement.

[0132] The present application provides a data processing method, such as Figure 4 As shown, the data processing method provided in the embodiment of the present application mainly includes steps S301-S303.

[0133] S301: Obtain user instructions.

[0134] User instructions can be text-based commands, questions, or any other form of information that the user hopes the big model can understand and respond to. For example, the user may enter the instruction: "What will the weather be like in Beijing tomorrow?"

[0135] S302. Input the user instruction into the big model so that the big model executes the user instruction. The big model is obtained by any of the above-mentioned optimization training methods for the big model.

[0136] If a user instruction is received, the user instruction is passed as input to the large model. The "large model" refers to the large model that has been trained with the above-mentioned preference alignment. The model can understand and process natural language, and generate corresponding outputs or perform specific tasks according to the instructions provided by the user. The optimization training method of the large model refers to the offline preference data set construction method and offline model training method provided in the above-mentioned embodiments.

[0137] S303, receiving the answer output by the large model.

[0138] The large model has processed the user's instructions and generated the corresponding output or answer. This answer may be a direct response to the user's question or the result of completing a task. For example, if the user asks for the weather forecast, the model may return a forecast for tomorrow's weather. This answer is then presented to the user.

[0139] The data processing method of the present application can provide users with efficient and accurate information services or auxiliary decision support.

[0140] Figure 5 Schematic diagram of the structure of a large model optimization training device in an embodiment of the present application, such as Figure 5 As shown, the large model optimization training device 50 provided in the embodiment of the present application mainly includes: a data acquisition module 51, an answer generation module 52, a scoring module 53, a preference value determination module 54, an answer selection module 55 and a data set construction module 56.

[0141] The data acquisition module 51 is used to obtain K sample instructions and N groups of hyperparameters, where K is greater than or equal to 1 and N is greater than or equal to 1. The answer generation module 52 is used to input the sample instruction into the basic model for each sample instruction, and use N groups of hyperparameters to control the basic model to process the sample instruction to obtain N candidate answers for the sample instruction, where the basic model is a large model that has undergone unsupervised pre-training and supervised fine-tuning. The scoring module 53 is used to score each candidate answer using M different reward models, and obtain M scoring results for the candidate answer, where M is greater than or equal to 1. The preference The value determination module 54 is used to perform weighted fusion on the M scoring results of the candidate answers to obtain the preference value of the candidate answer; the answer selection module 55 is used to select the first answer and the second answer from N candidate answers based on the preference value of the candidate answer for each sample instruction, and the preference value of the first answer is greater than the preference value of the second answer; the data set construction module 56 is used to construct a preference sample data set, the preference sample data set includes multiple preference samples, the preference samples include the sample instruction and the first answer and the second answer corresponding to the sample instruction, and the preference sample data set is used to optimize the basic model in an offline manner. Training.

[0142] The embodiment of the present application provides an optimization training device for a large model, which is used to perform the following process: first, obtain K sample instructions and N groups of hyperparameters; then for each sample instruction, input the sample instruction into the basic model, use N groups of hyperparameters to control the basic model to process the sample instruction, obtain N candidate answers for the sample instruction, and for each candidate answer, use M different reward models to score the candidate answers respectively, and obtain M scoring results for the candidate answers; then perform weighted calculation on the M scoring results of the candidate answers to obtain the preference value of the candidate answer; then for each sample instruction, select the first answer and the second answer from the N candidate answers based on the preference value of the candidate answer; finally, construct a preference sample data set, the preference sample data set includes multiple preference samples, the preference samples include sample instructions and the first answer and the second answer corresponding to the sample instructions, and the preference sample data set is used to optimize the basic model in an offline manner. Compared with the traditional online preference alignment training method, the preference alignment training of the large model can be completed fully automatically, avoiding human resource dependence and reducing the cost of preference alignment of the large model. It reduces the consumption of huge computing resources and makes the large model preference alignment training task easier to implement.

[0143] In a possible implementation, the answer selection module 55 is specifically configured to select the candidate answer with the largest preference value from N candidate answers as the first answer; and select the candidate answer with the smallest preference value from the N candidate answers as the second answer.

[0144] In one possible implementation, the preference value determination module 54 is specifically used to obtain the weight value of each scoring result of the candidate answer for the i-th candidate answer, the weight value of the scoring result is the score of the reward model that outputs the scoring result on the reward model evaluation benchmark RewardBench, and the score of the reward model is used to represent the ability to distinguish the quality of answers at the same spatial scale, where 1≤i≤N; weighting is performed based on the M scoring results of the candidate answer and the weight value of each scoring result to obtain the preference value of the candidate answer.

[0145] In a possible implementation, the preference value of the candidate answer is calculated by the following formula:

[0146]

[0147] Among them, score i represents the preference value of the i-th candidate answer, represents the score of the j-th reward model on the reward model evaluation benchmark RewardBench, represents the normalized scoring result of the j-th reward model for the i-th candidate answer;

[0148]

[0149] max(score j ) represents the maximum value of all the scoring results generated by the j-th reward model; min(score j ) represents the minimum value of all scoring results produced by the jth reward model.

[0150] In a possible implementation, it also includes: an offline training module, which is used to calculate the logarithmic probability of the first answer and the second answer in any preference sample in the preference sample data set based on a predefined loss function; and optimize the model parameters in the basic model based on the logarithmic probability.

[0151] In one possible implementation, the loss function is expressed as follows:

[0152]

[0153] Among them, θ represents the weight parameter of the basic model; E represents the expectation; is the preferred sample data set; σ represents the sigmoid function; π θ is the current generation strategy of the base model; π ref is the original generation strategy of the base model; β is a hyperparameter used to constrain the gap between the current generation strategy of the base model and the original generation strategy.

[0154] The large model optimization training device provided in the embodiment of the present application can execute the large model optimization training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0155] Figure 6 is a structural diagram of a data processing device in an embodiment of the present application, such as Figure 6 As shown, the data processing device 60 provided in the embodiment of the present application mainly includes: a user instruction acquisition module 61, a data processing module 62 and an answer receiving module 63.

[0156] Among them, the user instruction acquisition module 61 is used to obtain user instructions; the data processing module 62 is used to input the user instructions into the big model so that the big model executes the user instructions. The big model is obtained through the optimization training method of the big model in the above embodiment; the answer receiving module 63 is used to receive the answer output by the big model.

[0157] The data processing device provided in the embodiment of the present application can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0158] Figure 7 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device may include a large model optimization training device or a data processing device, such as Figure 7 As shown, the electronic device 700 includes a processor 710, a memory 720, an input device 730, and an output device 740; the number of the processor 710 in the electronic device can be one or more. Figure 7 A processor 710 is taken as an example; the processor 710, the memory 720, the input device 730 and the output device 740 in the electronic device can be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0159] The memory 720, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the optimization training method of the large model in the embodiment of the present invention, or the program instructions / modules corresponding to the data processing method in the embodiment of the present invention. The processor 710 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 720, that is, implementing the optimization training method or data processing method of the large model provided in the embodiment of the present invention.

[0160] The memory 720 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 720 may further include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The input device 730 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 740 may include a display device such as a display screen.

[0162] This embodiment also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to implement the optimization training method or data processing method for the large model provided in the embodiment of the present invention.

[0163] Of course, the storage medium containing computer executable instructions provided by an embodiment of the present invention is not limited to the method operations described above, and the computer executable instructions can also execute related operations in the optimization training method of the large model provided by any embodiment of the present invention.

[0164] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0165] It is worth noting that in the embodiment of the above-mentioned large model optimization training device, or in the embodiment of the above-mentioned data processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0166] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0167] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large model optimization training method, characterized in that: The method comprises: Get K sample instructions and N sets of hyperparameters, where K is greater than or equal to 1 and N is greater than or equal to 1; For each of the sample instructions, the sample instruction is input into a basic model, and the basic model is controlled by N sets of hyperparameters to process the sample instruction to obtain N candidate answers for the sample instruction, where the basic model is a large model that has undergone unsupervised pre-training and supervised fine-tuning; For each candidate answer, score the candidate answer using M different reward models to obtain M scoring results for the candidate answer, where M is greater than or equal to 1; Performing weighted calculation on the M scoring results of the candidate answers to obtain a preference value of the candidate answer; For each of the sample instructions, selecting a first answer and a second answer from the N candidate answers based on the preference values ​​of the candidate answers, wherein the preference value of the first answer is greater than the preference value of the second answer; Construct a preference sample data set, wherein the preference sample data set includes a plurality of preference samples, wherein the preference samples include sample instructions and the first answer and the second answer corresponding to the sample instructions, and the preference sample data set is used to optimize and train the basic model in an offline manner.

2. The method according to claim 1, characterized in that: Selecting a first answer and a second answer from the N candidate answers based on the preference values ​​of the candidate answers includes: Selecting the candidate answer with the largest preference value from the N candidate answers as the first answer; The candidate answer with the smallest preference value is selected from the N candidate answers as the second answer.

3. The method according to claim 1, characterized in that Performing weighted calculation on the M scoring results of the candidate answers to obtain the preference value of the candidate answers includes: For the i-th candidate answer, obtain the weight value of each scoring result of the candidate answer, the weight value of the scoring result is the score of the reward model that outputs the scoring result on the reward model evaluation benchmark RewardBench, and the score of the reward model is used to represent the ability to distinguish the quality of the answer at the same spatial scale, where 1≤i≤N; The preference value of the candidate answer is obtained by weighting the M scoring results of the candidate answer and the weight value of each scoring result.

4. The method according to claim 3, characterized in that The preference value of the candidate answer is calculated by the following formula: Among them, score i represents the preference value of the i-th candidate answer, represents the score of the j-th reward model on the reward model evaluation benchmark RewardBench, represents the normalized scoring result of the j-th reward model for the i-th candidate answer; in, max(score j ) represents the maximum value of all the scoring results generated by the j-th reward model; min(score j ) represents the minimum value of all scoring results generated by the j-th reward model.

5. The method according to claim 1, characterized in that Also includes: For any one of the preference samples in the preference sample data set, calculating the logarithmic probabilities of the first answer and the second answer in the preference sample based on a predefined loss function; Model parameters in the base model are optimized based on the log probability.

6. The method according to claim 5, characterized in that The loss function is expressed by the following formula: Among them, θ represents the weight parameter of the basic model; E represents the expectation; is the preferred sample data set; σ represents the sigmoid function; π θ is the current generation strategy of the base model; π ref is the original generation strategy of the base model; β is a hyperparameter used to constrain the gap between the current generation strategy of the base model and the original generation strategy.

7. A data processing method, characterized in that: The method comprises: Get user instructions; Inputting the user instruction into the large model so that the large model executes the user instruction, the large model being obtained by the optimization training method of the large model as claimed in any one of claims 1 to 6; Receive the answer output by the large model.

8. A large model optimization training device, characterized in that: The device comprises: A data acquisition module, used to acquire K sample instructions and N sets of hyperparameters, where K is greater than or equal to 1 and N is greater than or equal to 1; An answer generation module is used for inputting each sample instruction into a basic model, controlling the basic model to process the sample instruction using N sets of hyperparameters, and obtaining N candidate answers for the sample instruction, wherein the basic model is a large model that has undergone unsupervised pre-training and supervised fine-tuning; A scoring module, for scoring each candidate answer using M different reward models to obtain M scoring results for the candidate answer, where M is greater than or equal to 1; A preference value determination module, used for weighted fusion of the M scoring results of the candidate answers to obtain the preference value of the candidate answer; an answer selection module, configured to select, for each of the sample instructions, a first answer and a second answer from the N candidate answers based on the preference values ​​of the candidate answers, wherein the preference value of the first answer is greater than the preference value of the second answer; A data set construction module is used to construct a preference sample data set, wherein the preference sample data set includes multiple preference samples, and the preference samples include sample instructions and the first answer and the second answer corresponding to the sample instructions. The preference sample data set is used to optimize the training of the basic model in an offline manner.

9. An electronic device, characterized in that: The device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the optimization training method for the large model as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the optimization training method of the large model as described in any one of claims 1-7.

Images