Model training method and device based on preference learning and storage medium thereof

Abstract

The invention discloses a preference learning-based model training method and apparatus, and a storage medium. The method comprises the following steps of obtaining a training data set; according to the training data set and the word list of the pre-training model, obtaining a positive reply autoregression loss function; setting stop identifiers corresponding to the front reply and the back reply in the word list, and calculating a loss function of the stop identifiers; obtaining a probability loss function according to the probability values of the front reply and the back reply in the training data set; and obtaining a training loss function of the pre-training model according to the autoregression loss function, the loss function of the stop identifier and the probability loss function. The preference perception is performed by using the stop identifier, so that the perception ability of the model to different replies is improved; secondly, according to the probability values of the front reply and the back reply in the training data set, a probability loss function is obtained, the preference alignment ability of the model is improved, and the model training method for unifying instruction learning and preference learning is realized.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to a model training method, device and storage medium based on preference learning. Background Art

[0002] With the rise of the ChatGPT model, large models have gradually become the mainstream of research in the field of natural language processing. Large models refer to models with massive parameters in the field of deep learning. They have the characteristics of large scale, superior performance, strong generalization ability, and high computing resource requirements. With the continuous improvement of computing resources and the optimization of algorithms, large models have made breakthrough progress in many fields such as dialogue, image processing, and speech recognition. The training of large models usually involves two stages: instruction fine-tuning and preference learning. Instruction fine-tuning refers to giving the model the ability to follow instructions by learning dialogue data on the basis of the pre-trained model; preference learning refers to using human feedback reinforcement learning (Reinforcement Learning) on ​​the basis of instruction fine-tuning. g fromHuman Feedback (RLHF), enabling the model to generate responses that are more in line with human preferences.

[0003] The existing proximal policy optimization (PPO) algorithm requires a human preference dataset to train the reward model. Since it is costly to generate a high-quality human preference dataset, the generalization of the reward model is limited. At the same time, PPO training involves tuning a large number of parameters, which makes the PPO algorithm difficult to promote. Summary of the invention

[0004] To solve the above problems, the purpose of the present invention is to provide a model training method, device and storage medium based on preference learning, which performs preference perception through stop signs, improves the model's ability to perceive different replies, calculates the probability loss function according to the probability values ​​of positive and negative replies in the training data set, enhances the model's preference alignment capability, and improves the accuracy of large model training.

[0005] The technical solution adopted by the present invention to solve the problem is:

[0006] In a first aspect, an embodiment of the present application provides a model training method based on preference learning, the method comprising: obtaining a training data set, wherein the training data set consists of multiple training samples, and the training samples include training questions, positive replies, and negative replies; obtaining an autoregressive loss function of the positive replies based on the training data set and the vocabulary of the pre-trained model; setting stop signs corresponding to the positive replies and the negative replies in the vocabulary, and calculating the loss function of the stop sign; obtaining a probability loss function based on the probability values ​​of the positive replies and the negative replies in the training data set; obtaining the training loss function of the pre-trained model based on the autoregressive loss function, the loss function of the stop sign, and the probability loss function.

[0007] In a second aspect, an embodiment of the present application provides a large model training device based on preference learning, including: an acquisition module, used to acquire a training data set, wherein the training data set is composed of multiple training samples, and the training samples include training questions, positive replies and negative replies; an autoregressive loss module, used to obtain the autoregressive loss function of the positive reply based on the training data set and the vocabulary of the pre-trained model; a setting module, used to set stop signs corresponding to the positive replies and the negative replies in the vocabulary, and calculate the loss function of the stop sign; a probability loss module, used to obtain the probability loss function according to the probability values ​​of the positive replies and the negative replies in the training data set; a training loss module, used to obtain the training loss function of the pre-trained model based on the autoregressive loss function, the loss function of the stop sign and the probability loss function.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the model training method based on preference learning as described above is implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the model training method based on preference learning as described above.

[0010] In the embodiment of the present application, a training data set is obtained; based on the vocabulary of the training data set and the pre-trained model, an autoregressive loss function of positive replies is obtained; stop signs corresponding to positive replies and negative replies are set in the vocabulary, and the loss function of the stop sign is calculated; based on the probability values ​​of positive replies and negative replies in the training data set, a probability loss function is obtained; based on the autoregressive loss function, the loss function of the stop sign and the probability loss function, a training loss function of the pre-trained model is obtained. By using the stop sign for preference perception, the model's perception ability of different replies is improved; secondly, based on the probability values ​​of positive replies and negative replies in the training data set, a probability loss function is obtained to improve the model's preference alignment ability, thereby realizing a model training method that unifies instruction learning and preference learning.

[0011] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flow chart of a model training method based on preference learning according to an embodiment of the present invention;

[0013] Figure 2 for Figure 1 Flowchart of step S2000;

[0014] Figure 3 for Figure 1 Flowchart of step S3000;

[0015] Figure 4 for Figure 3 Flowchart of step S3500;

[0016] Figure 5 for Figure 1 Flowchart of step S4000;

[0017] Figure 6 for Figure 1 A flowchart of another embodiment of step S4000;

[0018] Figure 7 for Figure 6 Flowchart of step S4500;

[0019] Figure 8 This is a structural diagram of a large model training device based on preference learning according to an embodiment of the present invention;

[0020] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0022] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0023] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0024] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0025] The model training method, device and storage medium based on preference learning involved in the embodiment of the present invention are as follows: obtaining a training data set; obtaining an autoregressive loss function of positive replies according to the training data set and the vocabulary of the pre-trained model; setting stop signs corresponding to positive replies and negative replies in the vocabulary, and calculating the loss function of the stop sign; obtaining a probability loss function according to the probability values ​​of positive replies and negative replies in the training data set; obtaining a training loss function of the pre-trained model according to the autoregressive loss function, the loss function of the stop sign and the probability loss function. By using the stop sign for preference perception, the model's perception ability of different replies is improved; secondly, according to the probability values ​​of positive replies and negative replies in the training data set, the probability loss function is obtained to improve the model's preference alignment ability, thereby realizing a model training method that unifies instruction learning and preference learning.

[0026] Large models refer to machine learning models with large-scale parameters and complex computational structures. These models are usually built with deep neural networks and have billions or even hundreds of billions of parameters. Large models are designed to improve the expressiveness and predictive performance of the model and to handle more complex tasks and data. Large models are widely used in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems. Large models learn complex patterns and features by training massive amounts of data, have stronger generalization capabilities, and can make accurate predictions on unseen data.

[0027] At present, ChatGPT's explanation of the large model is more understandable and more similar to human-like induction and thinking ability: the large model is essentially a deep neural network model trained with massive data. Its huge data and parameter scale enable the emergence of intelligence and show human-like intelligence. Compared with small models, large models usually have more parameters and deeper layers, with stronger expression ability and higher accuracy, but also require more computing resources and time for training and reasoning. They are suitable for scenarios with large data volumes and sufficient computing resources, such as cloud computing, high-performance computing, artificial intelligence, etc.

[0028] Although existing large model training methods can process large amounts of data and achieve the effect of optimizing models, for example, the main goal of the PPO algorithm is to improve the policy gradient algorithm to make the training process more stable while maintaining good performance. However, in existing training methods, it is expensive to generate high-quality human preference data sets, the generalization of the reward model is limited, and PPO training involves the tuning of a large number of parameters, making the PPO algorithm difficult to promote.

[0029] Based on the above, the embodiments of the present invention provide a model training method, device and storage medium based on preference learning, by acquiring a training data set; obtaining an autoregressive loss function of positive replies based on the training data set and the vocabulary of the pre-trained model; setting stop signs corresponding to positive replies and negative replies in the vocabulary, and calculating the loss function of the stop sign; obtaining a probability loss function based on the probability values ​​of positive replies and negative replies in the training data set; obtaining a training loss function of the pre-trained model based on the autoregressive loss function, the loss function of the stop sign and the probability loss function. By using the stop sign for preference perception, the model's perception ability of different replies is improved; secondly, the probability loss function is obtained based on the probability values ​​of positive replies and negative replies in the training data set, the model's preference alignment ability is improved, and a model training method that unifies instruction learning and preference learning is realized.

[0030] See also Figure 1 , Figure 1 The flow chart of the model training method based on preference learning provided by the embodiment of the present invention is shown. Figure 1As shown, the model training method based on preference learning of an embodiment of the present invention includes the following steps:

[0031] Step S1000: Obtain a training data set, wherein the training data set consists of a plurality of training samples, and the training samples include training questions, positive responses, and negative responses.

[0032] It is understood that training data refers to data used to train data mining models during data mining. The selection of training data generally has the following requirements: the data sample is as large as possible, the data is diversified, and the data sample quality is high. In the embodiment of the present application, the training data set is a data set X consisting of preference question-answer pairs. Specifically, the training data set is as follows:

[0033] X={(q1,c1,r1),(q2,c2,r2),...(q n , c n , r n )}

[0034] Among them, (q i , c i , r i ) represents the i-th question-answer pair consisting of a training question, a positive (chosen) response, and a negative (rejected) response.

[0035] Step S2000: Obtain an autoregressive loss function for positive responses based on the training data set and the vocabulary of the pre-trained model.

[0036] Understandably, in the field of data science and Natural Language Processing (NLP), the vocabulary of a pre-trained model is a key concept, which refers to a list containing all or most of the vocabulary in a specific field. This list is crucial for many NLP tasks such as text analysis, information retrieval, machine translation, sentiment analysis, etc. The data vocabulary can be general, covering a wide range of topics, or it can be targeted to a specific field (such as medicine, law, finance, etc.) to contain terms and vocabulary unique to the field.

[0037] See also Figure 2 , Figure 2 FIG. 2 shows a schematic diagram of a specific implementation process of another embodiment of the above step S2000. Figure 2 As shown, step S2000 at least includes the following steps:

[0038] Step S2100: Calculate the probability value of each identifier of a positive reply in the vocabulary to obtain a first probability value.

[0039] It is understandable that in the field of natural language processing (NLP), token is a core concept, which refers to the smallest unit or basic element for processing text. In the Large Language Model (LLM), a token is the basic information unit for the model to process language. It can be a character, a word or even a phrase sentence. The definition of a token is not static, and its division granularity will be different in different contexts. For example, in the sentence "I like apples", "I" and "apple" can both be regarded as a token, but in the sentence "I like apple pie", "apple pie" is regarded as an independent token. This flexibility enables tokens to adapt to different language processing needs. Whether it is a word, phrase, punctuation, subword or character, it can be processed as a token.

[0040] A vocabulary is a collection of tokens used in NLP, and contains all the basic elements needed by the model to process language. The construction of the vocabulary is crucial to the performance and efficiency of the model. By adding tokens to the vocabulary, the model is better able to understand and generate language, because each token is assigned a numerical value or identifier and arranged in a sequence or vector, which is input or output from the model. This process ensures the coherence and consistency of the text, allowing the model to effectively handle various tasks such as writing, translation, and answering queries.

[0041] Specifically, the first probability value Indicates a positive response c i The probability value of the vth token in the vocabulary at the jth position.

[0042] Step S2200: Determine whether the positive reply is consistent with each identifier in the vocabulary, and obtain a second probability value.

[0043] It is understandable that the second probability value Indicates whether the token at the jth position in the word is consistent with the vth token in the word list. If they are consistent, it is 1, otherwise it is 0.

[0044] Step S2300: Obtain an autoregressive loss function according to the first probability value and the second probability value.

[0045] It is understandable that after obtaining the first probability value and the second probability value, it is necessary to summarize the positive responses c i The probability of occurrence of , determines the autoregressive loss function, so as to give the model the ability to answer questions. Specifically, the autoregressive loss function loss sft The calculation formula is as follows:

[0046]

[0047] Where |c| is c i , |V| is the length of the vocabulary.

[0048] Step S3000: Set stop signs corresponding to positive replies and negative replies in the vocabulary, and calculate the loss function of the stop signs.

[0049] It is understandable that the stop token is a functional token that mainly feeds back the signal that the user stops decoding and has no unique semantic information. Due to the particularity of this token, this application combines it with the idea of ​​text matching and adds [END c ] and [END r ], relying on the type of generated stop tokens, strengthens the model's ability to distinguish between correct and rejected responses.

[0050] See also Figure 3 , Figure 3 FIG. 5 is a schematic diagram showing a specific implementation process of another embodiment of the above step S3000. Figure 3 As shown, step S3000 at least includes the following steps:

[0051] Step S3100: Set a positive stop mark corresponding to a positive reply and a negative stop mark corresponding to a negative reply in the vocabulary.

[0052] It is understandable that two special tokens are added to the vocabulary of the pre-trained model: [END c ] and [END r ] are used to mark the stop signals of positive replies and negative replies respectively, so as to enhance the model's ability to distinguish correct and rejected replies.

[0053] Step S3200: Add a positive stop sign and a negative stop sign to the positive reply and the negative reply respectively.

[0054] It is understandable that based on the training data set X, [END c ] and [END r ] are added to the end of each, as shown in the following formula:

[0055] c i ={c i , [END c ]}

[0056] r i = {r i , [END r ]}

[0057] It is understandable that the last token can aggregate the semantic information of the entire input sentence. However, the token is currently only used to transmit the end signal of decoding, and the semantic representation contained in it is not utilized. The embodiment of the present application assigns two types to this token to learn the perception of different replies during training, and in the use stage, once the model is generated with [END r ] as the end character, we can know that the output does not meet human preferences and can be filtered and regenerated to ensure output quality.

[0058] Step S3300: Calculate the output probability of the positive stop sign in the pre-trained model to obtain a third probability value.

[0059] It can be understood that for a positive reply, the embodiment of the present application needs to set the label of the termination token of the positive reply to [ENDc], and then convert the termination token into a vocabulary index and send it to the model for prediction to obtain the output probability value of the termination token, that is, the third probability value [ ENDc] .

[0060] Step S3400: Calculate the output probability of the negative stop sign in the pre-trained model to obtain a fourth probability value.

[0061] It can be understood that, for a negative reply, the embodiment of the present application needs to set the label of the termination token of the negative reply to [ENDr], and then convert the termination token into a vocabulary index and send it to the model for prediction to obtain the output probability value of the termination token, that is, the fourth probability value [ ENDr] .

[0062] Step S3500: Obtain a loss function of the stop sign according to the third probability value and the fourth probability value.

[0063] It can be understood that the third probability value and the fourth probability value obtained through the above steps S3300 and S3400 are processed by negative logarithms to obtain the loss function of the stop sign.

[0064] See also Figure 4 , Figure 4 FIG. 5 is a schematic diagram showing a specific implementation process of another embodiment of the above step S3500. Figure 4 As shown, step S3500 at least further includes the following steps:

[0065] Step S3510: Obtain a negative log-likelihood loss function of a positive response according to the third probability value.

[0066] It can be understood that negative log-likelihood loss (NLL Loss) is a common form of likelihood loss function, which is used for classification problems. The performance of the model can be measured by calculating the difference between the probability distribution predicted by the model and the true distribution. In multi-classification problems, NLL Loss is obtained by calculating the negative value of the log probability between the predicted probability and the true label of each category. According to the third probability value obtained in the above steps, the negative log-likelihood loss function for positive responses is as follows:

[0067]

[0068] Among them, loss c is the loss function for positive stopping marks.

[0069] Step S3520: According to the fourth probability value, a negative log-likelihood loss function of the negative reply is obtained.

[0070] It can be understood that according to the fourth probability value obtained in the above steps, the negative log-likelihood loss function of the negative reply is as follows:

[0071]

[0072] Among them, loss r is the loss function for negative stop sign.

[0073] Step S3530, sum the negative log-likelihood loss function of the positive reply and the negative log-likelihood loss function of the negative reply to obtain the loss function of the stop sign.

[0074] It can be understood that the loss function of the positive stop sign and the loss function of the negative stop sign obtained in the above steps are summed to obtain the loss function of the stop sign as shown in the following formula:

[0075] loss stop =loss c +loss r

[0076] Step S4000: Obtain a probability loss function according to the probability values ​​of positive responses and negative responses in the training data set.

[0077] It is understandable that after learning the loss function in the above steps, the model has the ability to discriminate the tendency of replies. However, in terms of generating replies, due to the i and r i The training of the above steps is separated from each other, and the formula in the above steps still cannot fully explore the preference alignment ability of the model. To solve this problem, the embodiment of the present application introduces a loss function based on preference pairs to enhance the model in generating c i and ri differences.

[0078] See also Figure 5 , Figure 5 FIG. 4 is a schematic diagram showing a specific implementation process of another embodiment of the above step S4000. Figure 5 As shown, step S4000 at least further includes the following steps:

[0079] Step S4100: Calculate the probability values ​​of positive responses and sum them up to obtain the total probability of positive responses.

[0080] It is understood that probability is a numerical value that describes the likelihood of an event occurring, and probability is always between 0 and 1. To calculate the average of a set of probability values, the same method as calculating the average of other numerical values ​​can be used. Specifically, all probability values ​​are added up and then divided by the number of these probability values. In this embodiment, the sum of the average probability values ​​of each token is regarded as the reward score of the model's external feedback, c i The probability value of each token in is regarded as the bonus score for positive response. The total probability of positive response is shown in the following formula:

[0081]

[0082] Step S4200, calculate the occurrence probability values ​​of the negative reply and sum them up to obtain the total probability of the negative reply.

[0083] Similarly, the sum of the probability values ​​of each token in is regarded as the reward score of the model's external feedback, r i The probability value of each token in is regarded as the reward score of the negative reply. The total probability of the negative reply is calculated as follows:

[0084]

[0085] Step S4300: Obtain an average probability value of the positive reply according to the length of the positive reply and the sum of the probabilities of the positive reply.

[0086] It can be understood that according to the length of the positive response |c| and the sum of the probabilities of the positive responses, the average probability value of the positive response is obtained as shown in the following formula:

[0087]

[0088] Step S4400: Obtain an average probability value of the negative reply according to the length of the negative reply and the sum of the probabilities of the negative reply.

[0089] It can be understood that according to the length of the negative reply |r| and the sum of the probabilities of the negative replies, the average probability of the negative reply is obtained as shown in the following formula:

[0090]

[0091] See also Figure 6 , Figure 6 FIG. 4 is a schematic diagram showing a specific implementation process of another embodiment of the above step S4000. Figure 6 As shown, step S4000 at least further includes the following steps:

[0092] Step S4500: Obtain a reward value for a positive reply based on the average probability value of a positive reply and the average probability value of a negative reply.

[0093] It is understandable that after obtaining the average probability value of the positive response and the average probability value of the negative response, in order to facilitate the fitting of the positive response, the reward value of the positive response needs to be calculated.

[0094] See also Figure 7 , Figure 7 FIG. 4 is a schematic diagram showing a specific implementation process of another embodiment of the above step S4500. Figure 7 As shown, step S4500 at least further includes the following steps:

[0095] Step S4510: Obtain the sum of the average probability values ​​of the positive reply and the negative reply according to the average probability value of the positive reply and the average probability value of the negative reply.

[0096] It can be understood that the average probability value s of positive responses obtained in the above steps is c and the average probability of the negative reply s r After that, we need to sum them to get the average probability value and the following formula:

[0097] s c +s r

[0098] Step S4520: Divide the average probability value of the positive reply by the sum of the average probability values ​​to obtain the reward value of the positive reply.

[0099] It can be understood that in order to fully reflect the reward value of positive replies, the average probability value of positive replies is divided by the average probability value sum to obtain the reward value of positive replies. At this time, the reward value of positive replies is fitted to 1 as much as possible. sft The training can be maximized, so minimizing the probability loss function can minimize s r The bonus score is to expand the model to c i and r i The difference in rewards encourages the model to generate responses with larger reward scores. The reward value for a positive response is shown in the following formula:

[0100]

[0101] Step S4600: Perform negative logarithm processing on the reward value to obtain a probability loss function.

[0102] It can be understood that, consistent with the above step S3510, by performing negative logarithmic processing on the reward value, the probability loss function loss is obtained. contrast As shown in the following formula:

[0103]

[0104] Step S5000: Obtain the training loss function of the pre-trained model according to the autoregressive loss function, the stop sign loss function and the probability loss function.

[0105] It is understandable that after obtaining the autoregressive loss function, the stop sign loss function and the probability loss function in the above steps, in order to integrate instruction learning and preference learning into one training process in a multi-task training manner, it is necessary to add the above loss functions. Specifically, the training loss function loss of the pre-trained model is shown in the following formula:

[0106] loss=loss sft +loss stop +loss contrast =loss sft +loss c +loss r +loss contrast

[0107] See also Figure 8 , Figure 8 It is a structural diagram of a large model training device 600 based on preference learning provided in an embodiment of the present application. The entire process of the model training method based on preference learning provided in an embodiment of the present application involves the following modules in the large model training device based on preference learning: an acquisition module 610, an autoregressive loss module 620, a setting module 630, a probability loss module 640 and a training loss module 650.

[0108] The acquisition module 610 is used to acquire a training data set, wherein the training data set is composed of a plurality of training samples, and the training samples include training questions, positive responses, and negative responses;

[0109] An autoregressive loss module 620, for obtaining an autoregressive loss function for positive responses based on the training data set and the vocabulary of the pre-trained model;

[0110] A setting module 630, configured to set stop signs corresponding to positive responses and negative responses in the vocabulary, and calculate a loss function of the stop signs;

[0111] A probability loss module 640 is used to obtain a probability loss function according to the probability values ​​of positive responses and negative responses in the training data set;

[0112] The training loss module 650 is used to obtain the training loss function of the pre-trained model according to the autoregressive loss function, the stop sign loss function and the probability loss function.

[0113] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned device are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0114] Fig. 9 An electronic device 700 provided in an embodiment of the present application is shown. The electronic device 700 includes but is not limited to:

[0115] Memory 701, used for storing programs;

[0116] Processor 702 is used to execute the program stored in memory 701. When processor 702 executes the program stored in memory 701, processor 702 is used to execute the above-mentioned model training method based on preference learning.

[0117] The processor 702 and the memory 701 may be connected via a bus or other means.

[0118] The memory 701 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs and non-transitory computer executable programs, such as the model training method based on preference learning described in any embodiment of the present application. The processor 702 implements the above-mentioned model training method based on preference learning by running the non-transitory software program and instructions stored in the memory 701.

[0119] The memory 701 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store and execute the above-mentioned model training method based on preference learning. In addition, the memory 701 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 701 may optionally include a memory remotely arranged relative to the processor 702, and these remote memories may be connected to the processor 702 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The non-transient software programs and instructions required to implement the above-mentioned preference learning-based model training method are stored in the memory 701. When executed by one or more processors 702, the preference learning-based model training method provided by any embodiment of the present application is executed.

[0121] An embodiment of the present application also provides a storage medium storing computer executable instructions, which are used to execute the above-mentioned model training method based on preference learning.

[0122] In one embodiment, the storage medium stores computer executable instructions, which are executed by one or more control processors 702, for example, by a processor 702 in the above-mentioned electronic device 700, so that the above-mentioned one or more processors 702 can execute the preference learning-based model training method provided in any embodiment of the present application.

[0123] The above described embodiments are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0124] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A model training method based on preference learning, characterized in that: The following steps are involved: Acquire a training data set, wherein the training data set consists of a plurality of training samples, and the training samples include training questions, positive responses, and negative responses; Obtaining an autoregressive loss function of the positive response according to the training data set and the vocabulary of the pre-trained model; Setting stop marks corresponding to the positive reply and the negative reply in the vocabulary, and calculating a loss function of the stop marks; Obtaining a probability loss function according to the probability values ​​of the positive response and the negative response in the training data set; A training loss function of the pre-trained model is obtained according to the autoregressive loss function, the loss function of the stop mark and the probability loss function.

2. The model training method based on preference learning according to claim 1, characterized in that: The step of obtaining the autoregressive loss function of the positive response according to the vocabulary of the training data set and the pre-trained model includes: Calculate the probability value of each identifier of the positive reply in the vocabulary to obtain a first probability value; Determine whether the positive response is consistent with each identifier in the vocabulary to obtain a second probability value; The autoregressive loss function is obtained according to the first probability value and the second probability value.

3. The model training method based on preference learning according to claim 1, characterized in that: The step of setting stop marks corresponding to the positive reply and the negative reply in the vocabulary and calculating the loss function of the stop marks includes: Setting a positive stop mark corresponding to the positive reply and a negative stop mark corresponding to the negative reply in the vocabulary; Adding the positive stop mark and the negative stop mark to the positive reply and the negative reply respectively; Calculate the output probability of the positive stop sign in the pre-trained model to obtain a third probability value; Calculate the output probability of the negative stop sign in the pre-trained model to obtain a fourth probability value; A loss function of the stop sign is obtained according to the third probability value and the fourth probability value.

4. The model training method based on preference learning according to claim 3, characterized in that: The step of obtaining the loss function of the stop sign according to the third probability value and the fourth probability value includes: Obtaining a negative log-likelihood loss function of the positive response according to the third probability value; According to the fourth probability value, a negative log-likelihood loss function of the negative reply is obtained; The negative log-likelihood loss function of the positive reply and the negative log-likelihood loss function of the negative reply are summed to obtain the loss function of the stop sign.

5. The model training method based on preference learning according to claim 1, characterized in that: The obtaining a probability loss function according to the probability values ​​of the positive response and the negative response in the training data set includes: Calculating and summing the occurrence probability values ​​of the positive responses to obtain the total probability of the positive responses; Calculating the occurrence probability values ​​of the negative replies and summing them up to obtain the total probability of the negative replies; Obtaining an average probability value of the positive response according to the length of the positive response and the sum of the probabilities of the positive response; According to the length of the negative reply and the sum of the probabilities of the negative reply, an average probability value of the negative reply is obtained.

6. The model training method based on preference learning according to claim 5, characterized in that: The step of obtaining a probability loss function according to the probability values ​​of the positive response and the negative response in the training data set further includes: Obtaining a reward value for the positive response according to the average probability value of the positive response and the average probability value of the negative response; The reward value is subjected to negative logarithm processing to obtain the probability loss function.

7. The large model training method based on preference learning according to claim 6 is characterized in that: The step of obtaining the reward value of the positive reply according to the average probability value of the positive reply and the average probability value of the negative reply includes: Obtaining a sum of the average probability values ​​of the positive reply and the negative reply according to the average probability value of the positive reply and the average probability value of the negative reply; The reward value of the positive reply is obtained by dividing the average probability value of the positive reply by the sum of the average probability values.

8. A large model training device based on preference learning, characterized in that: include: An acquisition module is used to acquire a training data set, wherein the training data set is composed of a plurality of training samples, and the training samples include training questions, positive responses, and negative responses; An autoregressive loss module, used to obtain an autoregressive loss function of the positive response according to the training data set and the vocabulary of the pre-trained model; A setting module, used for setting stop marks corresponding to the positive reply and the negative reply in the vocabulary, and calculating a loss function of the stop marks; A probability loss module, used to obtain a probability loss function according to the probability values ​​of the positive response and the negative response in the training data set; A training loss module is used to obtain a training loss function of a pre-trained model according to the autoregressive loss function, the loss function of the stop mark and the probability loss function.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the model training method based on preference learning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the model training method based on preference learning as described in any one of claims 1 to 7 is implemented.

Images