Large language model alignment method and device, electronic equipment and readable storage medium

Abstract

The invention provides a large language model alignment method and device, electronic equipment and a readable storage medium, and relates to the technical field of machine learning. The method comprises the steps of obtaining an initial preference data set corresponding to a to-be-aligned large language model; based on the subsets of the plurality of initial preference data sets, training to obtain a reward model corresponding to the subset of each initial preference data set; determining a noise reduction model in the reward model; dividing the initial preference data set into a high-noise preference data set and a low-noise preference data set based on the noise reduction model; determining a first loss function corresponding to the high-noise preference data set and a second loss function corresponding to the low-noise preference data set, and optimizing the reward model based on the first loss function and the second loss function; and performing alignment training on the to-be-aligned large language model based on the optimized reward model. According to the invention, the influence of noise data on reward model training is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a large language model alignment method, device, electronic device and readable storage medium. Background Art

[0002] In the research and application of artificial intelligence and large language models (LLM), value alignment refers to aligning the "values" of artificial intelligence with human values ​​to ensure that artificial intelligence acts in a way that is harmless to humans and society, and avoids interference and harm to human rights. This process of value alignment is a key process to ensure that the behavior of the artificial intelligence system matches the developer's intentions and user expectations.

[0003] In related technologies, the LLM method of reinforcement learning from human feedback (RLHF) is mainly used to train value alignment, which is implemented as follows: the reward model is trained based on the collected manually annotated preference data set; the LLM is guided to perform value alignment training based on the reward model. In this process, the reward model plays a vital role. It aims to serve as an accurate proxy for human preferences, infer human values ​​and intentions based on preference data, and then guide LLM's behavior to be more closely in line with human preferences.

[0004] In the above process, RLHF relies on preference data from human feedback to train the reward model and use it to guide the value optimization path of LLM. However, when human annotators handle labeling tasks, they may make labeling errors due to a variety of factors such as lack of interest, lack of concentration, time constraints, or inherent biases, which leads to the problem of low consistency in the trained reward model, resulting in low accuracy of LLM value alignment. Summary of the invention

[0005] In view of this, the embodiments of the present disclosure provide a large language model alignment method, device, electronic device and readable storage medium to solve the problem that when human annotators handle labeling tasks, they may make labeling errors due to various factors such as lack of interest, inattention, time constraints or inherent biases, which may lead to low consistency of the trained reward model and low accuracy of LLM value alignment.

[0006] According to a first aspect of the present disclosure, a large language model alignment method is provided, comprising: obtaining an initial preference data set corresponding to a large language model to be aligned, wherein the preference data in the initial preference data set is manually annotated output data of the large language model to be aligned; based on multiple subsets of the initial preference data sets, training a reward model corresponding to each subset of the initial preference data set; the input of the reward model is the preference data, and the output is a reward score of the reward model for the preference data, and the reward score is used to measure the degree of conformity of the preference data with human values; determining a denoising model in the reward model; dividing the initial preference data set into a high-noise preference data set and a low-noise preference data set based on the denoising model; determining a first loss function corresponding to the high-noise preference data set and a second loss function corresponding to the low-noise preference data set, and optimizing the reward model based on the first loss function and the second loss function; and performing alignment training on the large language model to be aligned based on the optimized reward model.

[0007] According to a second aspect of the present disclosure, a large language model alignment device is provided, comprising: a data acquisition module, used to obtain an initial preference data set corresponding to a large language model to be aligned, wherein the preference data in the initial preference data set is the output data of the large language model to be aligned that has been manually annotated; a reward model determination module, used to train a reward model corresponding to each subset of the initial preference data set based on multiple subsets of the initial preference data set; the input of the reward model is the preference data, and the output is a reward score of the reward model for the preference data, and the reward score is used to measure the degree of conformity of the preference data with human values; a denoising model determination module, used to determine a denoising model in the reward model; a noise data division module, used to divide the initial preference data set into a high-noise preference data set and a low-noise preference data set based on the denoising model; a reward model optimization module, used to determine a first loss function corresponding to the high-noise preference data set and a second loss function corresponding to the low-noise preference data set, and optimize the reward model based on the first loss function and the second loss function; a large language model alignment training module, used to perform alignment training on the large language model to be aligned based on the optimized reward model.

[0008] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to perform the above-mentioned large language model alignment method.

[0009] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned large language model alignment method.

[0010] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0011] In the large language model alignment method provided in the exemplary embodiment of the present disclosure, an initial preference data set corresponding to the large language model to be aligned is obtained, and the preference data in the initial preference data set is the output data of the large language model to be aligned that has been manually annotated; based on a subset of multiple initial preference data sets, a reward model corresponding to each subset of the initial preference data set is trained, wherein the input of the reward model is the preference data, and the output is the reward score of the reward model for the preference data, and the reward score is used to measure the degree of conformity of the preference data with human values; determine the denoising model in the reward model; divide the initial preference data set into a high-noise preference data set and a low-noise preference data set based on the denoising model; determine the first loss function corresponding to the high-noise preference data set, and the second loss function corresponding to the low-noise preference data set, and optimize the reward model based on the first loss function and the second loss function; perform alignment training on the large language model to be aligned based on the optimized reward model. In the large language model alignment method provided in the embodiment of the present disclosure, on the one hand, the initial preference data set is divided into a high-noise preference data set and a low-noise preference data set by the denoising model, and the first loss function corresponding to the high-noise preference data set and the second loss function corresponding to the low-noise preference data set are determined respectively to optimize the reward model. By automatically identifying noisy data and adopting different processing strategies for high noise and low noise, the impact of incorrect or ambiguous human-labeled data during reward model training can be weakened, thereby saving labor costs while optimizing the training effect of the reward model. On the other hand, alignment training of the large language model based on the optimized reward model helps to improve the accuracy and efficiency of the value alignment of the large language model.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0014] Figure 1 A flowchart of a large language model alignment method according to an exemplary embodiment of the present disclosure is shown;

[0015] Figure 2A flowchart of a method for training a reward model based on subsets of multiple initial preference data sets in a large language model alignment method according to an exemplary embodiment of the present disclosure is shown;

[0016] Figure 3 A flow chart of a method for determining a denoising model in a reward model in a large language model alignment method according to an exemplary embodiment of the present disclosure is shown;

[0017] Figure 4 A flow chart of a method for determining a loss function in a large language model alignment method according to an exemplary embodiment of the present disclosure is shown;

[0018] Figure 5 A schematic block diagram of a large language model alignment device according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0022] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0025] The following describes the aspects of the present disclosure with reference to the accompanying drawings for explanation.

[0026] The present disclosure first provides a large language model alignment method. Figure 1 As shown, the large language model alignment method specifically includes the following steps:

[0027] S110: Acquire an initial preference data set corresponding to the large language model to be aligned, where the preference data in the initial preference data set is manually annotated output data of the large language model to be aligned;

[0028] S120: Based on the subsets of the multiple initial preference data sets, a reward model corresponding to each subset of the initial preference data set is trained; the input of the reward model is the preference data, and the output is a reward score of the reward model for the preference data, and the reward score is used to measure the degree of conformity between the preference data and human values;

[0029] S130: Determine a noise reduction model in the reward model;

[0030] S140: dividing the initial preference data set into a high-noise preference data set and a low-noise preference data set based on the noise reduction model;

[0031] S150: Determine a first loss function corresponding to the high noise preference data set and a second loss function corresponding to the low noise preference data set, and optimize the reward model based on the first loss function and the second loss function;

[0032] S160: Perform alignment training on the large language model to be aligned based on the optimized reward model.

[0033] In the large language model alignment method provided in the embodiment of the present disclosure, on the one hand, the initial preference data set is divided into a high-noise preference data set and a low-noise preference data set through a denoising model, and the first loss function corresponding to the high-noise preference data set and the second loss function corresponding to the low-noise preference data set are determined respectively to optimize the reward model. By automatically identifying noise data and adopting different processing strategies for high noise and low noise, the impact of human-labeled data errors or ambiguities during the training of the reward model can be weakened, while optimizing the training effect of the reward model, saving labor costs; on the other hand, alignment training of the large language model based on the optimized reward model helps to improve the accuracy and efficiency of the value alignment of the large language model.

[0034] Next, in another embodiment, the above steps are described in more detail.

[0035] In step S110, an initial preference data set corresponding to the large language model to be aligned is obtained, where the preference data in the initial preference data set is manually annotated output data of the large language model to be aligned.

[0036] In the embodiments of the present disclosure, the above-mentioned large language model (LLM) refers to a deep learning model trained using a large amount of text data. In the field of natural language processing, the above-mentioned large language model can learn language patterns, grammar and semantics by processing a large amount of text data, thereby understanding and generating human language.

[0037] In the disclosed embodiment, the large language model to be aligned is a large language model that needs to be aligned with human values. Value alignment refers to aligning the "values" of artificial intelligence with human values ​​to ensure that artificial intelligence acts in a way that is harmless to humans and society, and avoids interference and harm to human rights. The process of value alignment is a key process to ensure that the behavior of the artificial intelligence system matches the developer's intentions and user expectations.

[0038] In the disclosed embodiments, for example, a reinforcement learning from human feedback (RLHF) algorithm can be used to align the values ​​of the large language model to be aligned. Specifically, the implementation of guiding the alignment training of the large language model based on RLHF can be as follows: a reward model is trained using a large amount of preference data collected from human crowdsourcing annotators; and the language model is adjusted through reinforcement learning technology. In this process, the reward model plays a vital role. It is intended to serve as an accurate proxy for human preferences, infer human values ​​and intentions based on preference data, and then guide the behavior of the large LLM to more closely match human preferences.

[0039] In the disclosed embodiment, the above-mentioned initial preference data set is a set of preference data collected from human crowdsourced annotators in the above-mentioned RLHF. Exemplarily, the preference data may include the output data of the large language model to be aligned, and the manual annotation information of the output data. That is, after obtaining the output result of a certain input through the above-mentioned large language model to be aligned, the output result can be manually marked as a reply selected by a human or a reply rejected by a human.

[0040] In the embodiment of the present disclosure, the initial preference data set may include paired preference data. For example, for any given prompt word x, the large language model to be aligned may generate two different responses y c and r , where y c Represents the responses selected by human annotation, y r Represents replies that were rejected by human annotation.

[0041] In step S120, based on subsets of multiple initial preference data sets, a reward model corresponding to each subset of the initial preference data set is trained; the input of the above reward model is the preference data, and the output is the reward score of the reward model for the preference data, and the reward score is used to measure the degree of conformity between the preference data and human values.

[0042] In the disclosed embodiment, the reward model is used to evaluate the degree of conformity between the output data of the large language model to be aligned and human values.

[0043] In the disclosed embodiment, the above-mentioned reward model can be a reward model obtained by training based on the initial preference data set in the above-mentioned RLHF algorithm. The input data of the reward model is the preference data in the above-mentioned initial preference data set, and the output data is the reward score of the reward model for the preference data. The higher the reward score, the higher the degree of conformity of the corresponding preference data with human values.

[0044] In the embodiment of the present disclosure, based on the subsets of the multiple initial preference data sets, the reward model corresponding to each subset of the initial preference data set is trained, such as Figure 2 As shown, it can be achieved through steps S210 to S230:

[0045] S210: Divide the initial preference data set into a training data set and a verification data set based on a preset ratio.

[0046] In the disclosed embodiment, the above training data set is used to train the reward model.

[0047] In the embodiments of the present disclosure, the validation data set may be used to analyze the output distribution of the reward model trained based on the training data set, and may also be used to evaluate the generalization ability of the reward model.

[0048] In the embodiment of the present disclosure, the above preset ratio can be set based on experience and / or actual scenarios.

[0049] S220: Arrange and combine the preference data in the training data set to obtain a plurality of different subsets of the initial preference data sets.

[0050] In the embodiment of the present disclosure, the subset of the initial preference data set is a different data subset obtained by arranging and combining the preference data in the training data set.

[0051] In the disclosed embodiment, the above permutation and combination of the preference data in the training data set is a permutation and combination with replacement. For example, assuming that there are 100 pieces of preference data in the above training data set, 60 pieces of preference data are randomly selected from the 100 pieces of preference data each time to obtain a subset of the initial preference data set; the 60 pieces of preference data randomly selected last time are put back into the initial preference data set, and 60 pieces of preference data are randomly selected again to obtain another subset of the initial preference data set. And so on, until all the required subsets of the initial preference data set are obtained.

[0052] In the disclosed embodiment, by fusing the subsets of the above-mentioned initial preference data set in different arrangements and combinations, the diversity of the obtained data subsets can be ensured, thereby providing a multi-angle perspective for subsequent reward model training.

[0053] S230: Based on each subset of the initial preference data set, a corresponding reward model is trained.

[0054] In the disclosed embodiment, for each subset of the initial preference data set obtained above, a corresponding reward model is obtained based on training of the subset of the initial preference data set.

[0055] In the embodiment of the present disclosure, illustratively, assuming that N subsets of the initial preference data sets are obtained through the above steps S210 and S220, then N reward models can be trained accordingly.

[0056] In step S130 , a denoising model in the reward model is determined.

[0057] In an embodiment of the present disclosure, the noise reduction model is used to divide a subset of the initial preference data set into a high-noise preference data set and a low-noise preference data set.

[0058] In the process of training the reward model mentioned above, the initial preference dataset used is a manually annotated dataset. When human annotators handle the annotation task, they may make annotation errors due to a variety of factors such as lack of interest, lack of concentration, time constraints or inherent biases. In addition, since annotators are often paid based on the amount of annotation, there may be a motivation to minimize efforts to improve efficiency, which may cause annotation errors to show a certain correlation between different annotators. For example, annotators may prefer to choose examples that avoid difficult problems or lack substantive content to meet specific annotation constraints. In addition, cognitive biases, common misunderstandings and false memories can also affect the quality of annotations. Even with complete information and sufficient time, people may still give incorrect feedback when evaluating complex data. Based on the above factors, the initial preference dataset may be full of errors and fuzzy preferences, which in turn makes the trained reward model face the problem of low consistency. Improving this problem by improving the quality of annotators may significantly increase the data cost in the model training process.

[0059] In order to solve the above problems, the embodiments of the present disclosure can identify and isolate noise data from the initial preference data set through the above denoising model, thereby improving the training quality and generalization ability of the reward model. Specifically, the denoising model can be used to determine the high-noise data and low-noise data in the initial preference data, so that differentiated data processing strategies and loss functions can be applied according to the identified noise levels, effectively reducing the interference of noise data on the reward model training.

[0060] In the embodiment of the present disclosure, after the above step S230 of "training a corresponding reward model based on a subset of each initial preference data set", as follows Figure 3 As shown, the process of determining the noise reduction model in the reward model can also be implemented by following steps S310 and S320:

[0061] S310: Input the preference data in the verification data set into each reward model to obtain the output result of each reward model.

[0062] In the disclosed embodiment, by inputting the preference data in the verification data set into each reward model, the reward score of each reward model for the preference data in the verification data set is obtained.

[0063] In the disclosed embodiment, for N reward models trained based on subsets of N initial preference data sets, by inputting preference data in a verification data set into the N reward models, N reward scores corresponding to the preference data in the verification data set can be obtained.

[0064] S320: Determine a noise reduction model in each reward model based on the distribution of the output results of each reward model.

[0065] In the disclosed embodiment, the distribution of the output results of each reward model can be used to determine the denoising model in each reward model as follows: determine the identical distribution model in the reward model based on the distribution of the output results, and use the identical distribution model as the denoising model. Exemplarily, the distribution of the output results of each reward model can be analyzed by visualization and other means, and the output difference of each reward model can be obtained through comparison and evaluation; based on the output difference of each reward model, the identical distribution model with high consistency in output distribution is screened out from each reward model, and the obtained identical distribution model is used as the above-mentioned denoising model.

[0066] In the embodiments of the present disclosure, illustratively, for N reward models obtained by training based on subsets of N initial preference data sets, a validation data set can be used to analyze the output distribution of the N reward models, and the output differences of each model can be compared and evaluated through visualization and other means, and M identically distributed models with high consistency in output distribution can be screened out, where M≤N.

[0067] In the disclosed embodiment, the noise reduction model determined by the above steps S310 and S320 can more accurately represent human preferences under consensus, so based on the noise reduction model, high-noise data and low-noise data in the initial preference data set can be distinguished more accurately.

[0068] In step S140 , the initial preference data set is divided into a high-noise preference data set and a low-noise preference data set based on the noise reduction model.

[0069] In the disclosed embodiment, the high-noise preference data set is a set of high-noise data in the initial preference data set. The high-noise preference data set contains a large number of labeling errors, which will have a great impact on the accuracy and efficiency of reward model learning.

[0070] In the embodiment of the present disclosure, the low-noise preference data set is a set of low-noise data in the initial preference data set, and the low-noise preference data set contains fewer human annotation errors.

[0071] In the embodiments of the present disclosure, the above-mentioned division of the initial preference data set into a high-noise preference data set and a low-noise preference data set based on the noise reduction model can be implemented as follows: for each preference data in the initial preference data, the preference data is input into the noise reduction model to obtain a reward score of the noise reduction model for the preference data; based on the reward score, the initial preference data set is divided into a high-noise preference data set and a low-noise preference data set.

[0072] In the embodiments of the present disclosure, the above-mentioned division of the initial preference data set into a high-noise preference data set and a low-noise preference data set based on the reward score can be implemented as follows: for each preference data in the initial preference data, the mean value and standard deviation of the reward score are calculated; the preference data whose mean value and standard deviation of the reward score reach a preset threshold are divided into the low-noise preference data set, and the preference data whose mean value and standard deviation of the reward score do not reach the preset threshold are divided into the high-noise preference data set.

[0073] In the embodiment of the present disclosure, after M noise models with high consistency in output distribution are screened out from N reward models through step S320, the reward scores of the preference data in the initial preference data set are calculated according to the M identically distributed denoising models, and the average reward score and standard deviation of each preference data sample are calculated; each preference data sample is sorted according to the average reward score and standard deviation of each preference data sample, and all preference data are divided into two parts: low noise and high noise.

[0074] In the embodiment of the present disclosure, by dividing the initial preference data set into a high-noise preference data set and a low-noise preference data set based on a denoising model, high-noise data that may cause overfitting or insufficient generalization ability can be identified and isolated, and specific processing strategies can be applied to them, thereby improving the generalization ability of the reward model.

[0075] In step S150, a first loss function corresponding to the high noise preference data set and a second loss function corresponding to the low noise preference data set are determined, and the reward model is optimized based on the first loss function and the second loss function.

[0076] In the embodiment of the present disclosure, the above-mentioned first loss function is a loss function determined based on the preference data in the high-noise data set. The first loss function can be used to optimize the network parameters of the reward model, thereby reducing the negative impact caused by labeling errors, and helping the reward model to more robustly learn the true distribution in the data rather than overfitting to the noise labels.

[0077] In the embodiment of the present disclosure, the above-mentioned first loss function is a loss function determined based on the preference data in the low-noise data set, and the second loss function can be used to optimize the network parameters of the reward model, and optimize the adaptability and prediction accuracy of the reward model to complex data distribution.

[0078] In the embodiment of the present disclosure, for the keywords input into the large language model to be aligned, the initial preference data set includes the first preference data and the second preference data corresponding to the keywords. The first preference data is the selected preference data, and the second preference data is the unselected preference data. That is, for any given prompt word x, the large language model to be aligned can generate two different responses y cand r , where y c Represents the responses selected by human annotation, y r Represents replies that were rejected by human annotation.

[0079] In the embodiment of the present disclosure, the first loss function corresponding to the high noise preference data set and the second loss function corresponding to the low noise preference data set are determined as follows: Figure 4 As shown, it can be implemented by the following steps S410 to S430:

[0080] S410: Determine a first reward value corresponding to the first preference data and a second reward value corresponding to the second preference data based on the reward model.

[0081] In the embodiment of the present disclosure, the first reward value is output data obtained by inputting the first preference data into the reward model.

[0082] In the embodiment of the present disclosure, the second reward value is output data obtained by inputting the second preference data into the reward model.

[0083] In the embodiment of the present disclosure, for any given prompt word x, the large language model to be aligned can generate two different responses y c and r , where y c Represents the responses selected by human annotation, y r Represents a response that is rejected by a human. Further, the reward model can score these two responses separately and generate the first reward value R(y c ,x) and the second reward value R(y r ,x).

[0084] S420: For the low noise preference data set, based on the first reward value and the second reward value, determine a first loss function by using a maximum likelihood estimation algorithm.

[0085] In the embodiments of the present disclosure, the above-mentioned determination of the first loss function through the maximum likelihood estimation algorithm based on the first reward value and the second reward value can be implemented as follows: based on the first reward value and the second reward value, the negative log-likelihood loss of the reward model is determined through the maximum likelihood estimation algorithm; based on the first preference data and the second preference data, the target correction coefficient of the negative log-likelihood loss is determined, and the negative log-likelihood loss is corrected based on the determined target correction coefficient to obtain the first loss function.

[0086] In a specific embodiment, the process of determining the first loss function is described in detail below:

[0087] For the low-noise preference data subset, since it contains fewer human annotation errors, the embodiment of the present disclosure uses the preference learning loss function in the RLHF technology and combines it with the inverse reinforcement learning loss to train the reward model. The above process involves the comparison of paired data in the initial preference data set. Specifically, the process of determining the first loss function can be implemented as follows: For a given prompt word x, LLM generates two different responses: the response y selected by humans c And the reply y rejected by humans r ; Estimate the response y generated by the above LLM through the reward model c and r , generating a first reward value and a second reward value; based on

[0088] Two different responses y generated by the LLM for a given prompt word x through the reward model c and r (y c and r Representing the responses selected and rejected by humans respectively). The reward model will score these two responses separately and generate a reward value R(y c ,x) and R(y r ,x). Then, the probability prediction algorithm (Bradley-Terry) is used to process the first reward value and the second reward value, thereby updating the preference prediction to ensure that low-noise data can effectively contribute to the training process of the reward model. The specific implementation of updating the preference prediction based on the Bradley-Terry algorithm is as follows:

[0089]

[0090] Here, σ is the Sigmoid function (the activation function of the neural network), which is used to map the variable to between 0 and 1.

[0091] For the initial preference data set D, the network parameters of the reward model can be estimated by maximum likelihood estimation based on the above formula. Specifically, the negative log-likelihood loss obtained based on maximum likelihood estimation is:

[0092]

[0093] in, refers to the reward model, which is here represented by learned neural network parameters. Representing the neural network parameters, the disclosed embodiment can optimize the reward model through learning parameters based on the negative log-likelihood loss to adapt to different data characteristics and distributions.

[0094] Furthermore, due to the heterogeneity of the distribution of various preference data, there is a large overlap in the distribution of reward scores between the selected responses and the rejected responses output by the reward model. This overlap makes it difficult for the reward model to distinguish fuzzy samples by reward scores, affecting the overall performance of the reward model. Therefore, in order to solve this problem, the present invention modifies the negative log-likelihood loss function based on the target correction coefficient on the basis of the above-mentioned negative log-likelihood loss, and the specific implementation can be as follows:

[0095] In this specific embodiment, the selected replies are regarded as positive samples, and the rejected replies are regarded as negative samples, and an additional gradient descent step is performed on these two types of samples to expand the classification boundary between positive samples and negative samples, thereby enhancing the model's ability to distinguish these two types of samples. Specifically, the improved loss function can be expressed as:

[0096]

[0097] Among them, α represents the hyperparameter adjustment coefficient of the weight of the inverse reinforcement learning objective correction term, that is, the above-mentioned objective correction coefficient.

[0098] In this specific embodiment, a hyperparameter adjustment coefficient α of the inverse reinforcement learning regularization term weight is used to provide an adjustment mechanism for the reward model, so that the relationship between the traditional loss and the inverse reinforcement learning loss can be more flexibly balanced during the training process, thereby optimizing the model's adaptability to complex data distribution and prediction accuracy.

[0099] S430: For the high noise preference data set, based on the first reward value and the second reward value, determine a second loss function through a label smoothing algorithm.

[0100] In the disclosed embodiments, the high-noise preference dataset contains a large number of labeling errors, which have a significant impact on the accuracy and efficiency of the reward model learning. Furthermore, since traditional loss functions, such as the negative log-likelihood loss mentioned above, have a low tolerance for label noise, using traditional loss functions to train the reward model may cause the reward model to be updated in the wrong direction during the learning process, thereby affecting the performance and generalization ability of the final model.

[0101] To solve the above problem, this specific embodiment introduces label smoothing technology to modify the above negative log-likelihood loss function. The above label smoothing can reduce the excessive sensitivity of the reward model to noise labels and improve the robustness of the reward model to noise data by allocating weights between the true label and a certain proportion of uniform distribution.

[0102] The modified second loss function takes into account the existence of label noise and adjusts the target distribution of labels so that the reward model can handle high-noise data more smoothly during training. Specifically, the above second loss function can be expressed as:

[0103]

[0104] Among them, β∈[0,1] represents the proportional hyperparameter of label smoothing. The closer the value of this parameter is to 1, the greater the noise is and the lower the credibility of the corresponding preference data is.

[0105] In this specific embodiment, by introducing label smoothing technology, not only can the negative impact caused by labeling errors be reduced, but also the reward model can more robustly learn the true distribution in the data, rather than overfitting to the noise labels. In addition, through the application of inverse reinforcement learning technology and label smoothing technology, the ability of the reward model to process high-noise data is effectively improved, ensuring the robustness and generalization ability of the reward model in the face of complex and uncertain labeling information, thereby achieving more accurate and reliable performance in practical applications.

[0106] In the embodiments of the present disclosure, in view of the significant difference in error levels between the high-noise preference data set and the low-noise preference data set, a specific training loss function is designed for each type of data, and a differentiated target correction term is introduced into the preference learning loss function of the RLHF technology to optimize the performance of the loss function.

[0107] In the embodiment of the present disclosure, after determining the first loss function corresponding to the high noise preference data set and the second loss function corresponding to the low noise preference data set based on steps S410 to S430, the reward model can be optimized based on the determined first loss function and second loss function.

[0108] Exemplarily, the process of adjusting the network parameters of the reward model through the gradient descent algorithm based on the second loss function can be implemented as follows: determine the loss value corresponding to each preference data in the high-noise preference data set based on the second loss function, and sort the preference data based on the loss value; when the loss value is greater than a preset threshold, delete the corresponding preference data; adjust the network parameters of the reward model through the gradient descent algorithm based on the sorting result of the preference data.

[0109] Next, in combination with the above specific embodiments, the process of optimizing the reward model based on the first loss function and the second loss function is described in detail:

[0110] In this specific embodiment, for the high-noise preference data set, the reward model can be optimized through the following data screening mechanism: divide the high-noise preference data set into small batches of data subsets; sort the preference data in the subsets based on the loss value, and identify and prioritize those data with smaller losses based on the sorting results, that is, they are considered to be relatively "clean" data, and then apply gradient back propagation to this part of the data to optimize the network parameters. The above method introduces a data selection criterion based on the loss size in the optimization process of the reward model, ensuring that the reward model pays more attention to higher-quality data during the learning process.

[0111] In addition, this specific embodiment can also set a preset threshold based on the expected noise level, where the preset threshold is the deletion ratio of high-noise data. By deleting the preference data with large losses, the loss function can be adjusted more specifically. This strategy not only improves the efficiency of reward model training, but also improves the generalization ability of the reward model by reducing dependence on high-noise data.

[0112] The specific implementation of adjusting the network parameters of the reward model through the gradient descent algorithm based on the first loss function is the same as the prior art and will not be repeated here.

[0113] In step S160, alignment training is performed on the large language model to be aligned based on the optimized reward model.

[0114] In the disclosed embodiment, after optimizing the reward model, the optimized reward model can be used as a guidance signal to further perform alignment training on the LLM in combination with a reinforcement learning algorithm. Exemplarily, the reinforcement learning algorithm can be an algorithm such as proximal strategy optimization.

[0115] In the large language model alignment method provided in the embodiment of the present disclosure, on the one hand, the initial preference data set is divided into a high-noise preference data set and a low-noise preference data set through a denoising model, and the first loss function corresponding to the high-noise preference data set and the second loss function corresponding to the low-noise preference data set are determined respectively to optimize the reward model. By automatically identifying noise data and adopting different processing strategies for high noise and low noise, the impact of human-labeled data errors or ambiguities during the training of the reward model can be weakened, while optimizing the training effect of the reward model, saving labor costs; on the other hand, alignment training of the large language model based on the optimized reward model helps to improve the accuracy and efficiency of the value alignment of the large language model.

[0116] The above mainly introduces the solution provided by the embodiment of the present disclosure from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed in this article, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0117] The disclosed embodiment can divide the functional modules of the large language model alignment device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or software functional modules.

[0118] The present disclosure provides a large language model alignment device. Figure 5 As shown, the large language model alignment device 500 may include a data acquisition module 510, a reward model determination module 520, a noise reduction model determination module 530, a noise data division module 540, a reward model optimization module 550 and a large language model alignment training module 560, wherein:

[0119] A data collection module 510 is used to obtain an initial preference data set corresponding to the large language model to be aligned, where the preference data in the initial preference data set is manually annotated output data of the large language model to be aligned;

[0120] A reward model determination module 520 is used to train a reward model corresponding to each subset of the initial preference data set based on the subsets of the multiple initial preference data sets; the input of the reward model is the preference data, and the output is the reward score of the reward model for the preference data, and the reward score is used to measure the degree of conformity between the preference data and human values;

[0121] A denoising model determination module 530, used to determine a denoising model in a reward model;

[0122] A noise data division module 540 is used to divide the initial preference data set into a high noise preference data set and a low noise preference data set based on a noise reduction model;

[0123] A reward model optimization module 550, used to determine a first loss function corresponding to the high noise preference data set and a second loss function corresponding to the low noise preference data set, and optimize the reward model based on the first loss function and the second loss function;

[0124] The large language model alignment training module 560 is used to perform alignment training on the large language model to be aligned based on the optimized reward model.

[0125] In the embodiment of the present disclosure, the above-mentioned reward model determination module is specifically used to: divide the initial preference data set into a training data set and a verification data set based on a preset ratio; arrange and combine the preference data in the training data set to obtain multiple different subsets of the initial preference data sets; based on each subset of the initial preference data set, train a corresponding reward model.

[0126] In the embodiment of the present disclosure, the above-mentioned denoising model determination module is specifically used to: input the preference data in the verification data set into each reward model to obtain the output result of each reward model; based on the distribution of the output results of each reward model, determine the denoising model in each reward model.

[0127] In the embodiment of the present disclosure, the above-mentioned denoising model determination module is specifically used to: determine the same distribution model in the reward model based on the distribution of the output results, and use the same distribution model as the denoising model.

[0128] In the embodiment of the present disclosure, the above-mentioned noise data division module is specifically used to: for each preference data in the initial preference data, input the preference data into the noise reduction model to obtain the reward score of the noise reduction model for the preference data; based on the reward score, divide the initial preference data set into a high-noise preference data set and a low-noise preference data set.

[0129] In the embodiment of the present disclosure, the above-mentioned noise data division module is specifically used to: calculate the mean value and standard deviation of the reward score for each preference data in the initial preference data; divide the preference data whose mean value and standard deviation of the reward score reach the preset threshold into the low-noise preference data set, and divide the preference data whose mean value and standard deviation of the reward score do not reach the preset threshold into the high-noise preference data set.

[0130] In an embodiment of the present disclosure, for keywords input into a large language model to be aligned, the initial preference data set includes first preference data and second preference data corresponding to the keywords, the first preference data being selected preference data, and the second preference data being unselected preference data; the above-mentioned reward model optimization module is specifically used to: determine a first reward value corresponding to the first preference data and a second reward value for the second preference data based on the reward model; for a low-noise preference data set, determine a first loss function based on the first reward value and the second reward value through a maximum likelihood estimation algorithm; for a high-noise preference data set, determine a second loss function based on the first reward value and the second reward value through a label smoothing algorithm.

[0131] In the embodiment of the present disclosure, the above-mentioned reward model optimization module is specifically used to: determine the negative log-likelihood loss of the reward model through the maximum likelihood estimation algorithm based on the first reward value and the second reward value; determine the target correction coefficient of the negative log-likelihood loss based on the first preference data and the second preference data, and correct the negative log-likelihood loss based on the determined target correction coefficient to obtain a first loss function.

[0132] In the embodiment of the present disclosure, the above-mentioned reward model optimization module is specifically used to: adjust the network parameters of the reward model through the gradient descent algorithm based on the first loss function; adjust the network parameters of the reward model through the gradient descent algorithm based on the second loss function.

[0133] In the disclosed embodiment, the above-mentioned reward model optimization module is specifically used to: determine the loss value corresponding to each preference data in the high-noise preference data set based on the second loss function, and sort the preference data based on the loss value; when the loss value is greater than a preset threshold, delete the corresponding preference data; adjust the network parameters of the reward model through the gradient descent algorithm based on the sorting result of the preference data.

[0134] In addition, the specific implementation details of the above-mentioned large language model alignment device have been described in detail in the corresponding position of the large language model alignment method, so they will not be repeated here.

[0135] It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, based on the functions implemented by each module, the embodiments of the present disclosure may also divide the above-mentioned large language model alignment device into a noise recognition module, a noise processing module, and a training optimization module. Among them:

[0136] The noise recognition module is used to identify and distinguish the noise in human-labeled data, providing a basis for subsequent processing.

[0137] The noise processing module is used to apply differentiated data processing strategies and loss functions according to the identified noise level, effectively reducing the interference of noise data on reward model training.

[0138] The training optimization module is used to optimize the training process of the reward model using a customized gradient update strategy to ensure that the model can effectively learn on data with different noise levels.

[0139] Figure 6 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 6 , which shows a structural schematic diagram of an electronic device 600 suitable for implementing the embodiments of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the large language model alignment method of the embodiment described in the present disclosure. In RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0141] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0142] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the large language model alignment method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0143] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0144] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperTextTransferProtocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an adhoc peer-to-peer network), as well as any currently known or future developed network.

[0145] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0146] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0147] Obtaining an initial preference data set corresponding to the large language model to be aligned, where the preference data in the initial preference data set is manually annotated output data of the large language model to be aligned;

[0148] Based on subsets of multiple initial preference data sets, a reward model corresponding to each subset of the initial preference data set is trained; the input of the above reward model is the preference data, and the output is the reward score of the reward model for the preference data, and the reward score is used to measure the degree of conformity between the preference data and human values;

[0149] Determine the denoising model in the reward model;

[0150] Based on the denoising model, the initial preference data set is divided into a high-noise preference data set and a low-noise preference data set;

[0151] Determine a first loss function corresponding to the high noise preference data set and a second loss function corresponding to the low noise preference data set, and optimize the reward model based on the first loss function and the second loss function;

[0152] The large language model to be aligned is trained based on the optimized reward model.

[0153] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0154] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0155] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0157] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0158] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0160] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0161] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A large language model alignment method, characterized in that: The method comprises: Acquire an initial preference data set corresponding to the large language model to be aligned, where the preference data in the initial preference data set is manually annotated output data of the large language model to be aligned; Based on a plurality of subsets of the initial preference data sets, a reward model corresponding to each subset of the initial preference data sets is trained; the input of the reward model is the preference data, and the output is a reward score of the reward model for the preference data, and the reward score is used to measure the degree of conformity of the preference data with human values; Determining a denoising model in the reward model; Dividing the initial preference data set into a high-noise preference data set and a low-noise preference data set based on the noise reduction model; Determine a first loss function corresponding to the high-noise preference data set and a second loss function corresponding to the low-noise preference data set, and optimize the reward model based on the first loss function and the second loss function; The large language model to be aligned is trained based on the optimized reward model.

2. The large language model alignment method according to claim 1, characterized in that: The training based on the subsets of the multiple initial preference data sets to obtain a reward model corresponding to each subset of the initial preference data sets includes: Dividing the initial preference data set into a training data set and a validation data set based on a preset ratio; Arrange and combine the preference data in the training data set to obtain a plurality of different subsets of the initial preference data set; Based on each subset of the initial preference data set, the corresponding reward model is trained.

3. The large language model alignment method according to claim 2, characterized in that: The determining of the denoising model in the reward model comprises: Inputting the preference data in the verification data set into each of the reward models to obtain an output result of each of the reward models; Based on the distribution of the output results of each of the reward models, a noise reduction model in each of the reward models is determined.

4. The large language model alignment method according to claim 3, characterized in that: The determining, based on the distribution of the output results of each of the reward models, a noise reduction model in each of the reward models, comprises: An identical distribution model in the reward model is determined based on the distribution of the output results, and the identical distribution model is used as the denoising model.

5. The large language model alignment method according to claim 1, characterized in that: The dividing the initial preference data set into a high-noise preference data set and a low-noise preference data set based on the noise reduction model includes: For each preference data in the initial preference data, input the preference data into the denoising model to obtain a reward score of the denoising model for the preference data; The initial preference data set is divided into the high-noise preference data set and the low-noise preference data set based on the reward score.

6. The large language model alignment method according to claim 1, characterized in that: The initial preference data set includes first preference data and second preference data, wherein the first preference data is the preference data selected by humans, and the second preference data is the preference data not selected by humans; The determining of a first loss function corresponding to the high noise preference data set and a second loss function corresponding to the low noise preference data set includes: Determine a first reward value corresponding to the first preference data and a second reward value corresponding to the second preference data based on the reward model; For the low-noise preference data set, based on the first reward value and the second reward value, determine the first loss function by a maximum likelihood estimation algorithm; For the high-noise preference data set, based on the first reward value and the second reward value, the second loss function is determined by a label smoothing algorithm.

7. The large language model alignment method according to claim 1, characterized in that: The optimizing the reward model based on the first loss function and the second loss function includes: Adjusting the network parameters of the reward model by a gradient descent algorithm based on the first loss function; The network parameters of the reward model are adjusted by the gradient descent algorithm based on the second loss function.

8. A large language model alignment device, characterized in that: The device comprises: A data collection module, used to obtain an initial preference data set corresponding to the large language model to be aligned, wherein the preference data in the initial preference data set is manually annotated output data of the large language model to be aligned; A reward model determination module, used for training a reward model corresponding to each subset of the initial preference data set based on a plurality of subsets of the initial preference data set; the input of the reward model is the preference data, and the output is a reward score of the reward model for the preference data, and the reward score is used to measure the degree of conformity of the preference data with human values; A denoising model determining module, used to determine a denoising model in the reward model; A noise data division module, used for dividing the initial preference data set into a high noise preference data set and a low noise preference data set based on the noise reduction model; A reward model optimization module, used to determine a first loss function corresponding to the high-noise preference data set and a second loss function corresponding to the low-noise preference data set, and optimize the reward model based on the first loss function and the second loss function; The large language model alignment training module is used to perform alignment training on the large language model to be aligned based on the optimized reward model.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images