Alignment model training method, information processing method and device

Abstract

The invention provides an alignment model training method and device and an information processing method and device. The invention relates to the field of artificial intelligence, in particular to the technical field of deep learning, machine learning, natural language processing and large models, aims to solve the problem of task type dialogue generation application scenes in multiple fields such as customer service systems, smart home, medical consultation and education tutoring, and particularly relates to a training method, an information processing method and a device of an alignment model. According to the specific implementation scheme, a training sample set is obtained; sFT training is carried out on the basic model based on a DPO algorithm and a PPO algorithm, an alignment model is obtained, output of the alignment model conforms to preference characteristics, and the preference characteristics are obtained based on preference data in the training sample set. According to the scheme, the performance of the alignment model can be improved, and the output accuracy of the model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to the fields of deep learning, machine learning, natural language processing, and large model technology. It aims to solve application scenarios of task-based dialogue generation in multiple fields such as customer service systems, smart homes, medical consultation, and educational counseling, and specifically to a training method, information processing method, and device for an alignment model. Background Art

[0002] In the task-based dialogue generation technology based on language models, the path is usually followed: pre-training, supervised fine-tuning (SFT), manually annotated sorting data to train the reward model (RM), and finally using the SFT model and RM model for direct preference learning (DPO) or online reinforcement learning (PPO). Although this process can make the model output more inclined to human preferences, there are problems such as catastrophic forgetting, isolation of learning goals, and low training efficiency. Therefore, it is necessary to explore new training methods to improve the efficiency and effectiveness of model training while reducing resource consumption and the risk of catastrophic forgetting. Summary of the invention

[0003] The present disclosure provides a training method, an information processing method and a device for an alignment model.

[0004] According to a first aspect of the present disclosure, a method for training an alignment model is provided, the method comprising:

[0005] Obtain a training sample set, where the training sample set includes preference data;

[0006] Based on the DPO algorithm and the PPO algorithm, the basic model is trained by SFT to obtain an alignment model, the output of which is consistent with the preference feature, which is obtained based on the preference data in the training sample set.

[0007] According to a second aspect of the present disclosure, there is provided an information processing method, comprising:

[0008] Get input information;

[0009] Input information is input into the alignment model, and a response matching the input information is obtained from the alignment model output, wherein the response is generated based on the alignment model's understanding of the input information and the preference features learned by the alignment model during training; wherein the alignment model is obtained by training using the method of the first aspect.

[0010] According to a third aspect of the present disclosure, a training device for an alignment model is provided, the device comprising: a first acquisition module, used to acquire a training sample set, the training sample set comprising preference data; a training module, used to perform SFT training on a basic model based on a DPO algorithm and a PPO algorithm to obtain an alignment model, the output of the alignment model being consistent with a preference feature, the preference feature being obtained based on the preference data in the training sample set.

[0011] According to a fourth aspect of the present disclosure, there is provided an information processing device, including:

[0012] The second acquisition module is used to obtain input information; the processing module is used to input the input information into the alignment model, and obtain a response output by the alignment model that matches the input information, and the response is generated based on the alignment model's understanding of the input information and the preference features learned by the alignment model during training; wherein the alignment model is obtained by training through the method of the first aspect.

[0013] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0017] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0018] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0019] The solution disclosed in the present invention can improve the performance of the alignment model and increase the accuracy of the model output.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is an example diagram of an alignment process in the related art;

[0023] Figure 2 is a flowchart of a method for training an alignment model according to an embodiment of the present disclosure;

[0024] Figure 3 Schematic diagram of the training data source of DPO and PPO according to the embodiment of the present disclosure Figure 1 ;

[0025] Figure 4 Schematic diagram of the training data source of DPO and PPO according to the embodiment of the present disclosure Figure 2 ;

[0026] Figure 5 is a schematic diagram of a training data ratio change curve according to an embodiment of the present disclosure;

[0027] Figure 6 is a flowchart of an information processing method according to an embodiment of the present disclosure;

[0028] Figure 7 is a structural schematic diagram of a training device for an alignment model according to an embodiment of the present disclosure;

[0029] Figure 8 is a schematic diagram of the structure of an information processing device according to an embodiment of the present disclosure;

[0030] Fig. 9 is a schematic diagram of a scenario of a method for training an alignment model according to an embodiment of the present disclosure;

[0031] Fig.10 is a schematic diagram of a scenario of an information processing method according to an embodiment of the present disclosure;

[0032] Fig.11 It is a structural schematic diagram of an electronic device used to implement the training method and / or information processing method of the alignment model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0034] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0035] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0036] Before introducing the technical solutions of the embodiments of the present disclosure, the technical terms that may be used in the present disclosure are further explained:

[0037] DPO: DPO is an optimization technique designed to align the behavior of a model directly with human preferences. It does this by directly optimizing the model's response to preference data without explicitly training a reward model or performing a complex reinforcement learning process.

[0038] PPO: PPO is a reinforcement learning algorithm that maintains the stability of training by limiting the amplitude of policy updates to avoid excessive updates that lead to degradation of model performance.

[0039] SFT: SFT is a commonly used training technique in the fields of machine learning and natural language processing (NLP). It fine-tunes the model by introducing supervised data to improve the prediction accuracy of the model in specific tasks or fields.

[0040] RM: A machine learning model that learns how to score or rank different outputs by analyzing large amounts of data (often including human feedback). In the field of NLP, this usually means evaluating generated texts to ensure that they are not only grammatically correct, but also semantically consistent with human expectations and preferences.

[0041] In related technologies, alignment learning generally includes two stages: SFT and DPO or PPO. In the SFT stage, the output mode of the model for different task types is adjusted through supervised learning, such as the output mode of lyrics and the output model of dialogue; in the DPO or PPO stage, the model output is adjusted to be more inclined to human preferences through preference data or preference signals (provided by RM online in real time), such as safe output and the organization format of replies. Figure 1 An example diagram of an alignment process is shown, Figure 1 As shown in the figure, the first step is to collect demonstration data and formulate a supervision policy; the second step is to collect comparison data and train the RM model; the third step is to use reinforcement learning to optimize the strategy for the RM model.

[0042] For the above alignment process, a serial execution method is usually adopted, that is, SFT->DPO or SFT->PPO. This method is the current mainstream approach and has achieved significant benefits. However, the above method has three major defects:

[0043] (1) Multi-stage serial execution alignment algorithm. Since the learning objectives of SFT, DPO and PPO are different, they are easily affected by the "catastrophic forgetting" problem of large models, which causes the knowledge learned in the previous stage to be forgotten in the subsequent stage, resulting in unpredictable phenomena.

[0044] (2) According to the above content, although SFT, DPO and PPO force the model to learn different knowledge respectively in alignment learning, these knowledge are continuous and can complement each other. However, the current mainstream multi-stage serial execution alignment framework ignores the mutual promotion of learning objectives at different stages, resulting in suboptimal knowledge learning at different stages.

[0045] (3) The multi-stage serial execution of the alignment algorithm means that it takes longer time and more resources to complete the entire alignment stage.

[0046] Due to the instability and high resource consumption of PPO training, current explorations into integrated preference learning frameworks mainly focus on the integration of SFT and DPO.

[0047] Odds Ratio Preference Optimization (ORPO) proposes to add a new probability ratio to increase the probability of good output and reduce the probability of bad output, thereby achieving integrated learning of supervised task fine-tuning and direct preference learning. Specifically, given an input-output sequence (x, y),

[0048] The average log-likelihood of the output y is:

[0049]

[0050] The probability of outputting y is:

[0051]

[0052] Therefore, the odds ratio for positive and negative examples can be:

[0053]

[0054] Thus, the odds ratio loss function is obtained as:

[0055]

[0056] The overall training loss function of the model is:

[0057]

[0058] where m represents the number of characters of output y, y t represents the t-th character of output y, y<t represents the first t - 1 characters of output y, θ represents the model parameters, y w and y l represent positive and negative examples respectively, σ represents the activation (Sigmoid) function, represents the SFT loss function.

[0059] Intuitive Fine-Tuning (IFT) constructs an approximate complete response of the large language model to the instruction by introducing Temporal Residual Connection (TRC) in the time dimension, enabling the model to predict one more step based on its previous prediction result, more accurately estimating the preference of the large language model, and thus obtaining better training results. This approach can also endow the large language model with an ability similar to human intuition, generating a vague prediction of the complete response whenever it sees a certain instruction. Meanwhile, TRC also indirectly introduces Dynamic Relation Propagation (DRP), which can model the impact of the currently predicted word of the large language model on all future generation results, enhancing the causality and factuality in the generation process of the large language model.

[0060] However, the integrated preference learning framework in related technologies mainly has the following disadvantages:

[0061] (1) The exploration of the current integrated preference learning framework mainly focuses on the integration of SFT and DPO, ignoring the advantages of PPO.

[0062] (2) In order to simplify the loss function, the current integrated preference learning framework for the fusion of SFT and DPO often discards the original DPO loss function and chooses to use other methods to introduce the logic of preference learning, such as the probability ratio in ORPO, which will cause the loss of direct preference learning.

[0063] (3) The current integrated preference learning framework mainly focuses on the integration of monitoring task fine-tuning and direct preference learning, ignoring the integration of the advantages of direct preference learning and online reinforcement learning.

[0064] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the present disclosure proposes a new integrated preference learning framework, which effectively integrates the advantages of SFT, DPO and PPO and realizes a grand unified alignment learning algorithm.

[0065] The present disclosure provides a method for training an alignment model. Figure 1 It is a flowchart of a method for training an alignment model according to an embodiment of the present disclosure. The method for training an alignment model can be applied to a training device for an alignment model. The training device for the alignment model is located in an electronic device. The electronic device includes but is not limited to a fixed device and / or a mobile device. For example, a fixed device includes but is not limited to a server, and the server can be a cloud server or an ordinary server. For example, a mobile device includes but is not limited to: a mobile phone, a tablet computer, and a vehicle-mounted terminal. In some possible implementations, the method for training an alignment model can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the training method of the alignment model includes:

[0066] S101, obtaining a training sample set, where the training sample set includes preference data;

[0067] S102. Based on the DPO algorithm and the PPO algorithm, the basic model is trained by SFT to obtain an alignment model, the output of which is consistent with the preference feature, and the preference feature is obtained based on the preference data in the training sample set.

[0068] In the disclosed embodiment, the training sample set is the basis for training the alignment model, which includes a large amount of preference data. These data can be data annotated by multiple annotators to reflect the preference relationship between different inputs and outputs, or data output by the model collected online, using the input and output of the model as data reflecting the preference relationship.

[0069] In the disclosed embodiments, the base model may be any type of machine learning model or deep learning model, and the specific selection depends on the requirements of the task and the characteristics of the data, such as a generative dialogue model.

[0070] In the embodiment of the present disclosure, the alignment model can be a generative dialogue model. The application fields of the alignment model include but are not limited to intelligent e-commerce, education guidance, medical consultation, legal services, navigation services, etc.

[0071] In some embodiments, obtaining a training sample set includes: first, it is necessary to collect a large amount of conversation data as a training sample set. These data can come from multiple sources, such as historical conversation records, user surveys, online forums, or professional annotation teams. Then, preference data annotation: in the collected conversation data, it is necessary to clearly mark the preference relationship between the input (user questions or instructions) and the output (model response). This usually involves manual evaluation or rule-based automatic annotation to identify which responses are more in line with user expectations or preferences. Finally, construct a training sample set: organize the annotated conversation data into a structured training sample set, each sample contains input text, output text, and corresponding preference labels or scores.

[0072] In some implementations, preference features are extracted from the training sample set. These features are used to describe the preference relationship between input and output and are an important basis for subsequent training models.

[0073] In some embodiments, the basic model is initialized, and a suitable basic model is selected or designed as the starting point of training. The basic model is optimized using the DPO algorithm, and the probability distribution of the model is adjusted to better fit the preference data. The DPO algorithm focuses on the relative preference relationship between outputs, and improves the prediction accuracy of the model by optimizing the probability distribution. At the same time, the basic model is trained using the PPO algorithm. The PPO algorithm maintains the stability of training by limiting the amplitude of policy updates to avoid excessive updates that lead to degradation of model performance. The PPO algorithm focuses on policy optimization and finds the optimal policy by maximizing cumulative rewards. Based on DPO and PPO training, SFT training is introduced to fine-tune the basic model. SFT training fine-tunes the model in a supervised manner by introducing preference data, so that the model can more accurately output results that are consistent with the preference characteristics.

[0074] In some embodiments, an independent test set is used to evaluate the trained alignment model to check whether the performance of the model meets the requirements. The model is adjusted and optimized according to the evaluation results, including adjusting hyperparameters, improving model structure, etc.

[0075] By combining algorithms such as DPO and PPO and performing SFT training on the basic model, these algorithms can optimize the performance of the model from different angles and improve the quality and diversity of generated responses. Through this process, a generative dialogue model with strong generation capabilities and high domain adaptability can be constructed to provide intelligent dialogue solutions for all walks of life. For example, on e-commerce platforms, the alignment model can be used as a customer service robot to have natural conversations with users, answer users' questions and doubts, and improve customer service efficiency. Online education platforms can use alignment models to provide students with personalized tutoring and Q&A services to help students better understand and master knowledge. In the medical field, alignment models can assist doctors in making preliminary diagnoses and treatment recommendations, providing patients with more convenient and efficient medical services.

[0076] For example, suppose we train a generative dialogue model for e-commerce customer service. In the training sample set, we collected a large amount of dialogue data of users asking about product information, return and exchange policies, etc., and marked which replies can more accurately and friendly solve user problems. In the SFT training phase, we use a combination of DPO and PPO algorithms to let the model learn how to generate replies based on user input that are both consistent with product information and reflect the professional attitude of customer service.

[0077] The technical solution of the disclosed embodiment can effectively improve the performance of the alignment model by combining the DPO algorithm, the PPO algorithm and the SFT training. The DPO algorithm and the PPO algorithm optimize the strategy from different angles. The former focuses on the optimization of the probability distribution, while the latter maintains the stability of the training by limiting the amplitude of the strategy update. The SFT training fine-tunes the basic model by introducing preference data, so that the final alignment model can more accurately output results that match the preference characteristics. This joint training method not only improves the generalization ability of the model, but also enhances the accuracy of the model when dealing with complex preference relationships.

[0078] In some embodiments, based on the DPO algorithm and the PPO algorithm, SFT training is performed on the basic model, including: obtaining a first loss function of the DPO algorithm for the basic model; obtaining a second loss function of the PPO algorithm for the basic model; obtaining a third loss function of the SFT training for the basic model; determining a total loss function based on the first loss function, the second loss function and the third loss function; and updating the parameters of the basic model based on the total loss function.

[0079] In the disclosed embodiment, the DPO loss function is: according to the principle of the DPO algorithm, the loss of the basic model in optimizing the probability distribution is calculated. This usually involves comparing the difference between the probability distribution predicted by the model and the true probability distribution.

[0080] In the disclosed embodiment, the PPO loss function: uses the PPO algorithm to calculate the loss during the policy update, ensuring that the policy update can both improve performance and not be too drastic to cause training instability. The PPO loss function usually includes a policy improvement term and a policy constraint term.

[0081] In the disclosed embodiment, SFT loss function: During the SFT training phase, the loss between the model output and the true label is calculated based on the labeled data. This is usually a supervised loss function, such as cross entropy loss.

[0082] In some embodiments, determining a total loss function based on the first loss function, the second loss function, and the third loss function includes: selecting a suitable basic model, such as a pre-trained language model based on a codec (Transformer) structure. Preparing the data sets and parameter settings required for DPO, PPO, and SFT training. Adding the loss functions of DPO, PPO, and SFT according to certain weights to obtain a total loss function. The selection of weights can be adjusted according to specific tasks and data sets.

[0083] In some embodiments, the formula of the total loss function can be expressed as:

[0084]

[0085] in, represents the joint loss function of the DPO algorithm and the PPO algorithm, represents the first loss function (i.e. the loss function using DPO), represents the second loss function (i.e. the loss function using PPO), represents the third loss function (i.e., the loss function using SFT), θ represents the model parameters, and λ 1 represents an adjustable hyperparameter, λ 2 represents a tunable hyperparameter.

[0086] In some embodiments, the parameters of the base model are updated based on the total loss function, including: using an optimization algorithm such as gradient descent to update the parameters of the base model based on the total loss function. During the update process, it is necessary to ensure that the gradient is not too large to avoid gradient explosion or vanishing problems during training. Repeat the above steps until the performance of the model on the validation set is no longer significantly improved or the preset training rounds are reached.

[0087] For example, suppose we are training a generative dialogue model to answer users' questions about travel. First, choose a language model as the basis. Then, prepare the dataset: collect a large amount of dialogue data about travel and annotate it, including questions, replies, and possible intent and sentiment labels. Then, use the DPO and PPO algorithms to perform SFT training on the language model, and use the gradient descent algorithm to update the model parameters to minimize the total loss function until the model performs satisfactorily on the validation set. In this way, we can train a generative dialogue model that can accurately understand user intent and generate replies that are consistent with the context and style of the tourism field.

[0088] In this way, training the basic model by combining the DPO algorithm, PPO algorithm and SFT training can significantly improve the performance of the model. This joint training strategy can fully utilize the advantages of DPO in probability distribution optimization, the advantages of PPO in policy stability, and the fine-tuning ability of SFT on specific task data. By comprehensively considering these three loss functions, the model can learn more accurate, stable and task-specific representations and strategies, thereby showing better results in multiple fields such as generative dialogue, recommendation systems, and natural language understanding.

[0089] In some embodiments, SFT training is performed on the basic model based on the DPO algorithm and the PPO algorithm, including: determining a joint training method of the DPO algorithm and the PPO algorithm; and performing SFT training on the basic model based on the joint training method.

[0090] In the disclosed embodiment, the joint training method refers to a type of joint training method of DPO and PPO.

[0091] In the disclosed embodiment, the joint training methods of DPO and PPO are divided into at least two types, which are respectively recorded as the first joint training method and the second joint training method. The difference between the first joint training method and the second joint training method is at least that: the source of the training data of DPO is different. In the first joint training method, the DPO data is annotated offline and can be used immediately. In the second joint training method, the DPO data is sampled by PPO in real time, and the training time is longer.

[0092] In some implementations, determining the joint training method includes: acquiring training instruction information, where the training instruction information includes a type of the joint training method.

[0093] In some embodiments, SFT training is performed on the basic model based on the joint training method, including: preparing training data based on the type of selected joint training method; and performing SFT training on the basic model based on the DPO algorithm and the PPO algorithm based on the prepared training data.

[0094] In some embodiments, the DPO algorithm is used to fine-tune the model to maximize the likelihood of generating preferred completions and minimize the likelihood of generating undesirable completions. The DPO stage focuses on optimizing the preference consistency of the model. The model is further fine-tuned using the PPO algorithm to ensure the stability of training by limiting the amplitude of policy updates. The PPO stage focuses on improving the stability and performance of the model. When resources are sufficient, consider running the DPO and PPO algorithms in parallel, with each algorithm independently trained on its own data set and regularly exchanging model parameters or gradient information for more efficient training. Under the joint training framework based on DPO and PPO, SFT training can further refine the performance of the model on specific tasks.

[0095] For example, suppose we have a Transformer-based language model as a base model and want to apply it to the dialogue generation task. First, collect a large amount of dialogue data and annotate the preference pairs (i.e., preferred answers and unpopular answers) of each dialogue. Then, use the Transformer model as the base model. Then, use the DPO algorithm to fine-tune the model to optimize the performance of the model on the preference dataset; use the PPO algorithm to further fine-tune the model to ensure the stability of the training; and use the labeled dialogue data to perform SFT training on the model. During the training process, adjust the loss function and training strategy by combining the advantages of DPO and PPO. Finally, use the test set to evaluate the performance of the trained model on the dialogue generation task.

[0096] In this way, through the joint training method based on DPO and PPO, the advantages of DPO and PPO algorithms can be fully utilized to improve the model's preference consistency and stability. Through supervised training, the model can learn more about a specific task, thereby improving its generalization ability on the task. The parallel training method can significantly improve training efficiency and shorten model training time.

[0097] In some embodiments, SFT training is performed on the basic model based on a joint training method, including: when the joint training method is the first joint training method, respectively obtaining the first category preference data corresponding to the DPO algorithm, the PPO algorithm and the SFT training; wherein the first category preference data used by the DPO algorithm is extracted from pre-labeled training samples; based on the first category preference data corresponding to the PPO algorithm, the DPO algorithm and the SFT training, the DPO algorithm, the PPO algorithm and the SFT training are simultaneously applied to the basic model to train the basic model.

[0098] In the embodiment of the present disclosure, the first type of preference data is the preference data extracted from the pre-labeled training samples and used for DPO algorithm, PPO algorithm and SFT training. These data contain the preference information of human beings for different generated contents.

[0099] In some embodiments, the preparation of the first category of preference data is divided into two parts: data collection and labeling, and data partitioning. Among them, data collection and labeling include: pre-collecting and labeling a large number of training samples, which contain human preference information for different generated content. Extract the first category of preference data from these labeled training samples, and these data will be used for DPO algorithm, PPO algorithm and SFT training respectively. Among them, data partitioning includes: ensuring that the respective data sets of DPO, PPO and SFT are both independent of each other and have a certain correlation, so that their respective advantages can be fully utilized during joint training. Alternatively, only ensure that the respective data sets of DPO, PPO and SFT are independent of each other, so that their respective advantages can be utilized during joint training.

[0100] Figure 3 Schematic diagram showing the source of training data for DPO and PPO Figure 1 ,like Figure 3 As shown in the figure, the training data of DPO comes from the offline preference data of DPO, and the training data of PPO comes from the online real-time sampling data of PPO, and the two are independent of each other. The first type of preference data corresponding to PPO is screened out from the online real-time sampling data of PPO, and the first type of preference data corresponding to DPO is screened out from the offline preference data of DPO.

[0101] In some embodiments, in this joint training mode, the DPO algorithm, the PPO algorithm, and the SFT training will be performed simultaneously, each using its corresponding first-class preference data for training. Although DPO, PPO, and SFT are independent of each other during the training process, model parameters or gradient information can be exchanged regularly to promote knowledge sharing and complementarity between different algorithms. Use an appropriate parallel computing framework (such as a distributed training function) to ensure the efficiency and resource utilization of the training process.

[0102] In some embodiments, in this joint training mode, an independent optimizer and learning rate are set for each algorithm so as to be fine-tuned according to the respective training characteristics and needs. During the training process, the learning rate and other hyperparameters are dynamically adjusted according to the performance of the model on the validation set to improve the training effect.

[0103] In some embodiments, under this joint training mode, the training processes of DPO, PPO and SFT are started at the same time, and each uses its corresponding data set to fine-tune the basic model. During the training process, the performance indicators of the model (such as preference consistency, generation quality, stability, etc.) are monitored and adjusted as needed. When the performance of the model on the validation set reaches the preset standard or the training round reaches the preset upper limit, the training is stopped. The trained model is evaluated using the test set to verify its performance on a specific task. The performance of the model in terms of preference consistency, generation diversity, language fluency, etc. is analyzed. The trained model is deployed to the actual application scenario, and online testing and user feedback collection are performed. The model is continuously optimized and improved based on user feedback and actual application effects.

[0104] In this way, the basic model is trained with SFT based on the first joint training method. DPO, PPO and SFT each play their own advantages in the training process, and improve the overall performance of the model through information sharing and complementarity. The PPO algorithm's limit strategy update amplitude mechanism helps to ensure the stability of training and avoid performance fluctuations in the model during training. The DPO algorithm's ability to optimize preference data enables the model to better meet human preference needs for generated content. The parallel training method can significantly improve training efficiency and shorten model training time.

[0105] In some embodiments, SFT training is performed on the basic model based on a joint training method, including: when the joint training method is the second joint training method, respectively obtaining the second type of preference data corresponding to the DPO algorithm, the PPO algorithm and the SFT training; wherein the second type of preference data corresponding to the DPO algorithm is obtained based on the sampling data of the PPO algorithm; based on the second type of preference data corresponding to the PPO algorithm, the DPO algorithm and the SFT training, the DPO algorithm, the PPO algorithm and the SFT training are simultaneously applied to the basic model to train the basic model.

[0106] In the disclosed embodiment, the second type of preference data is specific preference data used for DPO, PPO and SFT training. These data may be generated based on the model's sampled data, user feedback or expert evaluation, and are intended to guide the model to learn how to generate user preferences.

[0107] In some embodiments, sampling data based on the PPO algorithm includes: first, using the PPO algorithm to perform preliminary training on the basic model, and collecting a large amount of sampling data through interaction with the environment or simulator. These sampling data contain the behavior of the model in different states and its corresponding reward or punishment information. Based on the sampling data of the PPO algorithm, the second type of preference data required by the DPO algorithm is generated. This can be achieved by analyzing the pros and cons of the model behavior in the sampling data, user feedback (if available), or expert evaluation. At the same time, prepare its independent second type of preference data for SFT training, which may come from labeled high-quality dialogue examples or user feedback. Then, data partitioning is performed to ensure that the respective data sets of DPO, PPO, and SFT are both independent and complementary to each other, so as to achieve knowledge sharing and complementarity in the joint training process.

[0108] In some embodiments, in the second joint training method, the second type of preference data of the DPO algorithm depends on the sampled data of the PPO algorithm. Therefore, it is necessary to complete the preliminary training of PPO first, and then generate the preference data of DPO based on its output. Once the second type of preference data required by DPO and SFT are ready, the parallel training process of the three learning algorithms can be started. Use an appropriate parallel computing framework to ensure that DPO, PPO and SFT can access the basic model and update parameters at the same time. Although the preference data of DPO depends on the output of PPO, during the training process, the three algorithms can still share some intermediate results or model parameters to promote overall performance improvement.

[0109] Figure 4 Schematic diagram showing the source of training data for DPO and PPO Figure 2 ,like Figure 4 As shown in Figure 2, the training data of DPO and PPO are both from PPO online real-time sampling data. The corresponding second-category preference data are screened out from PPO online real-time sampling data through the RM model.

[0110] In some embodiments, a pre-trained basic model is used as a starting point, and the model already has certain initial capabilities. In each round of iteration, the PPO algorithm continues to interact with the environment and collect new sampled data; based on the new sampled data, the preference data of the DPO is updated (if applicable), and the DPO is trained at the same time; the SFT training uses its independent preference data to fine-tune the basic model; the three algorithms act on the basic model at the same time, and the weights of the model are updated through back propagation.

[0111] In some embodiments, the performance of the model on the validation set is continuously monitored during the training process, including indicators such as generation quality, preference consistency, and stability. Dynamically adjust hyperparameters such as learning rate and optimizer parameters based on the performance on the validation set to optimize the training effect. Use the test set to comprehensively evaluate the trained model and verify its performance on the target task. Analyze the specific performance of the model in terms of preference consistency, generation diversity, language fluency, etc. Deploy the model with excellent performance to the actual application scenario, conduct online testing and collect user feedback. Continuously optimize and improve based on user feedback and actual application effects.

[0112] For example, suppose we are developing a legal consultation system, and the basic model is a Transformer-based legal consultation encoder. We hope to improve the anthropomorphism of the legal consultation recommended by the system. Based on the second joint training method, we can perform SFT training in the following steps: Use the PPO algorithm to conduct preliminary training on the legal consultation encoder, and collect sample data by simulating the interaction between users and the legal consultation system. These data include feedback information such as user clicks and dwell time on different legal consultations. Based on the PPO sample data, analyze the user's preference pattern (such as which type of legal consultation they like, which type of legal consultation they don't like, etc.), and generate the second type of preference data required by the DPO algorithm. At the same time, prepare its independent preference data for SFT training, which may come from high-quality legal consultation recommendation examples or user feedback. Start the joint training process of DPO, PPO and SFT. PPO continues to collect new sample data and update the model; DPO adjusts its preference data and trains the model based on the PPO sample data; SFT uses its independent preference data to fine-tune the legal consultation encoder. The three algorithms act on the basic model at the same time to improve the personalization of its recommended legal consultation and user satisfaction. Use the test set to evaluate the performance of the trained model in recommending legal advice, including indicators such as user satisfaction and recommendation accuracy. Deploy the model with excellent performance to the legal advice system for online testing and user feedback collection.

[0113] In this way, by performing SFT training on the basic model based on the second joint training method, the DPO algorithm relies on the sampled data of the PPO algorithm. This dependence enables DPO to capture the user's preference information more accurately. At the same time, the three algorithms complement each other during the training process and jointly improve the overall performance of the model. The parallel training method improves the training efficiency and enables the model to converge to a better solution faster. By continuously updating the preference data, the model can better adapt to changes in user preferences and improve the pertinence and accuracy of the generated content.

[0114] In some embodiments, the second type of preference data corresponding to the DPO algorithm and the PPO algorithm are obtained, including: when the PPO algorithm samples data online in real time, for each input x, two candidate outputs y1 and y2 are generated by the PPO strategy to form sample pairs (x, y1) and (x, y2); the two candidate outputs are scored using the reward model to obtain (x, y1, r1) and (x, y2, r2), respectively, where r1 and r2 represent the scores of (x, y1) and (x, y2), respectively; based on the relative sizes of r1 and r2, the second type of preference data corresponding to the PPO algorithm and the DPO algorithm are formed.

[0115] In order to compare or combine the advantages of the two algorithms, DPO and PPO, it is necessary to obtain their preference data on certain specific tasks, especially the second type of preference data, that is, data that sorts or compares the algorithm outputs based on specific criteria (such as the RM model).

[0116] In some embodiments, obtaining the second type of preference data corresponding to each of the DPO and PPO algorithms includes: first, it is necessary to set a clear reinforcement learning task environment, including state space, action space, reward function, etc. This environment will serve as the basis for algorithm training and evaluation. Then, during the training process of the PPO algorithm, for each input state x, the algorithm will generate two candidate outputs y1 and y2 according to the current strategy. For each sample pair (x, y1) and (x, y2), the scores r1 and r2 are calculated respectively using the RM model. Based on the relative size of r1 and r2, preference data is formed. If r1>r2, then record (x, y1) as preferring (x, y2); and vice versa. These preference data can be stored as a list, a database, or any data structure suitable for subsequent analysis.

[0117] In this way, through objective scoring based on the reward model, the performance of DPO and PPO algorithms in different scenarios can be more accurately compared, improving the accuracy of algorithm comparison. Based on the analysis of preference data, new methods that combine the advantages of DPO and PPO algorithms can be explored, such as hybrid strategies, strategy distillation, etc. By analyzing preference data, the hyperparameters or internal structure of the algorithm can be adjusted to further improve the performance of the algorithm.

[0118] In some embodiments, based on the relative sizes of r1 and r2, second-category preference data corresponding to the PPO algorithm and the DPO algorithm are formed respectively, including: if r1 is greater than r2, (x, y1) is used as the second-category preference data corresponding to the PPO algorithm, and (x, y1, y2) is used as the second-category preference data corresponding to the DPO algorithm; if r1 is less than r2, (x, y2) is used as the second-category preference data corresponding to the PPO algorithm, and (x, y2, y1) is used as the second-category preference data corresponding to the DPO algorithm.

[0119] In some embodiments, generating the second type of preference data corresponding to the PPO algorithm and the DPO algorithm includes: First, data collection. During the learning processes of PPO and DPO, for each input state x, two candidate outputs y1 and y2 are generated using their respective policies (or parameterized versions of the current policy). This can be achieved through a forward propagation network, depending on the implementation details of the algorithms. Then, a pre-trained RM model is designed or selected, which is capable of scoring state-action pairs. For each input state x and the corresponding two candidate outputs y1 and y2, their respective reward values r1 and r2 are calculated. Finally, the magnitudes of r1 and r2 are compared to form preference data. If r1 > r2, then it is considered that at state x, output y1 is superior to y2. At this time, (x, y1) is used as a positive sample (or called a "high-quality" sample) of the preference data of the PPO algorithm in the current state, and (x, y1, y2) is used as a preference data point of the DPO algorithm, where y1 is regarded as the "superior" option and y2 as the "sub-optimal" option. This format allows the DPO algorithm to further learn how to select the optimal one among multiple candidate outputs. If r1 < r2, the opposite operation is performed, that is, (x, y2) is used as the positive sample of the preference data of the PPO algorithm, and (x, y2, y1) is used as the preference data point of the DPO algorithm. The generated preference data is stored in the above format for subsequent use in training, evaluating, or optimizing the algorithms. These data can be stored in files, databases, or in-memory data structures, depending on the system architecture and performance requirements.

[0120] For example, assume that in an autonomous driving game, the vehicle needs to choose to go forward (y1) or turn left (y2) based on the current road conditions (input x). The PPO and DPO algorithms respectively generate these two candidate actions and evaluate their values (r1 and r2) through a reward model. If r1 (the reward for going forward) is greater than r2 (the reward for turning left), then (x, y1) will be used as the preference data of the PPO algorithm under the current road conditions, indicating that choosing to go forward is superior in similar situations. At the same time, (x, y1, y2) will be used as a preference data point of the DPO algorithm for it to further learn the strategy of choosing between going forward and turning left. Conversely, if r2 is greater than r1, then (x, y2) will become the preference data of the PPO, and (x, y2, y1) will be used for the DPO to learn. In this way, the DPO can learn how to identify the optimal one among multiple potential behaviors.

[0121] In this way, by comparing the reward values ​​of two candidate outputs and generating special preference data for each algorithm, it helps the algorithm learn the optimal behavior under different states more accurately. The DPO algorithm can use the candidate output pairs in the preference data generated by the PPO algorithm to learn how to choose the best one among multiple potential options, thereby achieving complementarity and synergy between algorithms. Objective evaluation based on the reward model can reduce noise and randomness in the training process, thereby improving the stability and reliability of the algorithm.

[0122] In some embodiments, SFT training is performed on the basic model based on the DPO algorithm and the PPO algorithm, including: obtaining a corresponding amount of preference data from the respective data pools according to the preset ratio of preference data required for the PPO algorithm, DPO algorithm and SFT training in each step of training; based on the preference data corresponding to the PPO algorithm, DPO algorithm and SFT training, respectively applying the PPO algorithm, DPO algorithm and SFT training to the basic model for joint optimization.

[0123] Here, the training data set in S101 can be stored in three data pools, the first data pool is used to store the preference data required by the DPO algorithm, the second data pool is used to store the preference data required by the PPO algorithm, and the third data pool is used to store the preference data required for SFT training.

[0124] In some embodiments, data preparation and matching include: first, a large amount of training data needs to be collected, and these data should cover preference information for different scenarios and tasks. These data can be user interaction data, expert annotation data, or feedback data in a simulation environment. Then, according to a preset ratio (for example, PPO:DPO:SFT=3:2:1), a corresponding amount of preference data is extracted from the data pool. This ratio can be adjusted according to actual task requirements and algorithm performance.

[0125] In some embodiments, based on the preference data corresponding to the PPO algorithm, the DPO algorithm, and the SFT training, the PPO algorithm, the DPO algorithm, and the SFT training are applied to the basic model for joint optimization, including: first, loading a pre-trained basic model, which has basic task execution capabilities but may not have been optimized for specific preferences. Then, the PPO algorithm is used to process a portion of the preference data, and the expected return is maximized by updating the policy network. PPO ensures a stable learning process by limiting the difference between the new and old policies; another portion of the preference data is processed by the DPO algorithm. DPO handles uncertainty more directly by optimizing the probability distribution, which may show better generalization ability in complex environments; another portion of the preference data is used for SFT training, which usually involves fine-tuning the model parameters to make its output more in line with specific preferences or standards. SFT training can be based on supervised learning methods, such as minimizing the loss function.

[0126] In some embodiments, in each training step, PPO, DPO, and SFT are run simultaneously, but their respective data streams and parameter update mechanisms are maintained. A unified optimizer can be set to coordinate the update step size and learning rate of different algorithms to ensure the stability and efficiency of the overall training. The performance of the model is checked regularly, and the data ratio or algorithm parameters are adjusted as needed.

[0127] For example, suppose we have a control system for an autonomous vehicle that needs to learn how to drive safely in different weather and road conditions. We can use the PPO algorithm to process a large amount of driving data collected from a simulation environment to optimize the driving strategy; use the DPO algorithm to process specific scene data containing uncertain factors (such as pedestrians suddenly crossing the road) to improve the model's responsiveness in complex environments; use SFT training to fine-tune the model so that it can accurately follow traffic rules and user preferences (such as avoiding high-speed driving or preferring smooth driving). Through such a joint optimization strategy, we can train a stable and flexible autonomous driving control system that can output safety prompts to users in a timely manner.

[0128] In this way, by combining the stability of PPO and the uncertainty handling ability of DPO, as well as the precise fine-tuning of SFT, the model can learn and adapt to specific preferences more accurately, improving model performance. The use of the DPO algorithm helps the model to show better generalization ability in complex and changing environments. The joint optimization strategy can process different algorithms in parallel, thereby speeding up the overall training speed. The data ratio and algorithm parameters can be adjusted according to the actual task requirements to adapt to different application scenarios.

[0129] In some embodiments, the proportion of preference data required for SFT training will gradually decrease as the number of training steps increases.

[0130] In some embodiments, during the SFT training process, as the number of training steps increases, the proportion of preference data is gradually reduced, which can be achieved by the following steps:

[0131] Initial stage: In the early stages of training, the model has not yet fully learned human preferences, so preference data accounts for a high proportion of the training data to ensure that the model can quickly capture the basic characteristics of human preferences.

[0132] Gradually reduce: As the number of training steps increases, the model gradually learns the rules of human preferences, and the proportion of preference data in the training data can be gradually reduced. This can be achieved by dynamically adjusting the data loading strategy. For example, in each training step, the amount of preference data loaded is calculated and adjusted according to the ratio of the current training step to the total training step.

[0133] Final stage: Near the end of training, the proportion of preferred data can be reduced to a lower level or even removed completely to allow the model to be generalized on a wider dataset, improving the model's robustness and generalization ability.

[0134] For example, suppose that during the SFT training process, the total number of training steps is 1 million. In the first 100,000 steps of training, the proportion of preference data in the training data is 50%; in the next 200,000 steps, the proportion gradually drops to 30%; in the next 300,000 steps, the proportion further drops to 10%; in the last 400,000 steps, the proportion of preference data remains at about 5% until the end of training.

[0135] Figure 5 A schematic diagram of the training data ratio change curve is shown in FIG. Figure 5 As shown, the number of training steps is 7000. From step 0 to step 2000, the proportion of SFT preference data gradually decreases from 1 to 0.98, and the proportion of the sum of DPO and PPO preference data gradually increases from 0 to 0.02; from step 2001 to step 3000, the proportion of SFT preference data gradually decreases from 0.98 to 0.96, and the proportion of the sum of DPO and PPO preference data gradually increases from 0.02 to 0.04; from step 3001 to step 4000, the proportion of SFT preference data is It gradually decreases from 0.96 to 0.92, and the sum of the preference data of DPO and PPO gradually increases from 0.02 to 0.08; from step 4001 to step 6000, the preference data of SFT gradually decreases from 0.92 to 0.9, and the sum of the preference data of DPO and PPO gradually increases from 0.1 to 0.9; from step 6001 to step 7000, the preference data of SFT remains at 0.1, and the sum of the preference data of DPO and PPO remains at 0.9. It is understandable that Figure 5The ratio change diagram shown is only for illustration and not for limitation. The ratio change among SFT, DPO and PPO can be set or adjusted according to the application scenario and training requirements of the model.

[0136] In this way, by gradually reducing the proportion of preference data, the reliance on preference data during training can be reduced while ensuring model performance, thereby improving training efficiency. Reducing the proportion of preference data in the later stages of training helps the model learn on a wider range of data sets, thereby improving the generalization ability of the model and enabling it to better adapt to different application scenarios. Over-reliance on preference data may cause the model to perform well on a specific data set but poorly on other data sets. By gradually reducing the proportion of preference data, the model can be prevented from overfitting to the preference data.

[0137] In some embodiments, in each step of training, keeping the preference data ratio of the PPO algorithm and the DPO algorithm the same can be achieved by:

[0138] Data allocation: In each step of training, the preferred data is allocated to both the PPO and DPO algorithms according to a preset ratio (e.g., equal ratio).

[0139] Parallel training: The PPO and DPO algorithms run in parallel in their respective environments, each using its assigned preference data for training.

[0140] Policy update: The PPO and DPO algorithms update their strategies based on their respective training results and synchronize or coordinate when necessary to ensure that the two algorithms remain consistent during the training process.

[0141] For example, in each step of training, assume that the total amount of preference data is 1,000 records. According to the preset equal proportion principle, the PPO and DPO algorithms are each assigned 500 records for training. The two algorithms run in parallel in their respective environments and update their strategies based on the assigned preference data. In this way, it can be ensured that the PPO and DPO algorithms can obtain the same amount of preference data support in each step of training, thereby maintaining the consistency and stability of the training process.

[0142] Continue with Figure 5 For example, in each step of training, the proportion of preference data of DPO and PPO is the same. For example, from step 6001 to step 7000, the proportion of SFT preference data is always 0.1, and the sum of the preference data of DPO and PPO is always 0.9, that is, the proportion of preference data of DPO and PPO is 0.45 respectively. It can be understood that the ratio change between DPO and PPO can be set or adjusted according to the application scenario and training requirements of the model.

[0143] In this way, by keeping the same percentage of preference data for the PPO and DPO algorithms, the training process can be simplified and the performance differences caused by uneven data distribution can be reduced. Training the PPO and DPO algorithms in parallel and using the same preference data for training can help maintain the stability of the training process and reduce performance fluctuations caused by data differences. The PPO and DPO algorithms each have their own advantages. Through parallel training and the same percentage of preference data, the advantages of both can be combined to further improve the performance of the model.

[0144] The present disclosure provides an information processing method. Figure 6 : is a flowchart of an information processing method according to an embodiment of the present disclosure. The information processing method can be applied to electronic devices, including but not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and servers can be cloud servers or ordinary servers. For example, mobile devices include but are not limited to: mobile phones, tablet computers, vehicle-mounted devices, personal computers, etc. In some possible implementations, the information processing method can also be implemented by a processor calling computer-readable instructions stored in a memory. Figure 6 As shown, the information processing method includes:

[0145] S601: Obtain input information;

[0146] S602: Input the input information to the alignment model, and obtain a response output by the alignment model that matches the input information. The response is generated based on the alignment model's understanding of the input information and the preference features learned by the alignment model during the training process.

[0147] The alignment model is obtained by training through any of the methods described above.

[0148] In the disclosed embodiment, the alignment model is a trained machine learning model that can generate responses that match input information while taking into account the preference features learned by the model during training.

[0149] In the disclosed embodiment, preference features are features such as personal preferences, behavioral habits or preference settings shown by users during use, which are learned by the alignment model during the training process and used to generate personalized responses. The user here can be an independent individual or a labeler representing a certain group.

[0150] In the disclosed embodiment, the input information is the original data or information provided by the user or other system to the alignment model, which is used to trigger the response generation process of the model.

[0151] In the disclosed embodiment, the reply is output information or result that matches the input information and is generated by the alignment model based on the input information and the learned preference features.

[0152] In some embodiments, obtaining input information includes: the system first needs to receive input information from a user, other system or data source. The input information can be in the form of text, voice, image, etc., depending on the design and application scenario of the alignment model. For text input, the system may need to perform preprocessing, such as word segmentation, stop word removal, part-of-speech tagging, etc., to facilitate subsequent processing. For voice input, the system needs to perform voice recognition to convert the voice into text. Image input may need to extract key information through image recognition technology or convert it into a format that can be processed by the model.

[0153] In some embodiments, the input information is input into an alignment model to obtain a response, including:

[0154] The preprocessed input information is fed into the alignment model. Based on the understanding of the input information, the alignment model generates a corresponding response based on the preference features learned during the training process. For example, these preference features are generated based on the preference data input by the annotator. For another example, these preference features may include preference settings, contextual information, and general knowledge or patterns learned by the model from a large amount of data during the training process.

[0155] In some embodiments, the process of the model processing input information generally includes three main stages: encoding (converting the input information into a representation understandable to the model), reasoning (performing internal calculations based on the input information and the learned preference features), and decoding (converting the internal calculation results into understandable output responses). Ultimately, the model outputs a response that matches the input information, which is intended to meet the user's query requirements, meet the user's preference settings, and reflect the knowledge and rules learned by the model as accurately as possible.

[0156] For example, suppose there is an intelligent customer service system based on an alignment model. The user asks the system through text input: "I want to make a reservation for a restaurant at 7 o'clock tomorrow night." The system first obtains the user's input information and feeds it into the alignment model. The alignment model generates a personalized reply based on the user's input information and the user's historical reservation behavior, preferred restaurant type, price range and other preference features learned during the training process, such as: "We have reserved the 'XXX' restaurant in the city center for you at 7 o'clock tomorrow night. The average consumption per person is about 200 yuan, which meets your preferences." Such a reply not only meets the user's query needs, but also fully considers the user's personalized preferences.

[0157] In this way, by aligning the preference features learned by the model, the system can generate responses that are more in line with the preference features, improving the intelligence and accuracy of the responses. The alignment model can not only process known types of input information, but also make reasonable inferences and responses to unknown or novel types of input information.

[0158] It should be understood that Figure 2 , Figure 3 , Figure 4 and Figure 5 The schematic diagram shown is only exemplary and not restrictive, and it is scalable, and those skilled in the art can Figure 2 , Figure 3 , Figure 4 and Figure 5 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0159] The present disclosure provides a training device for an alignment model, such as Figure 7 As shown, the training device of the alignment model may include: a first acquisition module 701, used to obtain a training sample set, which includes preference data; a training module 702, used to perform SFT training on the basic model based on the DPO algorithm and the PPO algorithm to obtain an alignment model, the output of the alignment model is consistent with the preference feature, and the preference feature is obtained based on the preference data in the training sample set.

[0160] In some embodiments, the training module 702 includes: a first acquisition submodule, used to obtain the first loss function of the DPO algorithm for the basic model; to obtain the second loss function of the PPO algorithm for the basic model; to obtain the third loss function of the SFT training for the basic model; a first determination submodule, used to determine the total loss function based on the first loss function, the second loss function and the third loss function; and a first training submodule, used to update the parameters of the basic model based on the total loss function.

[0161] In some embodiments, the training module 702 includes: a second determination submodule, used to determine the joint training method of the DPO algorithm and the PPO algorithm; and a second training submodule, used to perform SFT training on the basic model based on the joint training method.

[0162] In some embodiments, the second training submodule is used to: when the joint training mode is the first joint training mode, respectively obtain the first type of preference data corresponding to the DPO algorithm, the PPO algorithm and the SFT training; wherein the first type of preference data used by the DPO algorithm is extracted from pre-labeled training samples; based on the first type of preference data corresponding to the PPO algorithm, the DPO algorithm and the SFT training, the DPO algorithm, the PPO algorithm and the SFT training are simultaneously applied to the basic model to train the basic model.

[0163] In some embodiments, the second training submodule is used to: when the joint training mode is the second joint training mode, respectively obtain the second type of preference data corresponding to the DPO algorithm, the PPO algorithm and the SFT training; wherein the second type of preference data corresponding to the DPO algorithm is obtained based on the sampling data of the PPO algorithm; based on the second type of preference data corresponding to the PPO algorithm, the DPO algorithm and the SFT training, simultaneously apply the DPO algorithm, the PPO algorithm and the SFT training to the basic model to train the basic model.

[0164] In some embodiments, the device further comprises: a sampling module ( Figure 7 (not shown), used to generate two candidate outputs y1 and y2 for each input x through the PPO strategy when the PPO algorithm samples data online in real time, forming sample pairs (x, y1) and (x, y2); scoring module ( Figure 7 (not shown), used to score the two candidate outputs using the reward model, and obtain (x, y1, r1) and (x, y2, r2), respectively, where r1 and r2 represent the scores of (x, y1) and (x, y2), respectively; the generation module ( Figure 7 (not shown) is used to form the second type of preference data corresponding to the PPO algorithm and the DPO algorithm respectively based on the relative sizes of r1 and r2.

[0165] In some embodiments, the generating module ( Figure 7 Not shown), including: a first generating submodule, for using (x, y1) as the second category preference data corresponding to the PPO algorithm, and (x, y1, y2) as the second category preference data corresponding to the DPO algorithm, if r1 is greater than r2; a second generating submodule, for using (x, y2) as the second category preference data corresponding to the PPO algorithm, and (x, y2, y1) as the second category preference data corresponding to the DPO algorithm, if r1 is less than r2.

[0166] In some embodiments, the training module 702 includes: a third acquisition sub-module, used to obtain a corresponding amount of preference data from the respective data pools according to the preset ratio of preference data required for the PPO algorithm, DPO algorithm and SFT training in each step of training; a third training sub-module, used to apply the PPO algorithm, DPO algorithm and SFT training to the basic model based on the preference data corresponding to the PPO algorithm, DPO algorithm and SFT training, respectively, for joint optimization.

[0167] In some embodiments, the proportion of preference data required for SFT training will gradually decrease as the number of training steps increases.

[0168] In some embodiments, in each step of training, the PPO algorithm and the DPO algorithm have the same proportion of preference data.

[0169] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0170] The training device of the alignment model of the embodiment of the present disclosure can effectively improve the performance of the alignment model by combining the DPO algorithm, the PPO algorithm and the SFT training, which not only improves the generalization ability of the model, but also enhances the accuracy of the model output.

[0171] The present disclosure provides an information processing device, such as Figure 8 As shown, the information processing device may include: a second acquisition module 801, used to obtain input information; a processing module 802, used to input the input information into an alignment model, and obtain a response output by the alignment model that matches the input information, wherein the response is generated based on the alignment model's understanding of the input information and the preference features learned by the alignment model during training; wherein the alignment model is trained based on the training method described above.

[0172] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0173] The input control device of the alignment model of the embodiment of the present disclosure can generate a response that is more in line with human preference characteristics based on the preference characteristics learned by the alignment model, thereby improving the intelligence of the response.

[0174] The present disclosure provides a scenario diagram of a training method for an alignment model, such as Fig. 9 shown.

[0175] As mentioned above, the training method of the alignment model provided by the embodiment of the present disclosure is applied to an electronic device.

[0176] Specifically, the electronic device can perform the following operations:

[0177] Acquire a training sample set, wherein the training sample set includes preference data;

[0178] Based on the DPO algorithm and the PPO algorithm, the basic model is trained by SFT to obtain an alignment model, the output of which is consistent with the preference feature, which is obtained based on the preference data in the training sample set.

[0179] It should be understood that Fig. 9 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Fig. 9 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0180] The present disclosure provides a scenario diagram of an information processing method, such as Fig.10 shown.

[0181] As mentioned above, the information processing method provided by the embodiments of the present disclosure is applied to an electronic device.

[0182] Specifically, the electronic device can perform the following operations:

[0183] Get input information;

[0184] Input information is input into the alignment model, and a response matching the input information is obtained from the alignment model. The response is generated based on the alignment model's understanding of the input information and the preference features learned by the alignment model during training.

[0185] It should be understood that Fig.10 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Fig.10 Various obvious changes and / or substitutions can be made to the examples, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0186] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0187] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0188] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0189] like Fig.11As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0190] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0191] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the training method and / or information processing method of the alignment model. For example, in some embodiments, the training method and / or information processing method of the alignment model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via ROM 1102 and / or the communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the training method and / or information processing method of the alignment model described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the training method and / or information processing method of the alignment model in any other appropriate manner (eg, by means of firmware).

[0192] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0193] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0194] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0195] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0196] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0197] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0198] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0199] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for an alignment model, comprising: Acquire a training sample set, wherein the training sample set includes preference data; Based on the direct preference learning DPO algorithm and the online reinforcement learning PPO algorithm, the basic model is supervised fine-tuned SFT training to obtain an alignment model, the output of the alignment model is consistent with the preference feature, and the preference feature is obtained based on the preference data in the training sample set.

2. The method according to claim 1, wherein: Based on the direct preference learning DPO algorithm and the online reinforcement learning PPO algorithm, supervised fine-tuning SFT training is performed on the basic model, including: Obtaining a first loss function of the DPO algorithm for the basic model; Obtaining a second loss function of the PPO algorithm for the basic model; Obtaining a third loss function of the SFT training on the basic model; Determine a total loss function based on the first loss function, the second loss function and the third loss function; The parameters of the basic model are updated based on the total loss function.

3. The method according to claim 1, wherein: Based on the direct preference learning DPO algorithm and the online reinforcement learning PPO algorithm, supervised fine-tuning SFT training is performed on the basic model, including: Determining a joint training method of the DPO algorithm and the PPO algorithm; Based on the joint training method, SFT training is performed on the basic model.

4. The method according to claim 3, wherein: The performing SFT training on the basic model based on the joint training method includes: When the joint training mode is the first joint training mode, first-category preference data corresponding to the DPO algorithm, the PPO algorithm and the SFT training are respectively obtained; wherein the first-category preference data used by the DPO algorithm is extracted from pre-labeled training samples; Based on the first type of preference data respectively corresponding to the PPO algorithm, the DPO algorithm and the SFT training, the DPO algorithm, the PPO algorithm and the SFT training are simultaneously applied to the basic model to train the basic model.

5. The method according to claim 3, wherein: The performing SFT training on the basic model based on the joint training method includes: When the joint training mode is the second joint training mode, respectively obtaining the second type of preference data corresponding to the DPO algorithm, the PPO algorithm and the SFT training; wherein the second type of preference data corresponding to the DPO algorithm is obtained based on the sampling data of the PPO algorithm; Based on the second type of preference data corresponding to the PPO algorithm, the DPO algorithm and the SFT training respectively, the DPO algorithm, the PPO algorithm and the SFT training are simultaneously applied to the basic model to train the basic model.

6. The method according to claim 5, further comprising: When the PPO algorithm samples data online in real time, for each input x, two candidate outputs y1 and y2 are generated through the PPO strategy to form sample pairs (x, y1) and (x, y2); The two candidate outputs are scored using the reward model to obtain (x, y1, r1) and (x, y2, r2), respectively, where r1 and r2 represent the scores of (x, y1) and (x, y2), respectively; Based on the relative sizes of r1 and r2, the second type of preference data corresponding to the PPO algorithm and the DPO algorithm are formed respectively.

7. The method according to claim 6, wherein: The forming of the second type of preference data corresponding to the PPO algorithm and the DPO algorithm respectively based on the relative size of r1 and r2 includes: If r1 is greater than r2, (x, y1) is used as the second type of preference data corresponding to the PPO algorithm, and (x, y1, y2) is used as the second type of preference data corresponding to the DPO algorithm; If r1 is less than r2, (x, y2) is used as the second type of preference data corresponding to the PPO algorithm, and (x, y2, y1) is used as the second type of preference data corresponding to the DPO algorithm.

8. The method according to claim 1, wherein: Based on the direct preference learning DPO algorithm and the online reinforcement learning PPO algorithm, supervised fine-tuning SFT training is performed on the basic model, including: According to the preset ratio of the preference data required for the PPO algorithm, the DPO algorithm and the SFT training in each step of training, a corresponding amount of preference data is obtained from the respective data pools; Based on the preference data corresponding to the PPO algorithm, the DPO algorithm and the SFT training respectively, the PPO algorithm, the DPO algorithm and the SFT training are applied to the basic model respectively for joint optimization.

9. The method according to claim 8, wherein: The proportion of preference data required for the SFT training will gradually decrease as the number of training steps increases.

10. The method according to claim 8 or 9, wherein: In each step of training, the PPO algorithm and the DPO algorithm have the same proportion of preference data.

11. An information processing method, comprising: Get input information; The input information is input into an alignment model, and a response output by the alignment model that matches the input information is obtained, wherein the response is generated based on the alignment model's understanding of the input information and the preference features learned by the alignment model during training; wherein the alignment model is trained by the method described in any one of claims 1 to 10.

12. A training device for an alignment model, comprising: A first acquisition module, used to acquire a training sample set, wherein the training sample set includes preference data; A training module is used to perform supervised fine-tuning SFT training on the basic model based on the direct preference learning DPO algorithm and the online reinforcement learning PPO algorithm to obtain an alignment model, wherein the output of the alignment model is consistent with the preference feature, and the preference feature is obtained based on the preference data in the training sample set.

13. The device according to claim 12, wherein: The training module comprises: The first acquisition submodule is used to acquire the first loss function of the DPO algorithm for the basic model; acquire the second loss function of the PPO algorithm for the basic model; and acquire the third loss function of the SFT training for the basic model; A first determination submodule, configured to determine a total loss function based on the first loss function, the second loss function, and the third loss function; The first training submodule is used to update the parameters of the basic model based on the total loss function.

14. The device according to claim 12, wherein: The training module comprises: A second determination submodule is used to determine a joint training method of the DPO algorithm and the PPO algorithm; The second training submodule is used to perform SFT training on the basic model based on the joint training method.

15. The device according to claim 14, wherein: The second training submodule is used to: When the joint training mode is the first joint training mode, first-category preference data corresponding to the DPO algorithm, the PPO algorithm and the SFT training are respectively obtained; wherein the first-category preference data used by the DPO algorithm is extracted from pre-labeled training samples; Based on the first type of preference data respectively corresponding to the PPO algorithm, the DPO algorithm and the SFT training, the DPO algorithm, the PPO algorithm and the SFT training are simultaneously applied to the basic model to train the basic model.

16. The device according to claim 14, wherein: The second training submodule is used to: When the joint training mode is the second joint training mode, respectively obtaining the second type of preference data corresponding to the DPO algorithm, the PPO algorithm and the SFT training; wherein the second type of preference data corresponding to the DPO algorithm is obtained based on the sampling data of the PPO algorithm; Based on the second type of preference data corresponding to the PPO algorithm, the DPO algorithm and the SFT training respectively, the DPO algorithm, the PPO algorithm and the SFT training are simultaneously applied to the basic model to train the basic model.

17. The apparatus according to claim 16, further comprising: A sampling module, used for generating two candidate outputs y1 and y2 for each input x through the PPO strategy when the PPO algorithm samples data online in real time, forming sample pairs (x, y1) and (x, y2); A scoring module, used to score the two candidate outputs using a reward model to obtain (x, y1, r1) and (x, y2, r2), respectively, where r1 and r2 represent the scores of (x, y1) and (x, y2), respectively; A generating module is used to form the second type of preference data corresponding to the PPO algorithm and the DPO algorithm respectively based on the relative size of r1 and r2.

18. The device according to claim 17, wherein: The generating module comprises: A first generating submodule, for, if r1 is greater than r2, using (x, y1) as the second type of preference data corresponding to the PPO algorithm, and using (x, y1, y2) as the second type of preference data corresponding to the DPO algorithm; The second generating submodule is used to use (x, y2) as the second type of preference data corresponding to the PPO algorithm and (x, y2, y1) as the second type of preference data corresponding to the DPO algorithm if r1 is less than r2.

19. The device according to claim 12, wherein: The training module comprises: A third acquisition submodule is used to acquire a corresponding amount of preference data from respective data pools according to a preset ratio of preference data required for the PPO algorithm, the DPO algorithm and the SFT training in each step of training; The third training submodule is used to apply the PPO algorithm, the DPO algorithm and the SFT training to the basic model respectively based on the preference data corresponding to the PPO algorithm, the DPO algorithm and the SFT training, and perform joint optimization.

20. The device according to claim 19, wherein The proportion of preference data required for the SFT training will gradually decrease as the number of training steps increases.

21. The device according to claim 19 or 20, wherein: In each step of training, the PPO algorithm and the DPO algorithm have the same proportion of preference data.

22. An information processing device, comprising: A second acquisition module is used to acquire input information; A processing module, used for inputting the input information into an alignment model, obtaining a response output by the alignment model that matches the input information, wherein the response is generated based on the alignment model's understanding of the input information and the preference features learned by the alignment model during training; wherein the alignment model is obtained by training using the method described in any one of claims 1 to 10.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are for causing a computer to perform a method according to any one of claims 1-11.

25. A computer program product comprising a computer program stored on a storage medium, the computer program implementing the method according to any one of claims 1 to 11 when executed by a processor.

Images