Preference alignment training method, system and equipment of large language model and storage medium
By training and uploading the selector model locally on the client, combined with differential privacy protection, a global preference dataset is constructed, which solves the privacy leakage and data shortage problems caused by centralized data collection, and improves the preference alignment accuracy and training efficiency of large language models.
Patent Information
- Application Number
- CN202510891961.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-27
- Publication Date
- 2025-10-03
AI Technical Summary
Existing preference alignment training methods for large language models rely on centralized data collection, resulting in a high risk of user privacy data leakage and insufficient comprehensiveness and diversity of datasets, which affects the accuracy of the model's preference alignment.
The pre-trained selector model is broadcasted to the client through the server. The client trains a local selector model based on the local preference dataset and sends it to the server. The server aggregates it and builds a global preference dataset to train the target large language model. The client's local data is retained locally, and only the trained selector model is uploaded. Differential privacy noise is combined to protect user privacy.
It improves the preference alignment accuracy of large language models, enhances the comprehensiveness and diversity of global preference datasets, avoids user data leakage, and improves the efficiency and scalability of model training.
Smart Images
Figure CN120745752A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of natural language processing technology, and in particular to a preference alignment training method, system, device, and storage medium for a large language model. Background Art
[0002] With the advancement of natural language processing technology, large language models are widely used in financial systems (such as intelligent investment advisory systems and intelligent customer service systems) and healthcare systems (such as medical consultation assistance systems and intelligent guidance systems). Therefore, improving the preference alignment accuracy of large language models is crucial for accurately predicting the preferences of financial consumers or medical patients, thereby improving the service experience. Traditional large language model training methods, such as RLHF (Reinforcement Learning from Human Feedback), rely on centralized collection of user preference datasets to train the large language model based on the preference alignment of these datasets. However, because these datasets contain private user data, users are often reluctant to share and may even fabricate data. Furthermore, centralized data collection methods cannot fully utilize the diverse preference data distributed across a large number of users. This results in a less comprehensive and diverse dataset, and consequently, the preference alignment accuracy of large language models trained on these datasets is low. Therefore, improving the preference alignment accuracy of large language models while protecting user privacy has become an urgent issue. Summary of the Invention
[0003] The present application provides a preference alignment training method, system, device and storage medium for a large language model to improve the accuracy of preference alignment of a large language model while protecting user privacy.
[0004] In a first aspect, the present application provides a preference alignment training method for a large language model, which is applied to a preference alignment training system for a large language model. The preference alignment training system for a large language model includes a server and at least one client. The method includes:
[0005] The server broadcasts a pre-trained selector model to at least one of the clients;
[0006] When receiving the pre-trained selector model, each client trains the pre-trained selector model according to the local preference data set to obtain a local selector model, and sends the local selector model to the server;
[0007] The server aggregates the received local selector models to obtain a global selector model, constructs a global preference dataset based on the global selector model, and performs preference alignment training on the pre-trained large language model based on the global preference dataset to obtain a target large language model.
[0008] In a second aspect, the present application further provides a preference alignment training system for a large language model, the system comprising:
[0009] The server is configured to broadcast a pre-trained selector model to at least one of the clients; aggregate the received local selector models to obtain a global selector model, construct a global preference dataset based on the global selector model, and perform preference alignment training on the pre-trained large language model based on the global preference dataset to obtain a target large language model;
[0010] The client is used for each of the clients to train the pre-trained selector model according to the local preference data set when receiving the pre-trained selector model, obtain a local selector model, and send the local selector model to the server.
[0011] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the preference alignment training method for a large language model as described above when executing the computer program.
[0012] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the preference alignment training method for the large language model as described above.
[0013] The present application discloses a preference alignment training method, system, computer device and storage medium for a large language model, wherein the server broadcasts a pre-trained selector model to at least one of the clients; upon receiving the pre-trained selector model, each client trains the pre-trained selector model according to a local preference data set to obtain a local selector model, and sends the local selector model to the server; the server aggregates the received local selector models to obtain a global selector model, and constructs a global preference data set according to the global selector model, and performs preference alignment training on the pre-trained large language model based on the global preference data set to obtain a target large language model. The present application broadcasts a pre-trained selector model to the client through the server, and the client trains the pre-trained selector model according to the local preference data set to obtain a local selector model, and sends the local selector model back to the server, and the server aggregates the collected local selector models to obtain a global selector, and then constructs a global preference data set according to the global selector to perform preference alignment training on the large language model. The client's local preference dataset is retained locally, and the server only collects the local selector model trained by the client, avoiding the leakage of user data. Secondly, the user preference data of multiple clients are integrated to improve the comprehensiveness and diversity of the global preference dataset, and the accuracy of the preference alignment of the large language model obtained by training with this dataset is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0015] Figure 1 This is a schematic diagram of an application environment of a preference alignment training method for a large language model provided by an embodiment of the present application;
[0016] Figure 2 This is a schematic diagram of the effect of the user location relationship diagram provided by the first embodiment of the present application;
[0017] Figure 3 This is a schematic diagram of the effect of the user location relationship diagram provided by the second embodiment of the present application;
[0018] Figure 4 A schematic block diagram of a preference alignment training system for a large language model provided in an embodiment of the present application;
[0019] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0020] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0021] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.
[0022] It should be understood that the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.
[0023] It should be further understood that the term “and / or” used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0024] The embodiment of the present application provides a preference alignment training method, system, device and storage medium for a large language model. The preference alignment training method for a large language model can be applied to Figure 1In the application environment shown, the client communicates with the server through the network. The server broadcasts a pre-trained selector model to at least one client; when each client receives the pre-trained selector model, it trains the pre-trained selector model according to the local preference data set to obtain a local selector model, and sends the local selector model to the server; the server aggregates the received local selector models to obtain a global selector model, and constructs a global preference data set based on the global selector model, and performs preference alignment training on the pre-trained large language model based on the global preference data set to obtain a target large language model. The pre-trained selector model is broadcast to the client through the server, and the client trains the pre-trained selector model according to the local preference data set to obtain a local selector model, and sends the local selector model back to the server. The server aggregates the collected local selector models to obtain a global selector, and then constructs a global preference data set based on the global selector to perform preference alignment training on the large language model. The client's local preference dataset is retained locally, and the server only collects the local selector model trained by the client, avoiding the leakage of user data. Secondly, the user preference data of multiple clients are integrated to improve the comprehensiveness and diversity of the global preference dataset, and the accuracy of the preference alignment of the large language model obtained by training with this dataset is improved.
[0025] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.
[0026] See also Figure 2 , Figure 2 This is a schematic flow chart of a preference alignment training method for a large language model provided in an embodiment of the present application. The preference alignment training method for a large language model can be applied to a preference alignment training system for a large language model, and is used to broadcast a pre-trained selector model to a client through a server. The client trains the pre-trained selector model according to a local preference data set to obtain a local selector model, and sends the local selector model back to the server. The server aggregates the collected local selector models to obtain a global selector, and then constructs a global preference data set based on the global selector to perform preference alignment training on the large language model. The local preference data set of the client is retained locally, and the server only collects the local selector model trained by the client, thereby avoiding the leakage of user data. Secondly, the user preference data of multiple clients are integrated, which improves the comprehensiveness and diversity of the global preference data set, thereby improving the accuracy of the preference alignment of the large language model obtained by training with this data set.
[0027] like Figure 2 As shown, the preference alignment training method of the large language model specifically includes steps S101 to S103.
[0028] S101, the server broadcasts a pre-trained selector model to at least one of the clients;
[0029] In one embodiment, the server uniformly samples a group of clients from multiple clients and broadcasts the current initial binary selector model (i.e., the pre-trained selector model) to the selected clients. Using a binary selector instead of the traditional regression-based reward model significantly reduces computational overhead, enabling model training on resource-constrained edge devices and improving the efficiency and scalability of the entire system.
[0030] The broadcast method can be to send the parameters of the pre-trained selector model to the selected client through a secure communication protocol to ensure that the model parameters are not tampered with or leaked during the transmission process.
[0031] S102. Upon receiving the pre-trained selector model, each client trains the pre-trained selector model according to the local preference data set to obtain a local selector model, and sends the local selector model to the server.
[0032] In one embodiment, the client locally stores a preference data set, which includes multiple sets of "query-response pair-preference corresponding content" triplets.
[0033] The query refers to the original instruction entered by the user, such as "recommend a property and casualty insurance product." The response content is the multiple pieces of content output by the local large language model based on the original instruction. A response pair contains two pieces of response content. The preferred response content is the response content selected by the user.
[0034] In one embodiment, after receiving the pre-trained selector model broadcast by the server, the client performs a preset number of iterative training on the pre-trained selector model according to the local preference data set to obtain a local selector model. The preset number of times can be set by the user according to actual needs. Specifically, the received pre-trained selector model is used as the initial model, and the preset number of iterative updates are performed locally. Each iteration updates the model parameters by calculating the gradient of the loss function. The loss function adopts the cross entropy loss function, and the mathematical expression is, F m (φ)=E(x,y0,y1,i)~D m [L CE (i|φ; x, y0, y1)], where F m (φ) represents the expected loss on client m, L CE (i|φ; x, y0, y1) is the cross entropy loss function, which is used to measure the accuracy of the large language model φ in predicting the preference choice i for input x and responses y0 and y1. m is the local preference dataset of client m.
[0035] In one embodiment, after the client completes local training, it sends the updated local selector model back to the server.
[0036] Furthermore, before step S101, it also includes: the client collects local query instructions and at least one group of local response pairs corresponding to each local query instruction; the client determines the preferred response content in each group of the local response pairs; and generates the local preference data set based on the local query instructions, each local response pair and the preferred response content.
[0037] In one embodiment, a local query instruction may be input content generated by a user interacting with a local large language model. For each local query instruction, the local large language model generates at least one corresponding set of local response pairs. Preferred response content is determined within each set of local response pairs. Specifically, user selections may be monitored and the user's preferred response content determined based on the user's selections as the preferred response content.
[0038] The local query instructions, local response pairs and preference response content are integrated to generate a local preference dataset.
[0039] Furthermore, step S102 includes: the client trains the pre-trained selector model according to the local preference data set to obtain the local selector model; adds differential privacy noise to the parameters of the local selector model to obtain an encrypted local selector model; and uploads the encrypted local selector model to the server.
[0040] In one embodiment, in order to avoid inferring user preference data through model parameters, differential privacy noise is added to ensure that the presence or absence of any single data does not significantly affect the final uploaded model parameters, thereby protecting user privacy.
[0041] Specifically, the client sets the privacy protection strength and calculates parameter sensitivity, which is the maximum change in model parameters when a single piece of data is deleted or added. For the fully connected layer weights of the selector model, their sensitivity can be calculated using the gradient norm during training. Based on the privacy protection strength and sensitivity, noise is added to each parameter, and the noise is added to the parameters to obtain the encrypted local selector model.
[0042] The client sends the encrypted local selector model to the server.
[0043] S103. The server aggregates the received local selector models to obtain a global selector model, constructs a global preference dataset based on the global selector model, and performs preference alignment training on the pre-trained large language model based on the global preference dataset to obtain a target large language model.
[0044] In one embodiment, the server performs weighted average aggregation on the collected local selector models and updates the global selector model. The weighted formula is: Among them, φ r+1 represents the updated global selector model; M is the number of selected clients; A is the selected client set; φ m，r,k is the selector model of client m after K local iterations training in the rth round of communication.
[0045] In one embodiment, the server uses a pre-trained large language model to generate multiple responses to a stored set of commands. For each command, the server generates multiple responses to form all possible response pairs. A trained binary selector is then used to evaluate each pair, selecting the optimal response and constructing a synthetic preference dataset based on this.
[0046] In one embodiment, the server uses a global preference dataset to construct and fine-tune the pre-trained large language model using the DPO (Direct Preference Optimization) method. Specifically, in each training round, a batch of data is sampled from the global preference dataset, and the loss function of the pre-trained large language model of the current round on this batch of data is calculated. The parameters of the pre-trained large prediction model are updated accordingly. The loss function formula is:
[0047]
[0048] Wherein, θ represents the parameters of the target large language model, x is the preset instruction, (y0, y1) represents a response pair, i represents the index of the target response content, i=0 represents the preference for y0, i=1 represents the preference for y1; πθ(y i |x) represents the target large language model’s response y under the parameter θ i The generation probability under instruction x, πθ0(y i |x) represents the response y of the pre-trained large language model i The generation probability under instruction x, β is the balance parameter, and σ is the sigmoid function (a nonlinear activation function).
[0049] When the preset number of iterations is reached or the loss function value is less than the preset loss threshold, the iteration is stopped and the target large language model is obtained.
[0050] Furthermore, the global preference dataset is constructed based on the global selector model, including: the server inputs a preset instruction to the pre-trained large language model, obtains at least one global response content output by the pre-trained large language model based on the preset instruction, and generates at least one group of global response pairs based on at least one global response content, wherein a group of global response pairs contains two global response contents; each of the response pairs is evaluated based on the global selector model, a target response content is determined in each of the global response pairs, and the global preference dataset is generated based on the preset instruction, each of the global response pairs and each of the target response contents.
[0051] In one embodiment, the preset instruction is input text that guides the pre-trained large language model to generate response content. The server inputs each preset instruction into the pre-trained large language model to generate multiple different global response contents.
[0052] For the multiple global response contents corresponding to each instruction, multiple groups of global response pairs (y0, y1) are formed. For example, if the generated global response content includes y0, y1 and y2, the generated global response pairs are (y0, y1), (y1, y2) and (y0, y2).
[0053] In one embodiment, the instruction and the response pair corresponding to the instruction are input into the global selector model, and the global selector model performs prediction evaluation to determine the preferred target response content in each global response pair.
[0054] The server integrates each instruction, all global response pairs corresponding to the instruction, and the preferred target response content into a global preference dataset.
[0055] Furthermore, the server aggregates the received local selector models to obtain a global selector model, including: the server sends a data diversity analysis instruction to each of the clients, and receives the diversity score of each of the local preference data sets obtained by each of the clients executing the data diversity analysis instruction; determines the weight assigned to each of the local selector models based on the diversity score; and aggregates the local selector models based on the weight to obtain the global selector model.
[0056] In one embodiment, in order to avoid the global selector model obtained by weighted average aggregation being biased towards local features due to the single data of some clients during weighted average aggregation, this embodiment dynamically adjusts the weighted weight corresponding to each client according to the data diversity score of each client, so that the obtained global selector model can more comprehensively capture the diverse preferences of the user group.
[0057] In a specific embodiment, the server sends a data diversity analysis instruction to all clients participating in federated training, requesting them to calculate the diversity score of their local preference datasets and return the score results. The diversity score measures the preference coverage of the client's local preference dataset. A higher score indicates that the local data more comprehensively reflects the user's diverse preferences and contributes more to the global model.
[0058] After the server collects the diversity scores of all clients, it maps the scores to aggregate weights. This mapping can be done through linear normalization, directly normalizing the scores to weights; or through threshold determination, where the weights of the local selector models of clients with scores above or equal to a first preset threshold are set to a first value, and the weights of the local selector models of clients with scores below a second preset threshold and above or equal to a third preset threshold are set to a second value. The first, second, and third preset thresholds can all be set by the user based on actual needs.
[0059] The global selector model is obtained by performing weighted aggregation based on the weight coefficients of the local selector models of each client.
[0060] Furthermore, the server aggregates the received local selector models to obtain a global selector model, and also includes: when the server receives each local selector model, it performs statistical testing on the model parameters of each local selector model to obtain statistical testing results, and identifies abnormal local selector models and non-abnormal local selector models based on the statistical testing results; deletes the abnormal local selector models, and aggregates the non-abnormal local selector models to obtain the global selector model.
[0061] In one embodiment, the server performs statistical testing on the parameters of each local selector model. Specifically, the server may perform statistical testing through outlier testing and an abnormal classification detection method based on machine learning to determine abnormal model parameters.
[0062] In one embodiment, if the statistical test result shows that a local selector model has abnormal parameters, it indicates that the local selector model is an abnormal local selector model. If the statistical test result shows that a local selector model does not have abnormal parameters, it indicates that it is a non-abnormal local selector model.
[0063] In one implementation, anomalous local selector models are removed from the aggregation list, retaining only non-anomalous local selector models. These remaining non-anomalous local selector models are then aggregated to form a global selector model. Anomalous local selector models and non-anomalous local selector models are determined based on statistical tests to determine if the model parameters are abnormal. Only non-anomalous selector models are retained and aggregated to form a global selector model, thus preventing the global selector model from being misled by abnormal data due to abnormal data from some clients.
[0064] The above embodiment provides a preference alignment training method, system, computer device and storage medium for a large language model, wherein the server broadcasts a pre-trained selector model to at least one of the clients; upon receiving the pre-trained selector model, each client trains the pre-trained selector model according to a local preference data set to obtain a local selector model, and sends the local selector model to the server; the server aggregates the received local selector models to obtain a global selector model, and constructs a global preference data set according to the global selector model, and performs preference alignment training on the pre-trained large language model based on the global preference data set to obtain a target large language model. This application broadcasts a pre-trained selector model to the client through the server, and the client trains the pre-trained selector model according to the local preference data set to obtain a local selector model, and sends the local selector model back to the server. The server aggregates the collected local selector models to obtain a global selector, and then constructs a global preference data set according to the global selector to perform preference alignment training on the large language model. The client's local preference dataset is retained locally, and the server only collects the local selector model trained by the client, avoiding the leakage of user data. Secondly, the user preference data of multiple clients are integrated to improve the comprehensiveness and diversity of the global preference dataset, and the accuracy of the preference alignment of the large language model obtained by training with this dataset is improved.
[0065] See also Figure 3 , Figure 3 This is a schematic flow chart of a preference alignment training method for a large language model provided in an embodiment of the present application. This preference alignment training method for a large language model can be applied to a preference alignment training system for a large language model, and is used to randomly group the entire global preference dataset, thereby reducing the amount of training data per iterative round, reducing resource efficiency, and improving training efficiency.
[0066] like Figure 3 As shown, the preference alignment training method of the large language model specifically includes steps S201 to S203.
[0067] S201, the server randomly groups the global preference data set to obtain at least one set of training data;
[0068] In one implementation, in order to avoid excessive consumption of computing resources due to the use of the full global preference dataset, the global preference dataset is divided into multiple groups of small-scale training data.
[0069] Specifically, the global preference data set is first randomly shuffled, and the shuffled data is divided into multiple groups in turn according to the size of each group of training data set by the user.
[0070] S202: Process the current set of training data based on the pre-trained large language model to obtain a current loss function value corresponding to the pre-trained large language model;
[0071] In one embodiment, the server inputs the instructions and the response pairs corresponding to the instructions in each set of training data into the pre-trained large language model, and obtains the output content of the pre-trained large language model, where the output content includes the target response content and the probability of each response content in the response pair generated under the instruction, and calculates the current loss function value according to the loss function formula.
[0072] Among them, the loss function formula is:
[0073]
[0074] Where θ represents the parameters of the target large language model, x is the preset instruction, (y0, y1) represents a response pair, i represents the index of the target response content, i = 0 represents the preference for y0, and i = 1 represents the preference for y1; πθ(y i |x) represents the target large language model’s response y under the parameter θ i The generation probability under instruction x, πθ0(y i |x) represents the response y of the pre-trained large language model i The generation probability under instruction x, β is the balance parameter, and σ is the sigmoid function.
[0075] If the model responds to the better i The generation probability is significantly higher than the suboptimal response y 1-i , the loss function value is close to 0; on the contrary, if the model preference is wrong, the loss function value will increase.
[0076] S203. When the loss function value is greater than or equal to a preset loss function threshold, adjust the parameters of the pre-trained large language model, and process the next set of training data based on the adjusted pre-trained large language model until the obtained loss function value is less than the preset loss threshold, thereby obtaining the target large language model.
[0077] In one embodiment, if the current loss function value is greater than or equal to a preset loss function threshold, indicating that the current model parameters have a large error in their preference prediction for the data set, the gradient of the loss function with respect to the model parameters is calculated, and the optimizer adjusts the model parameters based on the gradient value to obtain a new pre-trained large language model. The new pre-trained large language model is then trained using the next set of data until the loss function value is less than the preset loss function threshold. Training is then stopped to obtain the target large language model. The preset loss function threshold can be set by the user according to actual needs.
[0078] In the above embodiment, the entire global preference data set is randomly grouped, so that the amount of training data in each round of iteration is reduced, resource efficiency is reduced, and training efficiency is improved.
[0079] See also Figure 4 , Figure 4 The embodiment of the present application provides a schematic block diagram of a preference alignment training system for a large language model, which is used to perform the aforementioned preference alignment training method for a large language model. The preference alignment training system for a large language model can be configured in various electronic devices.
[0080] like Figure 4 As shown, the preference alignment training system 300 for the large language model includes:
[0081] Server 301 is configured to broadcast a pre-trained selector model to at least one of the clients; aggregate the received local selector models to obtain a global selector model, construct a global preference dataset based on the global selector model, and perform preference alignment training on the pre-trained large language model based on the global preference dataset to obtain a target large language model;
[0082] The client 302 is configured to train the pre-trained selector model according to the local preference data set upon receiving the pre-trained selector model, obtain a local selector model, and send the local selector model to the server.
[0083] Furthermore, the server 301 is also used to input preset instructions to the pre-trained large language model, obtain at least one global response content output by the pre-trained large language model based on the preset instructions, and generate at least one group of global response pairs based on at least one global response content, wherein a group of global response pairs contains two global response contents; evaluate each of the response pairs based on the global selector model, determine the target response content in each of the global response pairs, and generate the global preference data set based on the preset instructions, each of the global response pairs and each of the target response contents.
[0084] Furthermore, the server 301 is further configured to:
[0085] The server randomly groups the global preference data set to obtain at least one set of training data;
[0086] Processing the current set of training data based on the pre-trained large language model to obtain a current loss function value corresponding to the pre-trained large language model;
[0087] When the loss function value is greater than or equal to a preset loss function threshold, the parameters of the pre-trained large language model are adjusted, and the next set of training data is processed based on the adjusted pre-trained large language model until the obtained loss function value is less than the preset loss threshold, thereby obtaining the target large language model.
[0088] Furthermore, the client 302 is also used to collect local query instructions and at least one group of local response pairs corresponding to each local query instruction; the client determines the preferred response content in each group of the local response pairs; and generates the local preference data set based on the local query instructions, each local response pair and the preferred response content.
[0089] Furthermore, the server 301 is also used to send data diversity analysis instructions to each of the clients, and receive diversity scores of each of the local preference data sets obtained by each of the clients executing the data diversity analysis instructions; determine the weights assigned to each of the local selector models based on the diversity scores; and aggregate each of the local selector models based on the weights to obtain the global selector model.
[0090] Furthermore, the server 301 is also used to perform statistical testing on the model parameters of each local selector model when receiving the local selector model, obtain statistical testing results, and identify abnormal local selector models and non-abnormal local selector models based on the statistical testing results; delete the abnormal local selector model, and aggregate the non-abnormal local selector model to obtain the global selector model.
[0091] Furthermore, the client 302 is also used to train the pre-trained selector model according to the local preference data set to obtain the local selector model; add differential privacy noise to the parameters of the local selector model to obtain an encrypted local selector model; and upload the encrypted local selector model to the server.
[0092] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0093] The above system can be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.
[0094] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device may be a notebook, a PDA, a cloud server, or other computing device.
[0095] See Figure 5 The computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0096] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions that, when executed, enable a processor to perform any preference alignment training method for a large language model.
[0097] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0098] The internal memory provides an environment for the operation of a computer program in a non-volatile storage medium. When the computer program is executed by a processor, the processor can execute a preference alignment training method for any large language model.
[0099] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0100] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0101] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:
[0102] The server broadcasts a pre-trained selector model to at least one of the clients;
[0103] When receiving the pre-trained selector model, each client trains the pre-trained selector model according to the local preference data set to obtain a local selector model, and sends the local selector model to the server;
[0104] The server aggregates the received local selector models to obtain a global selector model, constructs a global preference dataset based on the global selector model, and performs preference alignment training on the pre-trained large language model based on the global preference dataset to obtain a target large language model.
[0105] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program, and the computer program includes program instructions. The processor executes the program instructions to implement any preference alignment training method for a large language model provided in the embodiments of the present application.
[0106] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a SmartMedia Card (SMC), a Secure Digital (SD) card, a flash memory card, etc., equipped on the computer device.
[0107] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A preference alignment training method for a large language model, applied to a preference alignment training system for a large language model, wherein the preference alignment training system for a large language model comprises a server and at least one client, characterized in that: The method comprises: The server broadcasts a pre-trained selector model to at least one of the clients; When receiving the pre-trained selector model, each client trains the pre-trained selector model according to the local preference data set to obtain a local selector model, and sends the local selector model to the server; The server aggregates the received local selector models to obtain a global selector model, constructs a global preference dataset based on the global selector model, and performs preference alignment training on the pre-trained large language model based on the global preference dataset to obtain a target large language model.
2. The preference alignment training method for a large language model according to claim 1, characterized in that The constructing of a global preference dataset according to the global selector model includes: The server inputs a preset instruction to the pre-trained large language model, obtains at least one global response content output by the pre-trained large language model based on the preset instruction, and generates at least one group of global response pairs based on the at least one global response content, wherein a group of global response pairs includes two global response contents; Each of the response pairs is evaluated based on the global selector model, a target response content is determined in each of the global response pairs, and the global preference dataset is generated according to the preset instructions, each of the global response pairs, and each of the target response contents.
3. The preference alignment training method for a large language model according to claim 1, characterized in that The performing preference alignment training on the pre-trained large language model based on the global preference dataset to obtain a target large language model includes: The server randomly groups the global preference data set to obtain at least one set of training data; Processing the current set of training data based on the pre-trained large language model to obtain a current loss function value corresponding to the pre-trained large language model; When the loss function value is greater than or equal to a preset loss function threshold, the parameters of the pre-trained large language model are adjusted, and the next set of training data is processed based on the adjusted pre-trained large language model until the obtained loss function value is less than the preset loss threshold, thereby obtaining the target large language model.
4. The preference alignment training method for a large language model according to claim 1, characterized in that Upon receiving the pre-trained selector model, each of the clients trains the pre-trained selector model according to the local preference data set to obtain a local selector model, and before sending the local selector model to the server, the method further includes: The client collects local query instructions and at least one set of local response pairs corresponding to each local query instruction; The client determines a preferred response content in each group of the local response pairs; The local preference data set is generated based on the local query instruction, each of the local response pairs, and the preference response content.
5. The preference alignment training method for a large language model according to claim 1, characterized in that The server aggregates the received local selector models to obtain a global selector model, including: The server sends a data diversity analysis instruction to each of the clients, and receives a diversity score of each of the local preference data sets obtained by each of the clients executing the data diversity analysis instruction; determining a weight for each of the local selector models based on the diversity score; The local selector models are aggregated based on the weights to obtain the global selector model.
6. The preference alignment training method for a large language model according to claim 1, characterized in that The server aggregates the received local selector models to obtain a global selector model, further comprising: When receiving each of the local selector models, the server performs statistical testing on the model parameters of each of the local selector models to obtain statistical testing results, and identifies abnormal local selector models and non-abnormal local selector models based on the statistical testing results; The abnormal local selector model is deleted, and the non-abnormal local selector models are aggregated to obtain the global selector model.
7. The preference alignment training method for a large language model according to any one of claims 1 to 6, characterized in that: Upon receiving the pre-trained selector model, each of the clients trains the pre-trained selector model according to a local preference data set to obtain a local selector model, and sends the local selector model to the server, including: The client trains the pre-trained selector model according to the local preference dataset to obtain the local selector model; Adding differential privacy noise to the parameters of the local selector model to obtain an encrypted local selector model; The encrypted local selector model is uploaded to the server.
8. A preference alignment training system for a large language model, characterized by: include: A server, configured to broadcast a pre-trained selector model to at least one of the clients; Aggregating the received local selector models to obtain a global selector model, constructing a global preference dataset based on the global selector model, and performing preference alignment training on the pre-trained large language model based on the global preference dataset to obtain a target large language model; The client is used for each of the clients to train the pre-trained selector model according to the local preference data set when receiving the pre-trained selector model, obtain a local selector model, and send the local selector model to the server.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the preference alignment training method for a large language model according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the preference alignment training method for a large language model according to any one of claims 1 to 7.