Big language model fine tuning method and device based on federal learning, equipment and medium

Abstract

The invention discloses a federated learning-based large language model fine tuning method, apparatus and device, and a medium, relates to the technical field of federated learning, is applied to a server, and comprises the steps of obtaining a federated fine tuning task, and processing data in the federated fine tuning task by using a preset experience guidance model to obtain corresponding feature vectors; issuing the feature vector, a preset experience guidance model and a basic large language model to each client, so that each client performs model fine tuning processing on the basic large language model based on the local fine tuning data set, the feature vector and the preset experience guidance model to obtain a fine-tuned large language model, and performing aggregation processing on the fine-tuned large language model to obtain a new feature vector, and then skipping to the step of issuing the feature vector, the preset experience guidance model and the basic large language model to each client until a preset fine-tuning training ending condition is met. In this way, fine tuning of the large language model is achieved through personalized federal learning.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and particularly to a method, device, equipment and medium for fine-tuning large language models based on federated learning. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, LLMs (Large Language Models) have received great attention due to their powerful expressive ability and wide application potential, and have become an important tool in the field of NLP (Natural Language Processing). These models are trained with a large amount of text data and can generate coherent and logical text, supporting a series of complex tasks such as natural language understanding, text generation, and dialogue systems. With more and more large language models being open-sourced, the academic and industrial communities have been able to use these models as a starting point to develop application programs that meet specific needs.

[0003] In practical applications, due to the huge scale of large language model parameters, full-scale adjustment may face computational resource limitations. Therefore, researchers usually adopt the fine-tuning method, that is, using data in a specific domain to further train a pre-trained model. LoRA (Low-Rank Adaptation) is the most commonly used fine-tuning method at present. By inserting low-rank matrices in the key layers of the model and only adjusting a small part of the parameters, the training time and resource consumption are significantly reduced, which has alleviated the efficiency problem of large language model fine-tuning to a certain extent. Nevertheless, fine-tuning large language models still faces a series of challenges. First, efficient fine-tuning techniques such as LoRA simplify the training process by only adjusting a small part of the parameters in the model. However, considering the number of parameters of large language models, which are often in the billions or more, even 5%-10% of the parameters are quite large, and still require a large amount of computational resources and time.

[0004] Therefore, how to effectively utilize the training process data of all parties, combined with personalized federated learning technology, to jointly promote the improvement of the fine-tuning performance and accuracy of large language models while protecting data privacy and security is an urgent problem to be solved at present. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for fine-tuning large language models based on federated learning, which can utilize the training process data of all parties, combined with personalized federated learning technology, to jointly promote the improvement of the fine-tuning performance and accuracy of large language models. The specific solutions are as follows:

[0006] In a first aspect, the present application provides a method for fine-tuning a large language model based on federated learning, which is applied to a server side and includes:

[0007] Obtain a federated fine-tuning task generated based on an initial fine-tuning task created by a federated modeling task initiator, and use a preset experience guiding model to process the initial metadata and the federated learning global parameters in the federated fine-tuning task respectively to obtain a fine-tuning metadata feature vector and a global model feature vector; the preset experience guiding model is a model obtained by training a preset deep learning model based on training metadata and model training parameters in a historical model fine-tuning process;

[0008] Send the fine-tuning metadata feature vector, the global model feature vector, the preset experience guiding model, and the basic large language model to each client, so that each client performs model fine-tuning processing on the basic large language model based on a local fine-tuning data set, the fine-tuning metadata feature vector, the global model feature vector, and the preset experience guiding model to obtain a fine-tuned large language model, and perform noise addition processing on the fine-tuned large language model to obtain processed feature parameters;

[0009] Perform aggregation processing on the processed feature parameters to obtain new federated learning global parameters, and use the preset experience guiding model to process the new federated learning global parameters to obtain a new global model feature vector, and then jump to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guiding model, and the basic large language model to each client until a preset fine-tuning training end condition is met, so as to complete the fine-tuning of the basic large language model by each client.

[0010] Optionally, the obtaining of the federated fine-tuning task generated based on the initial fine-tuning task created by the federated modeling task initiator includes:

[0011] Obtain the initial fine-tuning task created by the federated modeling task initiator;

[0012] Obtain the fine-tuning task details adjusted by each client for the initial fine-tuning task based on the target business domain;

[0013] Integrate the fine-tuning task details to obtain a federated fine-tuning task.

[0014] Optionally, the using of the preset experience guiding model to process the initial metadata and the federated learning global parameters in the federated fine-tuning task respectively to obtain a fine-tuning metadata feature vector and a global model feature vector includes:

[0015] Use the metadata processing tool in the preset experience guidance model to extract features from the initial metadata in the federated fine-tuning task to obtain the fine-tuning metadata feature vector;

[0016] Initialize the federated learning global parameters at the locally preset aggregation node, and use the parameter processing tool in the preset experience guidance model to extract features from the initialized federated learning global parameters to obtain the global model feature vector.

[0017] Optionally, the client performs model fine-tuning on the base large language model based on the local fine-tuning dataset, the fine-tuning metadata feature vector, the global model feature vector, and the preset experience guidance model to obtain the fine-tuned large language model, and performs noise addition processing on the fine-tuned large language model to obtain the processed feature parameters, including:

[0018] The client obtains the local fine-tuning dataset and loads the local fine-tuning dataset, the preset experience guidance model, and the base large language model into the local data center;

[0019] The client obtains the current state data of the base large language model and uses the preset experience guidance model in the local data center to process the current state data, the fine-tuning metadata feature vector, and the global model feature vector to generate a model fine-tuning strategy and prediction parameter values;

[0020] The client fine-tunes the base large language model based on the model fine-tuning strategy to obtain the fine-tuning large language model during fine-tuning, and adjusts the current parameters of the fine-tuning large language model based on the prediction parameter values to obtain the fine-tuned large language model.

[0021] Optionally, the client fine-tunes the base large language model based on the model fine-tuning strategy to obtain the fine-tuning large language model during fine-tuning, including:

[0022] The client generates a current fine-tuning strategy based on the preset fine-tuning method corresponding to the target business domain and the model fine-tuning strategy, and fine-tunes the base large language model based on the current fine-tuning strategy to obtain the fine-tuning large language model during fine-tuning.

[0023] Optionally, the client performs noise addition processing on the fine-tuned large language model to obtain the processed feature parameters, including:

[0024] The client performs local blurring on the fine-tuned large language model and adds noise to obtain the processed feature parameters.

[0025] Optionally, aggregate the processed feature parameters to obtain new global parameters for federated learning, and use the preset experience guidance model to process the new global parameters for federated learning to obtain a new global model feature vector, and then jump to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client until the preset fine-tuning training end condition is met to complete the fine-tuning of the basic large language model by each client, including:

[0026] Obtain the processed feature parameters and model performance verification results uploaded by each client through a locally preset aggregation node, and aggregate the processed feature parameters to obtain new global parameters for federated learning;

[0027] Based on the model performance verification result, determine whether the preset fine-tuning training end condition is met;

[0028] If the preset fine-tuning training end condition is not met, use the preset experience guidance model to process the new global parameters for federated learning to obtain a new global model feature vector and jump to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client;

[0029] If the preset fine-tuning training end condition is met, stop the model fine-tuning training to complete the fine-tuning of the basic large language model by each client.

[0030] In a second aspect, the present application provides a large language model fine-tuning device based on federated learning, which is applied to a server and includes:

[0031] A feature vector obtaining module, configured to obtain a federated fine-tuning task generated based on an initial fine-tuning task created by a federated modeling task initiator, and use a preset experience guidance model to process the initial metadata and global parameters for federated learning in the federated fine-tuning task respectively to obtain a fine-tuning metadata feature vector and a global model feature vector; the preset experience guidance model is a model obtained by training a preset deep learning model based on training metadata and model training parameters in a historical model fine-tuning process;

[0032] A feature parameter obtaining module, configured to send the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client, so that each client performs model fine-tuning processing on the basic large language model based on a local fine-tuning data set, the fine-tuning metadata feature vector, the global model feature vector, and the preset experience guidance model to obtain a fine-tuned large language model, and perform noise addition processing on the fine-tuned large language model to obtain processed feature parameters;

[0033] A model fine-tuning module, configured to aggregate the processed feature parameters to obtain new global parameters for federated learning, and use the preset experience guidance model to process the new global parameters for federated learning to obtain a new global model feature vector, and then jump to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client until a preset fine-tuning training end condition is met, so as to complete the fine-tuning of the basic large language model by each client.

[0034] In a third aspect, the present application provides an electronic device, including:

[0035] A memory, configured to store a computer program;

[0036] A processor, configured to execute the computer program to implement the foregoing method for fine-tuning a large language model based on federated learning.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program; wherein, when the computer program is executed by a processor, the foregoing method for fine-tuning a large language model based on federated learning is implemented.

[0038] As can be seen from the above, the present application first obtains a federated fine-tuning task generated based on an initial fine-tuning task created by a federated modeling task initiator, and uses a preset experience guidance model to process the initial metadata and the federated learning global parameters in the federated fine-tuning task respectively to obtain a fine-tuned metadata feature vector and a global model feature vector; the preset experience guidance model is a model obtained by training a preset deep learning model based on training metadata and model training parameters in a historical model fine-tuning process; the fine-tuned metadata feature vector, the global model feature vector, the preset experience guidance model, and a basic large language model are sent to each client, so that each client fine-tunes the basic large language model based on a local fine-tuning dataset, the fine-tuned metadata feature vector, the global model feature vector, and the preset experience guidance model to obtain a fine-tuned large language model, and performs noise addition processing on the fine-tuned large language model to obtain processed feature parameters; the processed feature parameters are aggregated to obtain new federated learning global parameters, and the preset experience guidance model is used to process the new federated learning global parameters to obtain a new global model feature vector, and then jumps to the step of sending the fine-tuned metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client until a preset fine-tuning training end condition is met, so as to complete the fine-tuning of the basic large language model by each client. As can be seen from the above, the server of the present application first obtains a federated fine-tuning task, and then uses a preset experience guidance model to process the parameters in the federated fine-tuning task to obtain a fine-tuned metadata feature vector and a global model feature vector. Next, the feature vectors, the basic large language model, and the preset experience guidance model are sent to the client, so that each client fine-tunes the basic large language model, performs noise addition processing on the fine-tuned large language model to obtain processed feature parameters, then aggregates the parameters to obtain a new global model feature vector, and finally continues to fine-tune the basic large language model with the new global model feature vector, repeating the above operations to continuously adjust the basic large language model until a preset fine-tuning training end condition is met, so as to complete the fine-tuning of the basic large language model by each client. In this way, the training process data of all parties is effectively utilized, and combined with personalized federated learning technology, in a distributed environment, multiple clients participate in a large model fine-tuning modeling task based on the same basic large language model, so that each client generates a personalized model suitable for its own dataset, rather than jointly training a single common model, thereby ensuring the privacy of the original parameters of each client and improving the fine-tuning performance and accuracy of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0040] Figure 1 It is a system architecture diagram of a method for fine-tuning a large language model based on federated learning disclosed in this application;

[0041] Figure 2 It is a flowchart of a method for fine-tuning a large language model based on federated learning disclosed in this application;

[0042] Figure 3 It is a schematic structural diagram of a preset experience guidance model disclosed in this application;

[0043] Figure 4 It is a schematic structural diagram of a device for fine-tuning a large language model based on federated learning disclosed in this application;

[0044] Figure 5 It is a structural diagram of an electronic device disclosed in this application. Specific embodiments

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0046] Currently, the most commonly used fine-tuning method inserts low-rank matrices in the key layers of the model, only adjusting a small part of the parameters, thereby reducing a certain amount of training time and resource consumption. However, fine-tuning large language models still faces a series of challenges. Considering the number of parameters of large language models, which are often in the billions or even more, even 5%-10% of the parameters is quite large, still requiring a large amount of computing resources and time. Therefore, this application will specifically introduce a method for fine-tuning a large language model based on federated learning that can solve the above problems.

[0047] For the system architecture diagram of the method for fine-tuning a large language model based on federated learning in this application, refer to Figure 1As shown in the figure, the system mainly consists of a server and multiple clients. Among them, the server coordinates the modeling process through the sharing of large model fine-tuning experience and knowledge, including a basic large language model management module, a preset experience guidance model construction and management module, a federated personalized fine-tuning modeling task management module, a federated personalized fine-tuning modeling participating node management module, a gradient aggregation and update module, a global model distribution module, a monitoring and coordination module, etc. The basic large language model management module is responsible for managing the version control, storage, and distribution of the basic large language model; the preset experience guidance model construction and management module is responsible for constructing and managing the preset experience guidance model for predicting parameter change trends; the federated personalized fine-tuning modeling task management module is responsible for managing the life cycle of the entire federated fine-tuning modeling task, including task allocation, progress tracking, task coordination, etc.; the federated personalized fine-tuning modeling participating node management module is responsible for managing each client node participating in the federated modeling, including node registration, authentication, status monitoring, etc.; the gradient aggregation and update module is responsible for aggregating the gradient updates from each client and generating new global model parameters; the global model distribution module is responsible for distributing the new global model parameters to each client for the next round of iteration. The monitoring and coordination module is responsible for monitoring the status of the entire modeling process and coordinating the work between each module.

[0048] See Figure 2 As shown in the figure, an embodiment of the present invention discloses a method for fine-tuning a large language model based on federated learning, which is applied to a server and may include:

[0049] Step S11: Obtain a federated fine-tuning task generated based on an initial fine-tuning task created by a federated modeling task initiator, and use a preset experience guidance model to process the initial metadata and federated learning global parameters in the federated fine-tuning task respectively to obtain a fine-tuning metadata feature vector and a global model feature vector; the preset experience guidance model is a model obtained by training a preset deep learning model based on training metadata and model training parameters in a historical model fine-tuning process.

[0050] In this embodiment, before obtaining the federated fine-tuning task generated based on the initial fine-tuning task created by the federated modeling task initiator, the server first sets up a task management node and an aggregation node, and provides directory metadata such as the federated modeling task, the basic large language model, and the preset experience guidance model. Then, each client data center initiates registration to the server, and the identity of each client is confirmed through authentication. It should be noted that the federated modeling task refers to a distributed environment where multiple clients participate in the fine-tuning modeling task based on the same basic large language model. However, the goal of each client is to train a personalized model suitable for its own dataset, rather than jointly training a single common model. The federated modeling task is created by the federated modeling task initiator and is managed by the task management node of the server's federated modeling center and coordinated with each client.

[0051] In this embodiment, obtaining the federated fine-tuning task generated based on the initial fine-tuning task created by the federated modeling task initiator includes: obtaining the initial fine-tuning task created by the federated modeling task initiator; obtaining the fine-tuning task details after each client adjusts the initial fine-tuning task based on the target business domain; and obtaining the federated fine-tuning task by integrating the fine-tuning task details. Specifically, the federated modeling task initiator creates an initial fine-tuning task, specifying the specific basic large model and other parameters, including but not limited to fine-tuning training metadata such as resource requirements, data requirements, and fine-tuning methods. Next, each client adjusts the initial fine-tuning task according to the target business domain to confirm the final fine-tuning task details, and then integrates the fine-tuning task details to obtain the federated fine-tuning task.

[0052] In this embodiment, the preset deep learning model is trained based on the training metadata and model training parameters in the historical model fine-tuning process to obtain a preset experience guidance model. The preset experience guidance model is used to predict the possible change direction and degree of the next-step parameters according to the parameters in the current fine-tuning stage, so as to guide the fine-tuning process. The schematic structural diagram of the preset experience guidance model is as Figure 3As shown, it can be understood that the preset experience guidance model is mainly composed of a metadata processing tool (Train-MetaData-Process-Module), a parameter snapshot processing tool (Model-Snapshot-Process-Module), a federated learning global parameter processing tool (FL-Global-Process-Module), and a fine-tuning guidance parameter prediction tool (Para-Predict-Module). Among them, the metadata processing tool is responsible for processing various metadata related to the fine-tuning process, extracting useful feature information and preprocessing the feature information. The metadata mainly includes fine-tuning stage metadata such as the fine-tuning stage, stage duration, and stage target, fine-tuning hyperparameters such as the learning rate, regularization coefficient, and batch size, training data features such as data scale, data distribution, data noise level, and data missing value ratio, model metadata such as model architecture, number of layers, number of neurons, activation function, and optimization algorithm, and training configuration such as hardware configuration and resource situation. The parameter snapshot processing tool is responsible for processing the model parameter snapshot data during the current model fine-tuning process, capturing the complex patterns of parameter changes through the processing of the neural network, extracting useful feature information, and converting the feature information into an easily understandable feature representation. Among them, the model parameter snapshot data includes the parameter status of the current model, number of training rounds, training loss function value, resource occupancy, convergence speed, loss function fluctuation, etc., and is used to characterize the state data of the fine-tuning training process. The federated learning global parameter processing tool is responsible for processing the global parameter data from the central node of federated learning, preprocessing the global parameter data, and extracting the features of the global model. The fine-tuning guidance parameter prediction tool is the core module of the preset experience guidance model, which is used to predict the parameter change value and hyperparameter suggestion value through a deep learning model for the feature vectors from the metadata processing tool, the parameter snapshot processing tool, and the federated learning global parameter processing tool, and guide the subsequent fine-tuning process. The core of the fine-tuning guidance parameter prediction tool is to design different neural networks according to different fine-tuning stage settings to better mine the temporal features and the dependence relationships between different parameters.

[0053] In this embodiment, the process of using the preset experience guidance model to process the initial metadata and the global parameters of federated learning in the federated fine-tuning task to obtain the fine-tuned metadata feature vector and the global model feature vector includes: using the metadata processing tool in the preset experience guidance model to extract features from the initial metadata in the federated fine-tuning task to obtain the fine-tuned metadata feature vector; initializing the global parameters of federated learning at a locally preset aggregation node, and using the parameter processing tool in the preset experience guidance model to extract features from the initialized global parameters of federated learning to obtain the global model feature vector. That is to say, the server uses the metadata processing tool in the preset experience guidance model to extract features from the initial metadata, and preprocesses the extracted feature data to obtain the fine-tuned metadata feature vector. The preprocessing operations include but are not limited to data cleaning, data transformation, data integration, etc. After the aggregation node established on the server side initializes the global parameters of federated learning, the global model feature vector is generated using the global parameter processing tool of federated learning.

[0054] Step S12: Send the fine-tuned metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client, so that each client can perform model fine-tuning on the basic large language model based on the local fine-tuning dataset, the fine-tuned metadata feature vector, the global model feature vector, and the preset experience guidance model to obtain the fine-tuned large language model, and perform noise addition processing on the fine-tuned large language model to obtain the processed feature parameters.

[0055] In this embodiment, after each client obtains the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model sent by the server, the client obtains the local fine-tuning data set and loads the local fine-tuning data set, the preset experience guidance model, and the basic large language model into the local data center; the client obtains the current state data of the basic large language model and uses the preset experience guidance model in the local data center to process the current state data, the fine-tuning metadata feature vector, and the global model feature vector to generate a model fine-tuning strategy and prediction parameter values; the client fine-tunes the basic large language model based on the model fine-tuning strategy to obtain a large language model during fine-tuning, and adjusts the current parameters of the large language model during fine-tuning based on the prediction parameter values to obtain a fine-tuned large language model. Specifically, the task management module of the client receives the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model sent by the server, and then uses the data management module of the client to prepare the local fine-tuning data set and preprocess the local fine-tuning data set, including data cleaning and data formatting. Next, use the model loading module of the client to load the processed local fine-tuning data set, the preset experience guidance model, and the basic large language model into the local data center to prepare for local fine-tuning. The dynamic generation of fine-tuning strategy module of the client uses the preset experience guidance model to process the current state data of the basic large language model, the fine-tuning metadata feature vector, and the global model feature vector to generate a model fine-tuning strategy and prediction parameter values, where the prediction parameter values include prediction parameter values and recommended hyperparameters. The local fine-tuning training module of the client fine-tunes the basic large language model based on the model fine-tuning strategy to obtain a large language model during fine-tuning, and after the fine-tuning parameter update in this round, uses the prediction parameter values to adjust the current parameters and current hyperparameters of the large language model during fine-tuning to obtain a fine-tuned large language model. It should be noted that the adjustment methods of the current parameters include, but are not limited to, weighted addition, threshold addition, etc.

[0056] In this embodiment, the client generates a current fine-tuning strategy based on the preset fine-tuning method corresponding to the target business field and the model fine-tuning strategy, and fine-tunes the basic large language model based on the current fine-tuning strategy to obtain a large language model during fine-tuning. Specifically, the LORA method can be used to combine the model fine-tuning strategy to generate the current fine-tuning strategy, and then the basic large language model is fine-tuned based on the current fine-tuning strategy to obtain a large language model during fine-tuning.

[0057] In this embodiment, the client performs local fuzzification on the fine-tuned large language model and adds noise to obtain processed feature parameters. It can be understood that the local fuzzification can hide or blur some key information in the fine-tuned large language model to prevent the fine-tuned large language model from being copied by malicious attackers. At the same time, through the local fuzzification process of the fine-tuned large language model, the over-reliance of the fine-tuned large language model on specific training data can be reduced, thereby reducing the risk of overfitting. By adding noise, the fine-tuned large language model can learn more data variation patterns, thereby enhancing the processing ability for noisy data. This helps to improve the robustness of the fine-tuned large language model in practical applications to better handle various complex scenarios.

[0058] Step S13: Aggregate the processed feature parameters to obtain new federated learning global parameters, and use the preset experience guidance model to process the new federated learning global parameters to obtain new global model feature vectors, and then jump to the step of distributing the fine-tuning metadata feature vectors, the global model feature vectors, the preset experience guidance model, and the basic large language model to each client until the preset fine-tuning training end condition is met to complete the fine-tuning of the basic large language model by each client.

[0059] In this embodiment, the local preset aggregation node obtains the processed feature parameters and model performance verification results uploaded by each client, and aggregates the processed feature parameters to obtain new federated learning global parameters; that is, the server collects the processed feature parameters uploaded by each client and aggregates the processed feature parameters to generate new global model parameters.

[0060] In this embodiment, it is determined whether the preset fine-tuning training end condition is satisfied based on the model performance verification result; if the preset fine-tuning training end condition is not satisfied, the preset experience guidance model is used to process the new federated learning global parameters to obtain a new global model feature vector, and then it jumps to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client; if the preset fine-tuning training end condition is satisfied, the model fine-tuning training is stopped to complete the fine-tuning of the basic large language model by each client. Specifically, the client performs model performance verification on the local fine-tuning dataset, that is, determines whether the fine-tuned large language model converges or meets the preset fine-tuning training rounds. If the fine-tuned large language model converges or has met the preset fine-tuning training rounds, the model performance verification result is sent to the server for further verification. The server obtains the model performance verification result and determines whether the preset fine-tuning training end condition is satisfied based on the model performance verification result. At this time, some clients that meet the preset fine-tuning training end condition can exit the training first, and the remaining clients that do not meet the preset fine-tuning training end condition continue to perform fine-tuning training. When all clients meet the preset fine-tuning training end condition, the fine-tuning training of the system ends. It should be noted that for clients that do not meet the preset fine-tuning training end condition, the global model feature vector is extracted by using the federated learning global parameter processing tool in the preset experience guidance model to obtain a new global model feature vector, and then it jumps to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client, and repeats the process of local fine-tuning training, dynamic adjustment strategy, and fuzzy noise-added upload until the preset fine-tuning training end condition is satisfied.

[0061] As can be seen from the above, the server in this embodiment first obtains the federated fine-tuning task, and then uses the preset experience guidance model to process the parameters in the federated fine-tuning task to obtain the fine-tuning metadata feature vector and the global model feature vector. Next, the feature vectors, the basic large language model, and the preset experience guidance model are sent to the client, so that each client can fine-tune the basic large language model, perform noise addition processing on the fine-tuned large language model to obtain the processed feature parameters, then perform aggregation processing on the parameters to obtain a new global model feature vector, and finally continue to fine-tune the basic large language model with the new global model feature vector, repeating the above operations to continuously adjust the basic large language model until the preset fine-tuning training end condition is met, so as to complete the fine-tuning of the basic large language model by each client. In this way, the training process data of all parties is effectively utilized, and combined with the personalized federated learning technology, in a distributed environment, multiple clients participate in a large model fine-tuning modeling task based on the same basic large language model, so that each client can generate a personalized model suitable for its own dataset, rather than jointly training a single common model, thereby ensuring the privacy of the original parameters of each client and improving the fine-tuning performance and accuracy of the large language model.

[0062] Correspondingly, referring to Figure 4 as shown, the embodiment of the present application also provides a large language model fine-tuning device based on federated learning, which is applied to the server and may include:

[0063] A feature vector obtaining module 11, configured to obtain a federated fine-tuning task generated based on an initial fine-tuning task created by a federated modeling task initiator, and use a preset experience guidance model to process the initial metadata and the federated learning global parameters in the federated fine-tuning task respectively to obtain a fine-tuning metadata feature vector and a global model feature vector; the preset experience guidance model is a model obtained by training a preset deep learning model based on the training metadata and model training parameters in the historical model fine-tuning process;

[0064] A feature parameter obtaining module 12, configured to send the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client, so that each client can perform model fine-tuning processing on the basic large language model based on the local fine-tuning dataset, the fine-tuning metadata feature vector, the global model feature vector, and the preset experience guidance model to obtain a fine-tuned large language model, and perform noise addition processing on the fine-tuned large language model to obtain processed feature parameters;

[0065] The model fine-tuning module 13 is used to aggregate the processed feature parameters to obtain new global parameters for federated learning, and use the preset experience guidance model to process the new global parameters for federated learning to obtain a new global model feature vector, and then jump to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client until the preset fine-tuning training end condition is met, so as to complete the fine-tuning of the basic large language model by each client.

[0066] It can be seen that in this embodiment, the training process data of all parties is effectively utilized, and combined with the personalized federated learning technology, in a distributed environment, multiple clients participate in a large model fine-tuning modeling task based on the same basic large language model, so that each client can generate a personalized model suitable for its own dataset, rather than jointly training a single common model, thereby ensuring the privacy of the original parameters of each client and improving the fine-tuning performance and accuracy of the large language model.

[0067] In some specific embodiments, the feature vector obtaining module 11 includes:

[0068] An initial task obtaining unit, configured to obtain an initial fine-tuning task created by a federated modeling task initiator;

[0069] An initial task adjustment unit, configured to obtain the fine-tuning task details after each client adjusts the initial fine-tuning task based on the target business domain;

[0070] A federated task obtaining unit, configured to obtain a federated fine-tuning task by integrating the fine-tuning task details.

[0071] In some specific embodiments, the feature vector obtaining module 11 includes:

[0072] A metadata feature vector obtaining unit, configured to use the metadata processing tool in the preset experience guidance model to extract features from the initial metadata in the federated fine-tuning task to obtain a fine-tuning metadata feature vector;

[0073] A global model feature vector obtaining unit, configured to initialize the global parameters for federated learning at a locally preset aggregation node, and use the parameter processing tool in the preset experience guidance model to extract features from the initialized global parameters for federated learning to obtain a global model feature vector.

[0074] In some specific embodiments, the feature parameter obtaining module 12 includes:

[0075] A data loading unit, which is used for the client to obtain a local fine-tuning data set and load the local fine-tuning data set, the preset experience guidance model, and the basic large language model into the local data center;

[0076] A policy generation unit, which is used for the client to obtain the current state data of the basic large language model, and in the local data center, use the preset experience guidance model to process the current state data, the fine-tuning metadata feature vector, and the global model feature vector to generate a model fine-tuning policy and prediction parameter values;

[0077] A parameter adjustment sub-module, which is used for the client to fine-tune the basic large language model based on the model fine-tuning policy to obtain a fine-tuning large language model during fine-tuning, and adjust the current parameters of the fine-tuning large language model based on the prediction parameter values to obtain a fine-tuned large language model.

[0078] In some specific embodiments, the parameter adjustment sub-module includes:

[0079] A model fine-tuning unit, which is used for the client to generate a current fine-tuning policy based on a preset fine-tuning method corresponding to the target business domain and the model fine-tuning policy, and fine-tune the basic large language model based on the current fine-tuning policy to obtain a fine-tuning large language model during fine-tuning.

[0080] In some specific embodiments, the feature parameter acquisition module 12 includes:

[0081] A feature parameter acquisition unit, which is used for the client to perform local fuzzification on the fine-tuned large language model and add noise to obtain processed feature parameters.

[0082] In some specific embodiments, the model fine-tuning module 13 includes:

[0083] A parameter update unit, which is used to obtain the processed feature parameters and model performance verification results uploaded by each client through a local preset aggregation node, and perform aggregation processing on the processed feature parameters to obtain new federated learning global parameters;

[0084] A fine-tuning judgment unit, which is used to judge whether the preset fine-tuning training end condition is satisfied based on the model performance verification result;

[0085] A parameter processing unit, which is used to, if the preset fine-tuning training end condition is not satisfied, use the preset experience guidance model to process the new federated learning global parameters to obtain a new global model feature vector and jump to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client;

[0086] A fine-tuning end unit, configured to stop the model fine-tuning training to complete the fine-tuning of the base large language model by each of the clients if a preset fine-tuning training end condition is met.

[0087] Furthermore, an embodiment of the present application also discloses an electronic device. Figure 5 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure should not be considered as any limitation to the scope of use of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the method for fine-tuning a large language model based on federated learning disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0088] In this embodiment, the power supply 23 is used to provide working voltages for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application requirements, and no specific limitation is made here.

[0089] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a disk, or an optical disc, etc., and the resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0090] Among them, the operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, and it may be Windows Server, Netware, Unix, Linux, etc. The computer program 222 may further include a computer program capable of completing other specific tasks in addition to the computer program capable of implementing the method for fine-tuning a large language model based on federated learning executed by the electronic device 20 disclosed in any of the foregoing embodiments.

[0091] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the method for fine-tuning a large language model based on federated learning disclosed above. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details are not repeated here.

[0092] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method section.

[0093] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0094] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0095] Finally, it should also be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0096] The above has introduced the technical solution provided by this application in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A method for fine-tuning a large language model based on federated learning, characterized in that: Applied to the server, including: Obtain a federated fine-tuning task generated based on an initial fine-tuning task created by a federated modeling task initiator, and use a preset experience guidance model to process the initial metadata and federated learning global parameters in the federated fine-tuning task to obtain a fine-tuning metadata feature vector and a global model feature vector; the preset experience guidance model is a model obtained by training a preset deep learning model based on training metadata and model training parameters in a historical model fine-tuning process; The fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model and the basic large language model are sent to each client, so that each client performs model fine-tuning processing on the basic large language model based on the local fine-tuning dataset, the fine-tuning metadata feature vector, the global model feature vector and the preset experience guidance model to obtain a fine-tuned large language model, and performs noise processing on the fine-tuned large language model to obtain processed feature parameters; The processed feature parameters are aggregated to obtain new federated learning global parameters, and the new federated learning global parameters are processed using the preset experience guidance model to obtain a new global model feature vector, and then the process jumps to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model and the basic large language model to each client until the preset fine-tuning training end condition is met, so as to complete the fine-tuning of the basic large language model by each client.

2. The method for fine-tuning a large language model based on federated learning according to claim 1, characterized in that: The obtaining of the federated fine-tuning task generated based on the initial fine-tuning task created by the initiator of the federated modeling task includes: Obtain an initial fine-tuning task created by a federated modeling task initiator; Acquire the fine-tuning task details after each of the clients adjusts the initial fine-tuning task based on the target business field; The federated fine-tuning task is obtained by integrating the details of each fine-tuning task.

3. The method for fine-tuning a large language model based on federated learning according to claim 1, characterized in that: The method of using a preset experience guidance model to process the initial metadata and the federated learning global parameters in the federated fine-tuning task to obtain a fine-tuning metadata feature vector and a global model feature vector includes: Using a metadata processing tool in a preset experience guidance model to extract features from the initial metadata in the federated fine-tuning task to obtain a fine-tuning metadata feature vector; The federated learning global parameters are initialized at a local preset aggregation node, and the parameter processing tool in the preset experience guidance model is used to extract features of the initialized federated learning global parameters to obtain a global model feature vector.

4. The method for fine-tuning a large language model based on federated learning according to claim 1, characterized in that: The client performs model fine-tuning processing on the basic large language model based on the local fine-tuning data set, the fine-tuning metadata feature vector, the global model feature vector, and the preset experience guidance model to obtain a fine-tuned large language model, and performs noise processing on the fine-tuned large language model to obtain processed feature parameters, including: The client obtains a local fine-tuning dataset, and loads the local fine-tuning dataset, the preset experience-guided model, and the basic large language model into a local data center; The client obtains the current state data of the basic large language model, and uses the preset experience guidance model in the local data center to process the current state data, the fine-tuning metadata feature vector and the global model feature vector to generate a model fine-tuning strategy and a prediction parameter value; The client fine-tunes the basic large language model based on the model fine-tuning strategy to obtain a fine-tuned large language model, and adjusts current parameters of the fine-tuned large language model based on the predicted parameter values ​​to obtain a fine-tuned large language model.

5. The method for fine-tuning a large language model based on federated learning according to claim 4, characterized in that: The client fine-tunes the basic large language model based on the model fine-tuning strategy to obtain a fine-tuned large language model, including: The client generates a current fine-tuning strategy based on a preset fine-tuning method corresponding to the target business field and the model fine-tuning strategy, and fine-tunes the basic large language model based on the current fine-tuning strategy to obtain a fine-tuned large language model.

6. The method for fine-tuning a large language model based on federated learning according to claim 4, characterized in that: The client performs noise processing on the fine-tuned large language model to obtain processed feature parameters, including: The client performs local fuzzification on the fine-tuned large language model and adds noise to obtain processed feature parameters.

7. The method for fine-tuning a large language model based on federated learning according to any one of claims 1 to 6, characterized in that: The step of aggregating the processed feature parameters to obtain new federated learning global parameters, and using the preset experience guidance model to process the new federated learning global parameters to obtain a new global model feature vector, and then jumping to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model, and the basic large language model to each client until the preset fine-tuning training end condition is met, so as to complete the fine-tuning of the basic large language model by each client, including: Obtaining the processed feature parameters and model performance verification results uploaded by each of the clients through a locally preset aggregation node, and aggregating the processed feature parameters to obtain new federated learning global parameters; Determine whether a preset fine-tuning training end condition is met based on the model performance verification result; If the preset fine-tuning training end condition is not met, the new federated learning global parameters are processed using the preset experience guidance model to obtain a new global model feature vector and the process jumps to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model and the basic large language model to each client; If the preset fine-tuning training end condition is met, the model fine-tuning training is stopped to complete the fine-tuning of the basic large language model by each of the clients.

8. A large language model fine-tuning device based on federated learning, characterized in that: Applied to the server, including: A feature vector acquisition module is used to obtain a federated fine-tuning task generated based on an initial fine-tuning task created by a federated modeling task initiator, and to process the initial metadata and federated learning global parameters in the federated fine-tuning task using a preset experience guidance model to obtain a fine-tuning metadata feature vector and a global model feature vector; the preset experience guidance model is a model obtained by training a preset deep learning model based on training metadata and model training parameters in a historical model fine-tuning process; A feature parameter acquisition module, used to send the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model and the basic large language model to each client, so that each client performs model fine-tuning processing on the basic large language model based on the local fine-tuning dataset, the fine-tuning metadata feature vector, the global model feature vector and the preset experience guidance model to obtain a fine-tuned large language model, and performs noise processing on the fine-tuned large language model to obtain processed feature parameters; A model fine-tuning module is used to aggregate the processed feature parameters to obtain new federated learning global parameters, and use the preset experience guidance model to process the new federated learning global parameters to obtain a new global model feature vector, and then jump to the step of sending the fine-tuning metadata feature vector, the global model feature vector, the preset experience guidance model and the basic large language model to each client until the preset fine-tuning training end condition is met, so as to complete the fine-tuning of the basic large language model by each client.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the large language model fine-tuning method based on federated learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the large language model fine-tuning method based on federated learning as described in any one of claims 1 to 7 is implemented.

Images