A method and system for fine-tuning a large model based on domain adaptation of enterprise private data

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a domain self-adaptive large model fine-tuning method and system based on enterprise private data, the method comprising: each participating client extracts the feature representation of local data using a unified pre-training model, and generates a prototype set representing the category features of the local domain by using mean, clustering or random sampling methods; uploading the prototype set to the server after adding Gaussian random noise to realize differential privacy protection; the server aggregates all client prototypes to construct a global prototype training set, and performs global training of the adapter module in a single communication round using supervised learning classification loss without averaging the same prototypes of different clients, obtains a global adapter model and distributes it to the client; the client initializes the local model based on the global adapter model, fine-tunes it using a small amount of labeled data, and uses the global model as a teacher to prevent catastrophic forgetting through knowledge distillation, achieving a balance between global knowledge retention and local performance improvement. The present application can solve the problem that the existing technology cannot effectively align the domain features of different enterprises, resulting in a significant decline in the performance of the model in cross-domain reasoning tasks.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for fine-tuning a domain-adaptive large model based on enterprise private data. Background Technology

[0002] With the development of artificial intelligence technology, pre-trained large models have demonstrated excellent performance in multiple fields. However, in real-world enterprise applications, model deployment scenarios are highly heterogeneous and domain-diverse. Different enterprises often possess locally distributed, isolated data with significant domain characteristics. The privacy and regulatory constraints of this type of data (such as in finance, healthcare, and government) prevent unified training through traditional centralized learning, thus limiting the ability of general-purpose large models to be widely adapted and shared across enterprises. Federated learning offers a solution by coordinating multiple clients to jointly train the model without sharing data. However, mainstream federated learning methods largely rely on average aggregation mechanisms, which fail to effectively align the domain characteristics of different enterprises when data distributions differ significantly (i.e., in non-IID scenarios), leading to a significant decline in model performance in cross-domain inference tasks. Summary of the Invention

[0003] In view of the above technical problems, the present invention provides a domain-adaptive large model fine-tuning method and system based on enterprise private data, so as to solve the problem that the existing technology cannot effectively align the domain features of different enterprises, resulting in a significant decline in the performance of the model in cross-domain reasoning tasks.

[0004] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0005] According to one aspect of the present invention, a domain-adaptive large model fine-tuning method based on enterprise private data is proposed, the method comprising:

[0006] Each participating client uses a unified pre-trained model to extract feature representations from its local training data, and selects a portion of vectors from the feature representations of each category as prototypes of the corresponding category using one of mean sampling, cluster sampling, or random sampling at a predetermined sampling rate, to obtain a prototype set containing domain-specific prototypes of each category. A differential privacy mechanism is applied to the prototype set to add random noise to protect data privacy, and then the prototype set is sent to the server.

[0007] The server aggregates the prototype sets from all clients and constructs a global prototype training dataset. Without averaging the same type of prototypes across different clients, the server uses a supervised learning classification loss function to globally train the adapter module of the pre-trained model based on the prototype training dataset. This optimizes the global adapter model to adapt to the multi-domain data distribution of all clients. Once the training converges, the global adapter model is sent back to each client, completing the global adapter initialization for a single communication round.

[0008] Each client uses the received global adapter model as the initial model, selects a small number of labeled data samples locally to form a few-sample dataset, and performs local fine-tuning training on the global adapter model to improve the model performance in the local domain. By using the global adapter model as the teacher model, knowledge distillation constraints are applied to the local adapter model. The task loss of local fine-tuning and the knowledge distillation loss from the global adapter model are combined with preset weights to optimize together, preventing the local model from forgetting global knowledge, thereby achieving a balance between global knowledge retention and local domain adaptation.

[0009] Furthermore, the prototype set is generated using a mean sampling method. For each class of each client, the mean vector is calculated locally using the feature representation of the corresponding class data samples extracted by the pre-trained model, which serves as the domain-specific prototype of the corresponding class.

[0010] Furthermore, the prototype set is generated using a clustering sampling method. For each type of data sample from each client, a clustering algorithm is executed, and multiple centroid vectors obtained from the clustering are used as prototypes of the corresponding classes. The number of cluster centroids selected is determined by the predetermined sampling rate.

[0011] Furthermore, the prototype set is generated using a random sampling method, which randomly selects a portion of vectors from the feature representations of each type of data sample from each client as the prototype of the corresponding class according to a predetermined sampling rate.

[0012] Furthermore, when applying differential privacy mechanisms to the aforementioned prototype set to protect data privacy, the following are included:

[0013] Before the prototype set is uploaded to the client, random noise following a Gaussian distribution is added to each prototype vector. The amplitude of the noise is controlled by a preset perturbation coefficient and noise scale parameter to reduce the probability of inferring and reconstructing the original data sample from the prototype set.

[0014] Furthermore, during the training of the global adapter model, the server constructs a prototype dataset for training from the prototype sets uploaded by each client, and trains on the prototype dataset based on cross-entropy classification loss by fixing the parameters of the pre-trained model and optimizing only the parameters of the additional adapter modules. During training, prototype vectors from different clients but belonging to the same category are not averaged and aggregated, but are kept as independent samples in the training dataset to preserve the domain feature differences of each client. When the training loss decreases and converges to a predetermined threshold, the training of the global adapter model is completed, and the global adapter model is distributed to all clients in a single communication to replace their local adapter models.

[0015] Furthermore, during the local fine-tuning process with a small number of samples, each client selects a predetermined number of labeled samples for each category from its local data to form a few-sample training set. The local adapter model is initialized using the parameters of the global adapter model and fine-tuned on the few-sample training set. During the fine-tuning process, the global adapter model is used as the teacher model, and knowledge distillation constraints are applied to the local adapter model. That is, for the samples in the few-sample training set, the output of the local adapter model is compared with the output of the global adapter model to calculate the knowledge distillation loss. The knowledge distillation loss is then weighted and added to the local task loss according to preset weights to obtain the total loss, which is used to update the parameters of the local adapter model. This improves the accuracy of the local model in its local domain while preventing the local adapter model from deviating excessively from the global adapter model.

[0016] Furthermore, the training process of the global adapter model is convergent. When the loss function is a smooth non-convex function and the gradient satisfies the L-Lipschitz continuity condition, and the learning rate is chosen to make the step size of parameter update at each step bounded, the loss value of global prototype fine-tuning training will decrease monotonically with the iteration process. Moreover, as the number of iterations increases, the average norm of the model gradient gradually decreases and approaches zero, indicating that the training process converges to a stationary point.

[0017] According to a second aspect of this disclosure, a domain-adaptive large model fine-tuning system based on enterprise private data is provided, the system comprising:

[0018] The prototype generation and privacy protection module is used to extract feature representations of local training data from each participating client using a unified pre-trained model, and select a portion of vectors from the feature representations of each category as prototypes of the corresponding category using one of mean sampling, cluster sampling or random sampling at a predetermined sampling rate, to obtain a prototype set containing prototypes specific to each category domain. The module then applies a differential privacy mechanism to the prototype set to add random noise to protect data privacy, and finally sends the prototype set to the server.

[0019] The global adapter joint training module is used to aggregate the prototype set from all clients on the server, construct a global prototype training dataset, and perform global training on the adapter module of the pre-trained model based on the prototype training dataset without averaging the same type of prototypes from different clients, using a classification loss function of supervised learning, to optimize and obtain a global adapter model that adapts to the multi-domain data distribution of all clients. When the training converges, the global adapter model is sent back to each client to complete the global adapter initialization for a single communication round.

[0020] The local fine-tuning and adaptive module is used to train the global adapter model received by each client as the initial model, select a small number of labeled data samples locally to form a few-sample dataset, and fine-tune the global adapter model locally to improve the model performance in the local domain. By using the global adapter model as the teacher model, knowledge distillation constraints are applied to the local adapter model. The task loss of local fine-tuning and the knowledge distillation loss from the global adapter model are combined with preset weights to optimize together, preventing the local model from forgetting global knowledge, thereby achieving a balance between global knowledge preservation and local domain adaptation.

[0021] The technical solution of the present invention has the following beneficial effects:

[0022] By introducing "multi-domain prototypes" as an abstract representation of the original data, cross-enterprise domain knowledge mapping relationships are constructed through prototypes. On the client side, a unified pre-trained model is used to extract category feature representations from local private data, and class prototypes with domain characteristics are generated by means, clustering or random methods to replace the original samples and upload them to the server. This preserves key semantic distribution information while ensuring data privacy.

[0023] The server performs a one-time global adapter training based on a collection of prototypes from multiple enterprises, achieving cross-domain model alignment without multiple rounds of communication, significantly reducing training cycle and bandwidth overhead. Furthermore, to prevent prototype information leakage, this invention embeds a differential privacy mechanism before prototype uploading, adding Gaussian noise to ensure that client data is difficult to recover, thus improving the overall system's security and usability.

[0024] After completing the unified training of the global adapter, this invention allows each enterprise to further fine-tune the model locally using its own limited data to optimize the model's intra-domain performance. By introducing a knowledge distillation mechanism, the local fine-tuning process uses the global adapter as the teacher model, collaboratively optimizing local performance and global generalization ability, effectively preventing catastrophic forgetting phenomena. This is suitable for large-scale adaptive model deployment scenarios involving federated collaboration among multiple data-sensitive organizations. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a domain-adaptive large model fine-tuning method based on enterprise private data, as described in the embodiments of this specification.

[0026] Figure 2 This is a block diagram of a domain-adaptive large model fine-tuning system based on enterprise private data, as described in the embodiments of this specification. Detailed Implementation

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0028] Furthermore, the accompanying drawings are merely illustrative of this disclosure. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0029] This invention provides a method for fine-tuning a domain-adaptive large model of a product based on enterprise private data. (Refer to...) Figure 1 The diagram shown is a flowchart illustrating a domain-adaptive large model fine-tuning method based on enterprise private data according to an embodiment of the present invention. This method can be applied to electronic devices such as personal computers, servers, and controllers. The method can be executed by a device, which can be implemented by software and / or hardware. Specifically, the method may include the following steps S101-S106:

[0030] In step S101, each participating client uses a unified pre-trained model to extract feature representations of local training data, and selects a portion of vectors from the feature representations of each category as prototypes of the corresponding category using one of mean sampling, cluster sampling, or random sampling at a predetermined sampling rate, to obtain a prototype set containing domain-specific prototypes of each category. A differential privacy mechanism is applied to the prototype set to add random noise to protect data privacy, and then the prototype set is sent to the server.

[0031] Each client, based on the same pre-trained model, first performs a unified feature extraction operation on its local private data to obtain the feature representation of each data sample in a unified feature space (i.e., the feature embedding of the pre-trained model). Although all clients share the same category label space, due to significant differences in the data domains of each client, the feature embeddings of data of the same category in different clients still exhibit domain-related differences. To effectively express and preserve these domain-specific differences, one of the following methods—mean sampling, cluster sampling, or random sampling—is used to obtain the prototype set.

[0032] Specifically, each client is based on a unified pre-trained model, such as the CLIP model, which includes an image encoder. Each client uses an image encoder. For local private datasets Feature representation extraction is performed to obtain the feature vector for each data sample, represented as:

[0033] ;

[0034] here, This is the original data sample. This represents the category label to which the sample belongs. Feature vector. The dimension depends on the pre-trained model used (e.g., the feature dimension generated by the ViT or ConvNeXT encoder in CLIP). After extracting features, in order to reduce communication costs and effectively represent local data features, the client selects a portion of the feature representations for each category as a representative prototype set for that category in the local domain, according to a predetermined sampling rate. The prototype set is obtained by sampling using one or more of mean sampling, cluster sampling, or random sampling.

[0035] As a supplement, when applying differential privacy mechanisms to the prototype set to protect data privacy, the method includes: before the client uploads the prototype set, adding random noise that follows a Gaussian distribution to each prototype vector, the amplitude of which is controlled by a preset perturbation coefficient and noise scale parameter, in order to reduce the probability of inferring and reconstructing the original data sample from the prototype set.

[0036] To protect data privacy, the client adds random noise to the prototype vectors of each category before uploading the prototype set to achieve differential privacy. Specifically, the client adds noise to each prototype vector that follows a Gaussian distribution. Let the random noise be the prototype vector of client i. Then the prototype after adding random noise is:

[0037] ;

[0038] Among them, the perturbation coefficient q and noise scaling parameter s of the differential privacy mechanism are pre-set hyperparameters used to control the trade-off between privacy protection strength and model performance.

[0039] The specific parameters are selected according to the following criteria: the perturbation coefficient q determines the relative amplitude of the random noise; the noise scale parameter s determines the absolute variance of the added noise, and the larger the value, the stronger the privacy protection.

[0040] The core objective of differential privacy mechanisms is to significantly reduce the probability of attackers retrieving the original data from the uploaded prototype set by increasing randomness, thereby protecting data privacy.

[0041] After completing the above process, each client will have a prototype set protected by differential privacy. The data is sent to the server. The client does not upload the original data, but only the prototype after differential privacy processing, which significantly reduces communication costs and the risk of data leakage, and protects the privacy of the enterprise's private data.

[0042] In step S102, the server aggregates the prototype sets from all clients to construct a global prototype training dataset. Without averaging the same type of prototypes across different clients, the server uses a supervised learning classification loss function to globally train the adapter module of the pre-trained model based on the prototype training dataset. This optimizes the global adapter model to adapt to the multi-domain data distribution of all clients. Once the training converges, the global adapter model is sent back to each client, completing the global adapter initialization for a single communication round.

[0043] During the training of the global adapter model, the server constructs a prototype dataset for training from the prototype sets uploaded by each client. By fixing the parameters of the pre-trained model and optimizing only the parameters of the additional adapter modules, the server trains on the prototype dataset based on cross-entropy classification loss. During training, prototype vectors from different clients but belonging to the same category are not averaged and aggregated; instead, they are kept as independent samples in the training dataset to preserve the domain feature differences of each client. When the training loss decreases and converges to a predetermined threshold, the training of the global adapter model is completed, and the global adapter model is distributed to all clients in a single communication to replace their local adapter models.

[0044] The server aggregates the prototype sets from all clients to construct a global prototype training dataset. Without averaging the same type of prototypes across different clients, the server uses a supervised learning classification loss function to globally train the adapter module of the pre-trained model based on the prototype training dataset. This optimizes the model to a global adapter model that adapts to the multi-domain data distribution of all clients. Once the training converges, the global adapter model is sent back to each client, completing the global adapter initialization for a single communication round.

[0045] In practice, the first step is the construction of the global prototype training dataset, where the server receives a set of prototypes uploaded from all clients that have undergone differential privacy protection.

[0046] ;

[0047] Where N is the total number of clients, and K is the number of categories in the classification task. This represents the set of prototype vectors of the k-th category uploaded by client i.

[0048] The server aggregates the prototype collections from all clients to form a global prototype training dataset. The server does not average or merge prototypes of the same category from different clients; instead, it adds all prototype vectors independently to the training dataset as is.

[0049] ;

[0050] This non-uniform approach ensures that the data used for server-side training retains the domain feature differences from different clients, enabling the trained adapter to better generalize to multiple different data domains.

[0051] Next is the global adapter model training phase. The server uses the globally prototype training dataset DPDP constructed above to perform supervised learning training on the adapter module of the pre-trained model. The server keeps the parameters of the backbone network (image encoder) of the pre-trained model (e.g., the CLIP model) completely fixed, and only trains the parameters of the additional adapter module. This ensures that the feature extraction capability of the pre-trained model is not compromised, while significantly reducing the computational cost and overfitting risk of global training. During training, the cross-entropy classification loss function is used for supervised learning, and the specific formula is as follows:

[0052] ;

[0053] in, This represents the fixed parameters of the pre-trained model (not used in training, but only for forward propagation); This represents the parameters of the adapter module to be optimized; p is the prototype vector, and y is its true class label; This represents the class probability distribution predicted by the model. During each forward propagation, the input prototype vector p undergoes feature extraction from a fixed pre-trained model, and is then input into the adapter module to predict the class probability distribution. Then, the loss is calculated based on the cross-entropy loss function, and the adapter parameters are updated accordingly. .

[0054] Next is the single-round communication training convergence mechanism. Unlike the multi-round communication in traditional federated learning, this embodiment only requires a single communication round to complete global training. The server continuously trains the adapter module parameters on the constructed global prototype dataset until the cross-entropy loss function converges to a predetermined threshold or a stopping condition. The convergence condition is defined as the model being considered fully converged when the loss function changes below the predetermined threshold over multiple consecutive training epochs. Training stops upon convergence, and the server obtains the optimized global adapter model. When the loss function L is a smooth, non-convex function and the gradient satisfies the L-Lipschitz continuity condition, an appropriate learning rate is selected. The parameter update steps during training satisfy the following inequality to ensure that the model loss decreases monotonically:

[0055] ;

[0056] in, , is a positive constant; This represents the maximum difference between client prototypes. As the number of training epochs increases, the average gradient norm gradually approaches zero, ensuring that the training process converges to a stationary point.

[0057] Finally, after the server completes training convergence, it obtains the optimized global adapter model and sends it back to each client in a single communication to replace its local adapter model. Upon receiving the model, the client uses it as initial parameters for subsequent local task inference and fine-tuning, effectively achieving domain knowledge sharing and unified model initialization across clients, thereby improving generalization performance and global knowledge sharing capabilities among clients.

[0058] In step S103, each client uses the received global adapter model as the initial model, selects a small number of labeled data samples locally to form a few-sample dataset, performs local fine-tuning training on the global adapter model to improve the model performance in the local domain, and applies knowledge distillation constraints to the local adapter model by using the global adapter model as the teacher model. The task loss of local fine-tuning and the knowledge distillation loss from the global adapter model are combined with preset weights to optimize together, preventing the local model from forgetting global knowledge, thereby achieving a balance between global knowledge retention and local domain adaptation.

[0059] As a supplement, during the local fine-tuning process with a small number of samples, each client selects a predetermined number of labeled samples for each category from its local data to form a few-sample training set. The local adapter model is initialized using the parameters of the global adapter model and fine-tuned on the few-sample training set. During the fine-tuning process, the global adapter model is used as the teacher model, and knowledge distillation constraints are applied to the local adapter model. That is, for the samples in the few-sample training set, the output of the local adapter model is compared with the output of the global adapter model to calculate the knowledge distillation loss. The knowledge distillation loss is then weighted and added to the local task loss according to preset weights to obtain the total loss, which is used to update the parameters of the local adapter model. This improves the accuracy of the local model in its local domain while preventing the local adapter model from deviating excessively from the global adapter model.

[0060] In this process, each client uses the received global adapter model as the initial model, selects a small number of labeled data samples locally to form a few-sample dataset, and performs local fine-tuning training on the global adapter model to improve the model performance in the local domain. By using the global adapter model as the teacher model, knowledge distillation constraints are applied to the local adapter model. The task loss of local fine-tuning and the knowledge distillation loss from the global adapter model are combined with preset weights to optimize together, preventing the local model from forgetting global knowledge, thereby achieving a balance between global knowledge retention and local domain adaptation.

[0061] Specifically, during the local few-shot dataset construction phase, after receiving the global adapter model AG from the server, each client selects a small number of labeled data samples from its local private dataset to form a local few-shot training dataset. The specific construction process is as follows:

[0062] Assuming the classification task has K classes, each client selects a predetermined number of samples (e.g., 1-shot, 4-shot, or 8-shot) for each class from its local data to ensure that each class has a small number of samples for local training.

[0063] ;

[0064] Where: F is the number of samples selected for each category; This refers to all local data of client i. Let m be the selected m-th sample and its corresponding category label.

[0065] Next is the local adapter fine-tuning and initialization phase. The client uses the global adapter model AG received from the server to initialize the parameters of the local adapter model. The client trains the model on a local few-sample training set. Fine-tuning training is performed on the model to further improve its performance in the local domain, while the knowledge distillation mechanism effectively prevents catastrophic forgetting of the model during the fine-tuning process.

[0066] With the execution of the knowledge distillation constraint mechanism, during local fine-tuning training, each client employs the knowledge distillation mechanism, using the global adapter model as the teacher model and the local adapter model as the student model, to apply distillation constraints to the student model. Specifically, the client inputs each sample from its local few-sample training set into both the teacher and student models simultaneously, obtaining the corresponding output class probability distributions. The teacher model's output probability distribution is as follows: The student model output probability distribution is Knowledge distillation loss function Defined as the Kullback-Leibler divergence (KL divergence) between the output probabilities of the teacher model and the student model:

[0067] ;

[0068] Where c represents the category index and K represents the total number of categories.

[0069] As local task loss calculation and total loss optimization are performed, the client also calculates the task loss function LCELCE (cross-entropy loss) on a small number of sample data:

[0070] ;

[0071] It is a local adapter model The predicted class probability for the input sample.

[0072] The client combines the knowledge distillation loss and the task loss, using a preset weighting coefficient β to form the total loss function:

[0073] ;

[0074] The parameter β is used to adjust the degree of influence of knowledge distillation loss on the overall optimization. If β is large, the student model will tend to follow the teacher model more closely and retain more global knowledge; if β is small, the student model will focus more on the task-specific loss and pursue higher local adaptation accuracy.

[0075] Finally, during the local adapter parameter update phase, the client uses the calculated total loss function. The parameters of the local adapter model are updated using gradient descent.

[0076] ;

[0077] Here, α represents the learning rate for local fine-tuning. The ultimate goal of this optimization process is to improve performance within the client domain while preventing the local model from deviating excessively from the global generalization knowledge provided by the teacher model, thereby achieving: a balance between local domain model adaptation and global domain generalization ability; mitigating the risk of overfitting when local training data is limited; and avoiding catastrophic forgetting.

[0078] In one embodiment, the prototype set is generated using a mean sampling method. For each class of each client, the mean vector is calculated locally using the feature representation of the corresponding class data samples extracted by the pre-trained model, and serves as the domain-specific prototype of the corresponding class.

[0079] Alternatively, the prototype set is generated using a clustering sampling method. For each data sample feature representation of each client, a clustering algorithm is executed, and multiple centroid vectors obtained from the clustering are used as prototypes of the corresponding classes. The number of cluster centroids selected is determined by the predetermined sampling rate.

[0080] Furthermore, if the prototype set is generated using a random sampling method, a subset of vectors are randomly selected from the feature representations of each type of data sample from each client according to a predetermined sampling rate as the prototype of the corresponding class.

[0081] Specifically, in mean sampling, the client takes the arithmetic mean of the data feature representations for each category to obtain a single prototype representation for each category, as shown in the following formula:

[0082] ;

[0083] in Let i represent the set of data samples of the kth class for client i. This represents the number of samples.

[0084] In cluster sampling, the client performs clustering (such as the K-means clustering algorithm) on all data feature representations for each category. The number of cluster centers for each category is determined by the sampling rate r (e.g., Clustering is used to obtain multiple centroids of a category as the prototype of that category:

[0085] ;

[0086] in Represents the clustering process. This represents the number of cluster centers for category k.

[0087] In random sampling, the client randomly selects a subset of vectors from the feature representations of each category of data according to the sampling rate r, as the prototype for that category:

[0088] ;

[0089] The number of random selections ;

[0090] After the client completes the above steps for all categories, it will obtain a collection of prototypes:

[0091] .

[0092] Based on the same line of thought, such as Figure 2 The diagram shown is a structural block diagram of a domain-adaptive large model fine-tuning system based on enterprise private data, according to an embodiment of the present invention. The system includes:

[0093] The prototype generation and privacy protection module 201 is used to extract feature representations of local training data from each participating client using a unified pre-trained model, and select a portion of vectors from the feature representations of each category as prototypes of the corresponding category using one of mean sampling, cluster sampling or random sampling at a predetermined sampling rate, to obtain a prototype set containing prototypes specific to each category domain. The module then applies a differential privacy mechanism to the prototype set to add random noise to protect data privacy, and finally sends the prototype set to the server.

[0094] The global adapter joint training module 202 is used to aggregate the prototype set from all clients on the server, construct a global prototype training dataset, and perform global training on the adapter module of the pre-trained model based on the prototype training dataset without averaging the same type of prototypes from different clients, using a classification loss function of supervised learning, to optimize and obtain a global adapter model that adapts to the multi-domain data distribution of all clients. When the training converges, the global adapter model is sent back to each client to complete the global adapter initialization for a single communication round.

[0095] The local fine-tuning and adaptive module 203 is used to take the global adapter model received by each client as the initial model, select a small number of labeled data samples locally to form a few-sample dataset, and perform local fine-tuning training on the global adapter model to improve the model performance in the local domain. By using the global adapter model as the teacher model, knowledge distillation constraints are applied to the local adapter model. The task loss of local fine-tuning and the knowledge distillation loss from the global adapter model are combined with preset weights to optimize together, preventing the local model from forgetting global knowledge, thereby achieving a balance between global knowledge retention and local domain adaptation.

[0096] The specific details of the above system have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0097] This system introduces "multi-domain prototypes" as an abstract representation of the original data. Through prototypes, cross-enterprise domain knowledge mapping relationships are constructed. On the client side, a unified pre-trained model is used to extract category feature representations from local private data. Class prototypes with domain characteristics are generated by means, clustering, or random methods to replace the original samples and upload them to the server. This preserves key semantic distribution information while ensuring data privacy.

[0098] The server performs a one-time global adapter training based on a collection of prototypes from multiple enterprises, achieving cross-domain model alignment without multiple rounds of communication, significantly reducing training cycle and bandwidth overhead. Furthermore, to prevent prototype information leakage, this invention embeds a differential privacy mechanism before prototype uploading, adding Gaussian noise to ensure that client data is difficult to recover, thus improving the overall system's security and usability.

[0099] This invention also proposes a two-stage optimization strategy of "global sharing - local fine-tuning". After completing the unified training of the global adapter, each enterprise can further fine-tune the model locally on its own limited data to optimize the model's intra-domain performance. By introducing a knowledge distillation mechanism, the local fine-tuning process uses the global adapter as the teacher model to collaboratively optimize local performance and global generalization ability, effectively preventing catastrophic forgetting phenomena. This is suitable for large-scale adaptive model deployment scenarios involving federated collaboration among multiple data-sensitive organizations.

[0100] The accompanying drawings are merely illustrative of the processes included in the methods according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the drawings do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0101] It should be noted that although several modules or units of the system have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0102] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0103] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for fine-tuning a domain-adaptive large model based on enterprise private data, characterized in that, The method includes: Each participating client uses a unified pre-trained model to extract feature representations from its local training data, and selects a portion of vectors from the feature representations of each category as prototypes of the corresponding category using one of mean sampling, cluster sampling, or random sampling at a predetermined sampling rate, to obtain a prototype set containing domain-specific prototypes of each category. A differential privacy mechanism is applied to the prototype set to add random noise to protect data privacy, and then the prototype set is sent to the server. The server aggregates the prototype sets from all clients and constructs a global prototype training dataset. Without averaging the same type of prototypes across different clients, the server uses a supervised learning classification loss function to globally train the adapter module of the pre-trained model based on the prototype training dataset. This optimizes the global adapter model to adapt to the multi-domain data distribution of all clients. Once the training converges, the global adapter model is sent back to each client, completing the global adapter initialization for a single communication round. Each client uses the received global adapter model as its initial model, selects a small number of labeled data samples locally to form a few-sample dataset, and performs local fine-tuning training on the global adapter model to improve its local performance. The global adapter model is used as a teacher model to apply knowledge distillation constraints to the local adapter model. The task loss of local fine-tuning and the knowledge distillation loss from the global adapter model are combined with preset weights for joint optimization to prevent the local model from forgetting global knowledge, thus achieving a balance between global knowledge retention and local adaptation. During the local fine-tuning process with a small number of samples, each client selects a predetermined number of samples from its local data for each category. Labeled samples constitute a few-shot training set. The local adapter model is initialized using the parameters of the global adapter model and fine-tuned on the few-shot training set. During the fine-tuning process, the global adapter model is used as the teacher model, and knowledge distillation constraints are applied to the local adapter model. That is, for the samples in the few-shot training set, the output of the local adapter model is compared with the output of the global adapter model to calculate the knowledge distillation loss. The knowledge distillation loss is then weighted and added to the local task loss according to preset weights to obtain the total loss, which is used to update the parameters of the local adapter model. This improves the local model's accuracy in its local domain while preventing the local adapter model from deviating excessively from the global adapter model.

2. The domain-adaptive large model fine-tuning method based on enterprise private data according to claim 1, characterized in that, The prototype set is generated using a mean sampling method. For each class of each client, the mean vector is calculated locally using the feature representation of the corresponding class data samples extracted by the pre-trained model, and serves as the domain-specific prototype of the corresponding class.

3. The domain-adaptive large model fine-tuning method based on enterprise private data according to claim 1, characterized in that, The prototype set is generated using a clustering sampling method. For each type of data sample from each client, a clustering algorithm is executed, and multiple centroid vectors obtained from the clustering are used as prototypes of the corresponding classes. The number of cluster centroids selected is determined by the predetermined sampling rate.

4. The domain-adaptive large model fine-tuning method based on enterprise private data according to claim 1, characterized in that, The prototype set is generated using a random sampling method, which randomly selects a portion of vectors from the feature representations of each type of data sample from each client as the prototype of the corresponding class according to a predetermined sampling rate.

5. The domain-adaptive large model fine-tuning method based on enterprise private data according to claim 1, characterized in that, When applying differential privacy mechanisms to the aforementioned prototype set to protect data privacy, the following are included: Before the prototype set is uploaded to the client, random noise following a Gaussian distribution is added to each prototype vector. The amplitude of the noise is controlled by a preset perturbation coefficient and noise scale parameter to reduce the probability of inferring and reconstructing the original data sample from the prototype set.

6. The domain-adaptive large model fine-tuning method based on enterprise private data according to claim 1, characterized in that, During the training process of the global adapter model, the server constructs a prototype dataset for training from the prototype sets uploaded by each client, and trains the model on the prototype dataset based on cross-entropy classification loss by fixing the parameters of the pre-trained model and optimizing only the parameters of the additional adapter module. During training, prototype vectors from different clients but belonging to the same category are not averaged and aggregated. Instead, they are kept as independent samples in the training dataset to preserve the domain feature differences of each client. When the training loss decreases and converges to a predetermined threshold, the training of the global adapter model is completed, and the global adapter model is distributed to all clients in a single communication to replace their local adapter models.

7. The domain-adaptive large model fine-tuning method based on enterprise private data according to claim 1, characterized in that, The training process of the global adapter model is convergent. When the loss function is a smooth non-convex function and the gradient satisfies the L-Lipschitz continuity condition, and the learning rate is chosen to make the step size of parameter update at each step bounded, the loss value of global prototype fine-tuning training will decrease monotonically with the iteration process. Furthermore, as the number of iterations increases, the average norm of the model gradient gradually decreases and approaches zero, indicating that the training process converges to a stationary point.

8. A domain-adaptive large model fine-tuning system based on enterprise private data, the system comprising: The prototype generation and privacy protection module is used to extract feature representations of local training data from each participating client using a unified pre-trained model, and select a portion of vectors from the feature representations of each category as prototypes of the corresponding category using one of mean sampling, cluster sampling or random sampling at a predetermined sampling rate, to obtain a prototype set containing prototypes specific to each category domain. The module then applies a differential privacy mechanism to the prototype set to add random noise to protect data privacy, and finally sends the prototype set to the server. The global adapter joint training module is used to aggregate the prototype set from all clients on the server, construct a global prototype training dataset, and perform global training on the adapter module of the pre-trained model based on the prototype training dataset without averaging the same type of prototypes from different clients, using a classification loss function of supervised learning, to optimize and obtain a global adapter model that adapts to the multi-domain data distribution of all clients. When the training converges, the global adapter model is sent back to each client to complete the global adapter initialization for a single communication round. The local fine-tuning and adaptive module is used to train the global adapter model received by each client as an initial model. It selects a small number of labeled data samples locally to form a few-sample dataset, and performs local fine-tuning training on the global adapter model to improve its local performance. Furthermore, it applies knowledge distillation constraints to the local adapter model by using the global adapter model as a teacher model. The task loss of local fine-tuning is combined with the knowledge distillation loss from the global adapter model according to preset weights for joint optimization, preventing the local model from forgetting global knowledge and thus achieving a balance between global knowledge preservation and local adaptation. During the local fine-tuning process with a small number of samples, each client selects data from its local data for each category. A predetermined number of labeled samples are taken to form a few-shot training set. The local adapter model is initialized using the parameters of the global adapter model and fine-tuned on the few-shot training set. During the fine-tuning process, the global adapter model is used as the teacher model, and knowledge distillation constraints are applied to the local adapter model. That is, for the samples in the few-shot training set, the output of the local adapter model is compared with the output of the global adapter model to calculate the knowledge distillation loss. The knowledge distillation loss is then weighted and added to the local task loss according to preset weights to obtain the total loss, which is used to update the parameters of the local adapter model. This improves the local model's accuracy in its local domain while preventing the local adapter model from deviating excessively from the global adapter model.

Images