Context learning method for implicit federal large model

Abstract

The invention discloses an implicit federal large model context learning method. The method comprises the following steps: 1) all clients and servers extract context vectors through a large model; 2) aggregating global context vectors by the server; 3) the server calibrates the injection coefficient; and 4) the server distributes the global context vector and the corrected injection coefficient, and the client receives and injects the local large model. The method breaks through federated learning communication bottleneck through an implicit federated large model context learning party, balances privacy protection and knowledge sharing, realizes balance among communication efficiency, privacy security and task performance, and provides an efficient solution for edge device cooperative training and large model customization in the vertical field.

Description

Technical Field

[0001] The technical field of the present invention covers federated learning, contextual learning, large model optimization, and privacy computing, and specifically is an implicit federated large model contextual learning method. Background Art

[0002] In recent years, the scale of large language models (LLMs) has continued to expand, with models such as LLaMA3.1-405B and GPT3, each with hundreds of billions of parameters, emerging. With the exponential growth of model size, LLMs have demonstrated advanced cognitive capabilities not possessed by smaller models, such as multi-step logical reasoning, contextual understanding, and adaptation to few-shot tasks. However, this expansion in model size has also brought with it technical bottlenecks and challenges inherent in the laws of nature:

[0003] In federated learning (FL) scenarios, parameter transmission and training for LLM face fundamental hardware resource constraints. For example, a single parameter transmission for LLaMA3.1-405B over a 100 Mbps network takes 36 hours, far exceeding the typical FL communication cycle (typically minutes). Furthermore, LLM training demands superlinearly on GPU memory and computing power. Even with a distributed training strategy, hardware costs and energy consumption remain difficult to adapt to practical deployment requirements. Federated learning clients are often distributed across different regions or industries, resulting in highly heterogeneous (non-IID) data distribution. For example, one client may only have a small amount of labeled data for a specific domain (such as healthcare), while other clients may have data concentrated in finance or education. This fragmented nature makes it difficult to effectively integrate multi-source knowledge when aggregating the global model, exacerbating the "data silo" problem and significantly reducing the model's generalization ability on specific tasks.

[0004] To alleviate these issues, in-context learning (ICL) has gained attention as a fine-tuning-free approach. ICL guides the model by inputting task examples (i.e., contextual cues) to generate outputs that meet task requirements, avoiding the computational overhead of parameter updates. However, directly applying ICL in federated scenarios, while enabling zero-shot reasoning through task example cues, requires collecting example data from the client to construct contextual cues, which poses the risk of user privacy leakage (e.g., example data may contain sensitive information). Furthermore, ICL relies on the existing cognitive capabilities of the LLM and does not enhance high-level reasoning capabilities for specific tasks, resulting in insufficient task adaptability (e.g., accuracy is only 50% in zero-shot scenarios). Summary of the Invention

[0005] This paper aims to address the communication, privacy, and computational challenges of traditional large models in distributed environments by providing an implicit federated large model context learning method. This method encodes client-local data into low-dimensional context vectors through implicit context learning, effectively reducing data transmission requirements. Compared with federated large model fine-tuning methods, it significantly reduces communication overhead while effectively improving the LLM's ability to understand specific tasks in federated scenarios while protecting privacy.

[0006] The technical solution for achieving the purpose of the present invention is:

[0007] An implicit federated large model context learning method, the method involving the following entities:

[0008] Client: Deployed by the data owner, it has local data storage capabilities and is used to extract task-related context information;

[0009] Server: Deployed in a central node or headquarters data center, it has stronger computing resources and can perform cross-client context vector aggregation and context injection coefficient optimization.

[0010] Large Language Model (LLM): refers to a pre-trained language model deployed on both the client and server. It is used to generate example prompts, participate in inference, and extract activations. The client model does not need to update parameters and only performs vector extraction and transmission operations.

[0011] Context sample template;

[0012] The method comprises the following steps:

[0013] The federated learning system of the method is composed of multiple edge computing nodes, i.e., clients, and a central coordination server, i.e., server;

[0014] All clients and servers extract local context vector structures through the big model:

[0015] 1) All clients and servers first design a context sample template based on the current task type. All clients and servers construct a context sample prompt for each currently stored local data according to the context sample template. The sample prompt includes an input content and its target label. All clients and servers input their respective context sample prompts one by one into the local pre-trained large language model and extract the intermediate activation vector of the token position used for prediction in each layer of the model. The activation vector includes the output vector of the Attention layer and the output vector of the MLP layer. Let the output of the i-th sample prompt at the l-th layer be , , there are N sample prompts in total, forming the sample activation set:

[0016]

[0017] The activation vector sample dimension of each layer is averaged to obtain the fused local context vector:

[0018] ,

[0019] The context vector structure that constitutes the cross-layer organization is expressed as:

[0020]

[0021] Where L is the total number of model layers. Both the client and the server perform the above context extraction process and generate their own context vector structure to support the aggregation and injection of the federated context structure into the large model to achieve enhanced effects.

[0022] 2) Server aggregated global context vector structure:

[0023] All clients upload their local context vector structures to the central server. At the same time, the server extracts the local context vector at each layer according to the process of extracting the context vector structure in step 1), which is recorded as:

[0024]

[0025] For each client k (k=1,2,...,K), its local context vector at layer l is:

[0026]

[0027] The server performs average aggregation on the context vectors of all clients and itself at layer l to obtain the global context vector of layer l:

[0028]

[0029] The aggregation calculation of this layer requires indexing each layer of the model Repeat the process in sequence, and finally the server obtains the global context vector structure organized across layers:

[0030] ;

[0031] 3) The server injects the global context vector into the original large model and calibrates the injection coefficients using the server’s local calibration dataset;

[0032] 4) The server distributes the global context vector structure and the corrected injection coefficients to the client, which receives and injects them into the local large model:

[0033] The server aggregates the global context vector structure V global and its corrected injection coefficient The global context vector is distributed to all clients together. The client injects the global context vector into the residual stream of the local frozen large model according to the coefficients to generate a task-enhanced large model, and uses the task-enhanced large model for inference tasks.

[0034] The specific process of step 3) is as follows:

[0035] The server will first use its own retained calibration sample set to calibrate each layer in the global context vector structure. The vector, i.e., the output vector of the Attention and MLP sub-layers, is injected into the residual stream of the original large model. The learning and calibration process of the linear injection coefficient during the injection process is as follows:

[0036] The server first aggregates the global context vector structure:

[0037] The server injects the data into the original large model that is identical to the client and keeps its parameters frozen. The structure of the original large model remains unchanged, and only the context vector is inserted during inference. Next, the server uses its locally retained calibration sample set to perform the following calibration process:

[0038] The server randomly selects a batch of calibration samples from the locally retained labeled dataset, denoted as:

[0039]

[0040] in For sample output, For the corresponding labels, the calibration samples are distributed identically with the client samples;

[0041] For a given calibration sample , the server passes the input x to the large model for layer-by-layer forward propagation. Assuming that at layer l and position t, the large model first obtains the output of the previous layer of the residual stream during standard forward calculation: At the same time, the large model will calculate the original Attention activation of the sample at the corresponding position in the lth layer and MLP activation ;

[0042] Server from Take out the global context vector of layer l , and inserted into the residual stream in the linear combination. For each calibration sample, the residual update process of the t-th position of the l-th layer is as follows:

[0043]

[0044] in:

[0045] and They are the lth layer Attention and MLP global context vector obtained by aggregation in step 2);

[0046] and are the original Attention and MLP activation of the sample at layer l and position t;

[0047] is the residual output of the previous layer, It is the lth layer Attention mixed injection, It is the MLP mixed injection of the lth layer;

[0048] It is a hierarchical learnable scalar coefficient used to adjust the mixing ratio of the context vector and the original query activation to obtain the final corrected injection coefficient;

[0049] The entire update does not change the model's weight matrix, but only performs a linear combination operation on the residual flow;

[0050] The server inserts the l=1,2,...,L layers into the global context vector according to the above residual update process and updates the residual stream until the model outputs the final predicted distribution;

[0051] After all layers are injected, the model outputs the final predicted probability distribution. The server minimizes the following perplexity loss function based on the true label y of the calibration sample:

[0052]

[0053] Where P(·) represents the model prediction probability after applying the injection coefficients of all layers. The server uses optimization algorithms such as gradient descent to calculate the set of injection coefficients of all layers. Update until the loss is minimized.

[0054] The context vector structure in step 1) is used to represent a multi-dimensional context representation set organized across layers.

[0055] The context vector structure refers to:

[0056] For a pre-trained language model with L layers, each layer extracts two activation vectors from multiple task samples, namely the sample activation vectors of the Attention layer and the MLP layer. The sample activation vectors of each layer are mean-fused to form the following representation:

[0057]

[0058] in and respectively The fusion vector of the Attention layer and the MLP layer.

[0059] This technical solution method achieves the following improvements:

[0060] (1) Lightweight communication: Transmission vectors replace model parameters (e.g., LLaMA parameter transmission time is reduced from 36 hours to seconds).

[0061] (2) Privacy and security: Data and parameters are kept local, and only low-dimensional vectors are shared.

[0062] (3) Improved performance with few samples: Experiments show that Fed-I2CL performs significantly better than zero-sample performance.

[0063] While protecting privacy, this technical solution effectively improves the ability of large language models to understand specific tasks in federated scenarios. Compared with the fine-tuning method of federated large models, it can effectively reduce data transmission requirements and greatly reduce communication overhead. It provides an efficient and secure solution for the distributed deployment of large models in privacy-sensitive fields such as healthcare, finance, and the Internet of Things, and promotes the universal application of AI in edge computing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of the structural framework of the embodiment. DETAILED DESCRIPTION

[0065] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the present invention is not limited thereto.

[0066] Example:

[0067] This example uses a modern enterprise office scenario. Multiple branches or subsidiaries serve as federated learning clients, each deploying local servers. These servers store a large amount of sensitive internal data, such as project documents, meeting minutes, technical manuals, and internal chat logs. Due to the high sensitivity of this data, traditional federated learning requires frequent transmission of model parameters, which can lead to communication bottlenecks and privacy risks.

[0068] Reference Figure 1An implicit federated large model context learning method is provided to achieve efficient collaborative learning, the method comprising the following steps:

[0069] 1) All clients and servers extract local context vector structures through the large model:

[0070] All clients and servers in the enterprise first design a unified context sample template based on tasks such as enterprise document classification, question and answer generation, and employee knowledge recommendation. All clients and servers construct context sample prompts for each currently stored local data according to the context sample template. The sample prompt includes an input content and its target label. All clients and servers input their respective context sample prompts one by one into the local pre-trained large language model, and extract the intermediate activation vector of the token position of the sample prompt used for prediction in each layer of the model. The activation vector includes the output vector of the Attention layer and the output vector of the MLP layer. Let the output of the i-th sample prompt at the l-th layer be , , there are N sample prompts in total, forming the sample activation set:

[0071]

[0072] The activation vector sample dimension of each layer is averaged to obtain the fused local context vector:

[0073] ,

[0074] The context vector structure that constitutes the cross-layer organization is expressed as:

[0075]

[0076] Where L is the total number of model layers. Both the client and the server perform the above context extraction process and generate a context vector structure respectively.

[0077] 2) Server aggregated global context vector structure:

[0078] All clients upload their local context vector structures to the headquarters server. At the same time, the server extracts the local context vector at each layer according to the process of extracting the context vector structure in step 1), which is recorded as:

[0079]

[0080] For each client k (k=1,2,...,K), its local context vector at layer l is:

[0081]

[0082] The server performs average aggregation on the context vectors of all clients and itself at layer l to generate a global context vector at layer l:

[0083]

[0084] The aggregation calculation of this layer requires indexing each layer of the model Repeat the process in sequence, and finally the server obtains the global context vector structure organized across layers:

[0085]

[0086] For example, the context vector for a technical department might focus on technical terminology and R&D scenarios, while the vector for the marketing department might reflect product promotional language and customer preferences. Each time a client uploads a context vector to the headquarters server, only KB of data is required, far less than the traditional model parameter transmission volume, significantly reducing the communication burden.

[0087] 3) The server injects the global context vector into the original large model and calibrates the injection coefficients using the server's local calibration dataset:

[0088] The server first uses standardized enterprise task datasets, such as typical document classification samples, to classify each layer in the global context vector structure. The vector, i.e., the output vector of the Attention and MLP sub-layers, is injected into the residual stream of the original large model. The learning and calibration process of the linear injection coefficient during the injection process is as follows:

[0089] The server first aggregates the global context vector structure:

[0090] The server injects the data into the original large model, which is the same as the client model and keeps its parameters frozen. The structure of the original large model remains unchanged, and only the context vector is inserted during inference. Next, the server uses its locally retained enterprise task dataset to perform the following calibration process:

[0091] The server randomly selects a batch of calibration samples from the locally retained enterprise task dataset, denoted as:

[0092]

[0093] in For sample output, For the corresponding labels, these calibration samples are distributed identically with the client samples; for a given calibration sample , the server passes the input x to the large model for layer-by-layer forward propagation. Assuming that at layer l and position t, the large model first obtains the output of the previous layer of the residual stream during standard forward calculation: At the same time, the large model will calculate the original Attention activation of the sample at the corresponding position in the lth layer and MLP activation ;

[0094] Server from Take out the global context vector of layer l , and inserted into the residual stream in the linear combination. For each calibration sample, the residual update process of the t-th position of the l-th layer is as follows:

[0095]

[0096] in:

[0097] and They are the lth layer Attention and MLP global context vector obtained by aggregation in step 2);

[0098] and are the original Attention and MLP activation of the sample at layer l and position t;

[0099] is the residual output of the previous layer, It is the lth layer Attention mixed injection, It is the MLP mixed injection of the lth layer;

[0100] It is a hierarchical learnable scalar coefficient used to adjust the mixing ratio of the context vector and the original query activation to obtain the final corrected injection coefficient;

[0101] The entire update does not change the model's weight matrix, but only performs a linear combination operation on the residual flow;

[0102] The server inserts the l=1,2,...,L layers into the global context vector according to the above residual update process and updates the residual stream until the model outputs the final predicted distribution;

[0103] After all layers are injected, the model outputs the final predicted probability distribution. The server minimizes the following perplexity loss function based on the true label y of the calibration sample:

[0104]

[0105] Where P(·) represents the model prediction probability after applying the injection coefficients of all layers. The server uses optimization algorithms such as gradient descent to calculate the set of injection coefficients of all layers. Update until the loss is minimized, automatically adjusting the mix of Attention and MLP activations in different layers by minimizing perplexity. For example, for document archiving tasks, the attention injection coefficient of the middle layer may be higher to enhance the ability to model reference relationships between contexts.

[0106] 4) The server distributes the global context vector structure and the corrected injection coefficients to the client, which receives and injects them into the local large model:

[0107] The server aggregates the global context vector structure V global and its corrected injection coefficient This is distributed to all clients. Local clients inject the global context vector into their local large model according to the injection strategy, generating a task-enhanced large model. This task-enhanced large model is then applied to local business scenarios. For example, the technical department model excels at searching internal manuals and answering technical questions, while the legal department model improves contract analysis. The original data and model parameters remain locally, with knowledge shared only through low-dimensional vectors, enabling secure and efficient information collaboration within the enterprise.

[0108] Technical problem solving effect:

[0109] 1) Communication efficiency: Transmission time is reduced from hours to seconds, adapting to the network limitations of IoT devices.

[0110] 2) Privacy and security: Avoid direct transmission of voice or video data to meet the requirements of privacy regulations such as GDPR.

[0111] 3) Heterogeneous data compatibility: Unify multi-source data (voice, text, sensor signals) through context templates to alleviate the Non-IID problem.

[0112] experiment:

[0113] Experiment 1:

[0114] Experimental setup:

[0115] Dataset partitioning: The server retains 200 samples, and the remaining samples are randomly distributed to clients according to the Dirichlet (0.5) distribution.

[0116] .

[0117] Dataset labels:

[0118] .

[0119] Through the analysis and comparison of experimental results, the following conclusions can be drawn:

[0120] Comparison between zero-sample and Fed-I2CL:

[0121] Fed-I2CL's performance is significantly better than zero-shot performance, indicating that aggregating the context vectors extracted from each client can effectively improve the performance of large models on task datasets. This also indicates that this method can effectively improve the LLM's ability to understand specific tasks in federated scenarios.

[0122] Experiment 2:

[0123] .

[0124] Through the analysis and comparison of experimental results, the following conclusions can be drawn:

[0125] Comparison between zero-sample and Fed-I2CL:

[0126] The performance of Fed-I2CL is better than that of client-local ICL, which shows that the method of aggregating the context vectors extracted by each client can effectively aggregate the context information of multiple clients, achieve better context learning effects, and alleviate the data island problem.

Claims

1. An implicit federated large model context learning method, characterized by: The method involves the following entities: Client: Deployed by the data owner, it has local data storage capabilities and is used to extract task-related context information; Server: Deployed in a central node or headquarters data center, it can aggregate context vectors across clients and optimize context injection coefficients. Large model: refers to the pre-trained language model deployed on both the client and server. It is used to generate sample prompts, participate in inference and activation extraction. The client model does not need to update parameters and only performs vector extraction and transmission operations. Context sample template; The method comprises the following steps: The federated learning system of the method is composed of multiple edge computing nodes, i.e., clients, and a central coordination server, i.e., server; All clients and servers extract local context vector structures through the big model: 1) All clients and servers first design a context sample template based on the current task type. All clients and servers construct a context sample prompt for each currently stored local data according to the context sample template. The sample prompt includes an input content and its target label. All clients and servers input their respective context sample prompts one by one into the local pre-trained large language model and extract the intermediate activation vector of the token position used for prediction in each layer of the model. The activation vector includes the output vector of the Attention layer and the output vector of the MLP layer. Let the output of the i-th sample prompt at the l-th layer be , , there are N sample prompts in total, forming the sample activation set: The activation vector sample dimension of each layer is averaged to obtain the fused local context vector: ， The context vector structure that constitutes the cross-layer organization is expressed as: Where L is the total number of model layers. Both the client and the server perform the above context extraction process and generate their own context vector structure to support the aggregation and injection of the federated context structure into the large model to achieve enhanced effects. 2) Server aggregated global context vector structure: All clients upload their local context vector structures to the central server. At the same time, the server extracts the local context vector at each layer according to the process of extracting the context vector structure in step 1), which is recorded as: For each client k (k=1,2,...,K), its local context vector at layer l is: The server performs average aggregation on the context vectors of all clients and itself at layer l to obtain the global context vector of layer l: The aggregation calculation of this layer requires indexing each layer of the model Repeat the process in sequence, and finally the server obtains the global context vector structure organized across layers: ； 3) The server injects the global context vector into the original large model and calibrates the injection coefficients using the server’s local calibration dataset; 4) The server distributes the global context vector structure and the corrected injection coefficients to the client, which receives and injects them into the local large model: The server aggregates the global context vector structure V global and its corrected injection coefficient The global context vector is distributed to all clients together. The client injects the global context vector into the residual stream of the local frozen large model according to the coefficients to generate a task-enhanced large model, and uses the task-enhanced large model for inference tasks.

2. The implicit federated large model context learning method according to claim 1 is characterized in that: The specific process of step 3) is as follows: The server will first use its own retained calibration sample set to calibrate each layer in the global context vector structure. The vector, i.e., the output vector of the Attention and MLP sub-layers, is injected into the residual stream of the original large model. The learning and calibration process of the linear injection coefficient during the injection process is as follows: The server first aggregates the global context vector structure: The server injects the data into the original large model that is identical to the client and keeps its parameters frozen. The structure of the original large model remains unchanged, and only the context vector is inserted during inference. Next, the server uses its locally retained calibration sample set to perform the following calibration process: The server randomly selects a batch of calibration samples from the locally retained labeled dataset, denoted as: in For sample output, For the corresponding labels, the calibration samples are distributed identically with the client samples; For a given calibration sample , the server passes the input x to the large model for layer-by-layer forward propagation. Assuming that at layer l and position t, the large model first obtains the output of the previous layer of the residual stream during standard forward calculation: At the same time, the large model will calculate the original Attention activation of the sample at the corresponding position in the lth layer and MLP activation ; Server from Take out the global context vector of layer l , and inserted into the residual stream in the linear combination. For each calibration sample, the residual update process of the t-th position of the l-th layer is as follows: in: and They are the lth layer Attention and MLP global context vector obtained by aggregation in step 2); and are the original Attention and MLP activation of the sample at layer l and position t; is the residual output of the previous layer, It is the lth layer Attention mixed injection, It is the MLP mixed injection of the lth layer; It is a hierarchical learnable scalar coefficient used to adjust the mixing ratio of the context vector and the original query activation to obtain the final corrected injection coefficient; The entire update does not change the model's weight matrix, but only performs a linear combination operation on the residual flow; The server inserts the l=1,2,...,L layers into the global context vector according to the above residual update process and updates the residual stream until the model outputs the final predicted distribution; After all layers are injected, the model outputs the final predicted probability distribution. The server minimizes the following perplexity loss function based on the true label y of the calibration sample: Where P(·) represents the model prediction probability after applying the injection coefficients of all layers. The server uses optimization algorithms such as gradient descent to calculate the set of injection coefficients of all layers. Update until the loss is minimized.

3. The implicit federated large model context learning method according to claim 1, characterized in that: The context vector structure in step 1) is used to represent a multi-dimensional context representation set organized across layers. The context vector structure refers to: For a pre-trained language model with L layers, each layer extracts two activation vectors from multiple task samples, namely the sample activation vectors of the Attention layer and the MLP layer. The sample activation vectors of each layer are mean-fused to form the following representation: in and respectively The fusion vector of the Attention layer and the MLP layer.

Images