Federal large model knowledge collaborative training method supporting multi-modal heterogeneous client

Abstract

The invention discloses a federal large model knowledge collaborative training method supporting multi-modal heterogeneous clients, which comprises the following steps: each client receives a model initialization parameter issued by a central server, and applies adaptive differential privacy noise to independently train a heterogeneous lightweight model based on local private data; updating the model to which the noise is applied and uploading a modal identifier of the model to a central server side; after model updating and modal identification of each client are received, based on a modal perception weighted consensus fusion mechanism, knowledge of each client is fused to update a global large model; and the central server side issues the updated presentation layer parameters of the global large model to the client side for initialization of the next round of local training. According to the method, on the premise that a public data set or specific task setting is not needed, comprehensive compatibility of data isomerism, client dynamic participation, model diversity and privacy protection requirements is achieved, and the adaptability, stability and knowledge utilization efficiency of large model federation training are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and privacy computing technology, and particularly relates to a federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients. BACKGROUND

[0002] With the wide application of large language models (LLMs) and multi-modal large models in natural language processing, computer vision and other fields, the deployment demand of large models in high-sensitive scenarios such as medical diagnosis, financial risk control and intelligent government affairs is increasing. However, the training of large models relies on a large amount of high-quality data, and the data of various institutions in reality cannot be shared due to reasons such as privacy protection, compliance supervision or commercial competition, forming a serious "data island" problem.

[0003] Under this background, as a distributed machine learning paradigm, federated learning has become an important technical path to realize the collaborative training of large models across institutions, because of its core concept of "data not moving, model moving". In recent years, federated learning has gradually extended from the application of traditional small-scale models to large model scenarios, supporting multiple participants to jointly optimize a global large model under the premise of ensuring data privacy, and providing a feasible technical framework for the landing of large models in privacy-sensitive fields.

[0004] Although federated learning has shown application potential in the field of large models, due to the large parameter scale of large models, the limited capacity of client devices, the diversity of data modalities and the higher privacy requirements, there are still several key challenges in actual application.

[0005] Firstly, insufficient client resources: mainstream federated learning methods usually require clients and central servers to use the same model structure, i.e. model isomorphism constraint. However, large models usually have a large parameter scale and complex network architecture, which puts high requirements on computing resources, memory capacity and energy consumption level, while most client devices are limited by hardware conditions and are difficult to carry out local training and inference with such a large scale model, resulting in low training efficiency or even inability to run.

[0006] Secondly, lack of multi-modal heterogeneous data fusion mechanism: in real application scenarios, different clients may have different types of data modalities, such as images, texts or time series signals, and existing methods lack effective fusion mechanism for multi-modal heterogeneous data, resulting in low efficiency of knowledge integration.

[0007] Thirdly, balance between privacy protection and large model performance: in order to further enhance privacy protection, differential privacy technology is often introduced into the federated learning process, but traditional differential privacy mechanism usually adopts a unified noise addition strategy, which fails to fully consider the functional differences between different levels within the large model, and is prone to cause distortion of key layer information or increase of privacy leakage risk of non-key layer, making it difficult to achieve fine-grained privacy-utility trade-off.

[0008] To address the above problems, existing research attempts to improve the federated learning framework through knowledge distillation, meta-learning or parameter aggregation. However, these methods still have obvious defects when facing the actual demand of large models. For example, although some solutions allow clients to use lightweight models, they do not solve the confidence evaluation problem in cross-modal knowledge fusion, resulting in interference of low-quality or irrelevant modal knowledge in global model updating; some other methods use fixed strength noise disturbance for privacy protection, which cannot dynamically adjust the disturbance degree according to the importance of the gradient, so it is difficult to achieve a good balance between model performance and privacy protection. In addition, existing technologies generally lack systematic support for the diversity of client model structures, still relying on strong homogeneity assumptions, limiting their applicability in heterogeneous device environments. SUMMARY

[0009] To address the deficiencies of existing technologies, the present application aims to provide a federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients, allowing each client to use different lightweight models for local training. When uploading, adaptive privacy noise is added according to the importance of each layer of the model; at the central server end, the knowledge of each client is weighted and fused according to data modalities and prediction confidence, and diversity is preserved; finally, the optimized representation layer knowledge is fed back to the client, realizing safe, efficient and flexible collaborative learning.

[0010] The purpose of the present application is achieved by the following technical solutions: To achieve the above-mentioned application purposes, the federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients provided by the embodiments includes the following steps: Step 1: Each client receives the model initialization parameters issued by the central server end, independently trains the heterogeneous lightweight model deployed by the client based on local private data, and obtains local model updates; Step 2: During local training, each client applies adaptive differential privacy noise to the local model updates, where the noise intensity is dynamically adjusted according to the functional importance of the model layers; Step 3: Each client uploads the model updates with noise and its modal identifier to the central server end; Step 4: The central server end receives the model updates and modal identifiers from each client, and updates the global large model by fusing the knowledge of each client based on the modal-aware weighted consensus fusion mechanism; Step 5: The central server end issues the updated representation layer parameters of the global large model to the client for initialization of the next round of local training.

[0011] In one embodiment, in step 1, the heterogeneous lightweight model of each client supports multi-modal input, including image, text or time series data.

[0012] In one embodiment, in step 2, the adaptive differential privacy noise is applied to the local model update, including: For the model level with high semantic relevance to the target task, a low noise intensity is adopted to retain key semantics; For the model level with low semantic relevance to the target task, a high noise intensity is adopted to strengthen privacy protection.

[0013] In one embodiment, the semantic relevance of the target task is measured by the average gradient amplitude, calculated as follows: , In the formula, is the semantic relevance of the target task, is the number of local training steps, is the gradient of the i-th layer at the j-th step, with a value range of [1, n]; a threshold value is set is the average of the gradient amplitudes of all layers, if , the semantic relevance of this layer is high; if , the semantic relevance of this layer is low.

[0014] Further, the corresponding noise intensity is obtained based on the following calculation formula: , In the formula, is the reference noise scale, , The higher the semantic relevance of the target task is, the smaller the noise intensity is.

[0015] In one embodiment, in step 4, the modal perception weighted consensus fusion mechanism is used to fuse the knowledge of each client to update the global large model, including: Classify the clients according to the modal identifier; Obtain the prediction output of each client model on the public sample using the public dataset; Calculate the prediction confidence of each client based on the variance of its prediction output; Determine the modal matching factor of each client according to the consistency of its modal identifier and the task modal of the public dataset; Assign a fusion weight to each client according to the prediction confidence and the modal matching factor; Based on the fusion weight, perform weighted averaging on the prediction output of each client to generate a weighted soft label; ​​​​​updating the global large model parameters to minimize the difference between the global large model output and the weighted soft labels.

[0016] In one embodiment, the loss function used to update the global large model parameters also includes a diversity regularization term, which is constructed through a contrastive learning mechanism to encourage the preservation of unique representations of low-confidence clients.

[0017] In one embodiment, in step 3, the contrastive learning mechanism includes: For any client, pull its representation layer output closer to the representation layer output of positive sample clients and push it farther away from the representation layer output of negative sample clients; wherein the positive sample clients are clients with the same modality as the client and high confidence, and the negative sample clients are clients with different modalities or the same modalities but low confidence, and the calculation formula is as follows: , In the formula, represents the total number of clients participating in this round of fusion, is the representation layer output of the client , represents a similarity calculation function, represents the negative sample set of the client , is a temperature coefficient. The positive sample represents the client representation consistent with the modality of the client and with high confidence, and the negative sample represents the client representation different from the modality of the client or the same modality but with low confidence.

[0018] In one example, the central server updates the global large model by minimizing the following loss function: , In the formula, represents the probability distribution of the soft label; represents the target probability distribution, i.e., the consensus result after multi-client knowledge fusion; is used to measure the difference between two probability distributions, is the diversity regularization term loss, is a regularization coefficient.

[0019] The present application also provides a federated large model knowledge collaborative training system for multi-modal heterogeneous clients, which is used to implement the federated large model knowledge collaborative training method, and includes a plurality of clients and a central server end. The client is used for local training based on local private data through a heterogeneous lightweight model training module; the adaptive privacy processing module is used for applying differential privacy noise dynamically adjusted according to model level function importance to local model updating; the modal identification module is used for identifying the data modal of the client, and then the secure communication module is used for performing encrypted communication with the central server end; The central server end is used for realizing weighted consensus fusion based on the modal matching degree and the prediction confidence of the client through a modal perception fusion module; the global model updating module is used for updating the global large model according to the fusion result; the representation layer issuing module is used for issuing the updated global large model parameters to the client; and then the secure communication interface is used for performing encrypted communication with the client.

[0020] The application further provides a federated large model knowledge collaborative training device supporting a multi-modal heterogeneous client, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the federated large model knowledge collaborative training method supporting the multi-modal heterogeneous client when executing the computer program.

[0021] Compared with the prior art, the application has at least the following beneficial effects: (1) The application constructs a federated large model knowledge collaborative training mechanism of a multi-modal heterogeneous client, and systematically solves the key bottlenecks in the application of model integration and knowledge distillation in federated learning in the prior art. Compared with the traditional method, the application realizes the comprehensive compatibility of data heterogeneity, client dynamic participation, model diversity and privacy protection requirements without relying on public data sets or specific task settings, and significantly improves the adaptability, stability and knowledge utilization efficiency of federated training of a large model.

[0022] (2) The application allows the deployment of lightweight models with different structures by the client, and designs a weighted consensus fusion mechanism of the central server end, thereby realizing effective collaboration of heterogeneous models under non-independent and identically distributed data and improving the applicability of the system in complex edge environments.

[0023] (3) The application introduces a diversity regularization mechanism on the basis of weighted fusion, thereby suppressing interference information while retaining the differentiated characteristics of low-weight clients, improving the learning ability of the global model for rare patterns, and enhancing the robustness and generalization performance of the system.

[0024] (4) The application proposes a double-weighting strategy based on modal matching degree and prediction confidence, dynamically evaluates the knowledge contribution degree of each client, makes the output with high confidence and modal correlation obtain a higher fusion weight, effectively prevents the optimization direction from being dominated by false knowledge, and improves the knowledge integration accuracy and learning efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or prior art description will be briefly introduced.

[0026] Figure 1 The overall framework diagram of the federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients provided by the present application. DETAILED DESCRIPTION

[0027] In order to make the objects, technical solutions and advantages of the present application clearer, the following will combine the drawings and specific embodiments to further illustrate the present application. The specific embodiments described herein are merely intended to explain the present application, and do not limit the protection scope of the present application.

[0028] The existing federated learning method has significant limitations when combining model integration and knowledge distillation mechanism: the traditional integration strategy assumes that the model is trained on homogeneous data and the participation process is stable, which is difficult to adapt to the asynchronous iteration characteristics of federated learning, such as client data heterogeneity, large device capability difference, and only part of the clients participating in each round; the existing fusion mechanism often ignores the potential useful representation contained in the low-confidence or consensus-deviating clients, simply suppressing their output, resulting in knowledge waste; the equal weight strategy is generally used in the fusion process, without considering the difference in client prediction credibility and modal correlation, which is easily disturbed by low-quality knowledge. The present application proposes a federated large model knowledge collaborative training method for multi-modal heterogeneous clients. The main technical innovations of this method are as follows: 1. Support for deploying lightweight models with different structures for local training, breaking through the model homogeneity constraint; 2. Introduce gradient-correlation adaptive differential privacy mechanism, dynamically adjust the noise intensity according to the importance of model level function; 3. Construct a modal-aware weighted consensus fusion mechanism to intelligently allocate weights based on data modal matching degree and prediction confidence; Terminology explanation: 1. Federated large model collaborative training: a technical paradigm in which multiple clients and a central server collaboratively train a large global model without sharing raw data.

[0029] 2. Heterogeneous clients: the state of each client device participating in federated learning, which has differences in hardware capability, network condition, data modal or local model structure.

[0030] 3. Lightweight model: a neural network structure deployed on the client with a significantly smaller parameter scale than the global large model.

[0031] 4. Heterogeneous data: the phenomenon that the local data held by each client in federated learning differs in modal type, sample distribution, feature space, or label structure.

[0032] 5. Gradient-relevance adaptive differential privacy: a privacy protection mechanism that dynamically adjusts the noise intensity according to the relevance of the level where the model parameters are located to the semantic of the target task.

[0033] 6. Modal-aware weighted consensus fusion: a technique that dynamically allocates fusion weights according to the matching degree of the client data modal to the task (modal awareness) and the reliability of its predicted output (confidence) when aggregating models on the server side of federated learning.

[0034] 7. Confidence: an evaluation index of the reliability of the model's prediction results, usually quantified by the distribution characteristics of the output layer logits.

[0035] The core of the present application is to construct a large model federated collaborative training process for multi-modal heterogeneous clients: each client first uses a lightweight heterogeneous model based on local private data to complete local training; then, before uploading the model update, adaptive differential privacy noise is added according to the importance of each layer to the task; then, the processed model update is uploaded to the central server along with the modal identifier; after receiving, the central server side weights and fuses the knowledge of each client according to the modal matching degree and prediction confidence, generating an updated global large model; finally, the central server side selectively distributes the representation layer parameters of the model to participating clients for initializing the next round of training.

[0036] The corresponding model framework is shown in Figure 1 The functions of each part are analyzed in detail as follows: S1, each client receives the model initialization parameters distributed by the central server, independently trains the lightweight heterogeneous model based on local private data, and obtains local model updates.

[0037] In the embodiment, each participating client deploys a lightweight model with different structures for local training according to its device capability and data modal. The parameter scale of the lightweight model is significantly smaller than that of the global large model, which is suitable for resource-constrained environments such as mobile devices, medical terminals, or edge computing nodes, avoiding the problem of excessive computational load caused by model homogeneity. After deploying lightweight models with different structures, multiple rounds of local stochastic gradient descent (SGD) training are performed based on local private data sets After training, the local model parameters are saved, where is the global communication round of federated learning, is the local iteration number.

[0038] S2. During local training, each client applies adaptive differential privacy noise to the local model update, where the noise intensity is dynamically adjusted according to the hierarchical functional importance of the model.

[0039] In this embodiment, when the client uploads local model update information to the central server, a gradient-correlation adaptive differential privacy mechanism is adopted. Before the client uploads the local model update gradient, the noise intensity is dynamically adjusted according to the functional importance of the model level.

[0040] The functional importance of a model hierarchy is measured by the mean of its gradient magnitudes, calculated as follows: , In the formula, The importance of functions at each model level. For local training steps, For the first Layer The gradient of the step is in the range [1, n]; a threshold is set. The mean of the gradient magnitudes across all layers, if ,Should The higher the semantic relevance of the target task at each layer, the higher its functional importance; if ,Should Layer-specific target tasks have low semantic relevance and low functional importance.

[0041] The corresponding noise intensity is further obtained based on the following calculation formula: , In the formula, As the reference noise scale, , The higher, The smaller the value, the lower the noise intensity.

[0042] Low noise intensity is added to key layers with strong task relevance (such as the classification head) to preserve semantic integrity, while high noise intensity is added to non-key layers with weak relevance (such as the shallow feature extraction module) to enhance privacy protection. This achieves a fine balance between privacy protection and model utility, avoiding the accuracy loss or privacy leakage caused by traditional static noise.

[0043] The gradient is clipped and noise is added to ensure that the overall performance meets the requirements. -Differential Privacy: , In the formula, For the processed first The estimated value of the layer gradient, For the first The original model gradient of the layer, For the first layer preset clipping threshold, for added noise, for noise scale, for identity matrix.

[0044] S3, each client uploads the model updated after applying noise and its modal identifier to the central server.

[0045] In the embodiment, the client uploads the gradient processed by S2 and the modal identifier, such as "image" and "text", to the central server through an encrypted channel to ensure transmission security.

[0046] S4, the central server receives the model updates and modal identifiers of each client, and updates the global large model by fusing the knowledge of each client based on the modal-aware weighted consensus fusion mechanism.

[0047] In the embodiment, after the central server receives the data (modal, gradient) uploaded by each client, it classifies the clients by modal and averages the representation layer output of the model of the same modal: , wherein, is the representation layer output of the model, is the set of clients of the modal , is the global communication round of federated learning, the th client completes local round training in the th global communication round, and the representation layer output of the local model obtained is

[0048] The central server uses a public dataset to let each client model infer the sample and output . represents the logits output of the client to . At the same time, the confidence weight of each client is calculated: , wherein, is the variance of to measure the prediction confidence; is the modal matching factor: if the modal of the client is consistent with , then , otherwise ; and generates a weighted soft label : ; In addition, a diversity regularization term is introduced to encourage low-confidence clients to retain unique representations. A contrastive learning mechanism is adopted: , where, denotes the total number of clients participating in the current round of fusion, is the representation layer output of client , denotes the similarity calculation function, denotes the negative sample set of client , is the temperature coefficient. Positive samples denote clients whose representations are consistent with the modalities of client and have high confidence, and negative samples denote clients whose representations are inconsistent with the modalities of client or have low confidence.

[0049] By assigning higher fusion weights to clients with modal matching and stable predictions, and appropriately reducing the weights of clients with modal irrelevance or low confidence and retaining their diversity features, the direction of global model optimization is prevented from being disturbed by false knowledge.

[0050] Finally, the central server end updates the global large model by minimizing the following loss function: , where, denotes the probability distribution of the soft label; denotes the target probability distribution, i.e., the consensus result after multi-client knowledge fusion; is used to measure the difference between the two probability distributions, is the diversity regularization term loss, is the regularization coefficient.

[0051] S5, the central server end distributes the updated representation layer parameters of the global large model to the clients, which are used for initialization of the next round of local training.

[0052] The central server end distributes the representation layer parameters in the global large model to the clients as the basis for initialization of the next round of local training. A knowledge closed loop is formed, allowing clients to continuously absorb global knowledge and improve their local model representation capabilities.

[0053] In order to clearly show a federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients, the embodiment also provides a federated large model knowledge collaborative training system of multi-modal heterogeneous clients, which is used to implement the federated large model knowledge collaborative training method and includes a plurality of clients and a central server end. The client is used to perform local training based on local private data through a heterogeneous lightweight model training module, to apply differential privacy noise dynamically adjusted according to model level function importance to local model updates through an adaptive privacy processing module, to identify the data modality of the client through a modality identification module, and to perform encrypted communication with the central server end through a secure communication module. The central server end is used to realize weighted consensus fusion based on the modality matching degree and the prediction confidence of the client through a modality perception fusion module, to update the global large model according to the fusion result through a global model update module, to distribute the updated global large model parameters to the client through a representation layer distribution module, and to perform encrypted communication with the client through a secure communication interface.

[0054] In addition, the embodiment also provides a federated large model knowledge collaborative training device supporting multi-modal heterogeneous clients, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients when executing the computer program.

[0055] The specific embodiments described above have explained the technical solutions and advantages of the present application, and it should be understood that the above description is only the most preferred embodiment of the present application and is not used to limit the present application, and any modification, supplement and equivalent replacement made within the principle range of the present application should be included in the protection range of the present application.

Claims

1. A federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients, characterized in that, The method comprises the following steps: Step 1: Each client receives the model initialization parameters issued by the central server end, independently trains the heterogeneous lightweight model deployed by the client based on local private data, and obtains local model updates; Step 2: Each client applies adaptive differential privacy noise to the local model updates during local training, and the noise intensity is dynamically adjusted according to the hierarchical function importance of the model; Step 3: Each client uploads the model updates and their modal identification after applying noise to the central server end; Step 4: The central server end receives the model updates and modal identification of each client, and updates the global large model based on the modal perception weighted consensus fusion mechanism by fusing the knowledge of each client; Step 5: The central server end issues the representation layer parameters of the updated global large model to the client for initialization of the next round of local training.

2. The federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients according to claim 1, characterized in that, In step 1, the heterogeneous lightweight model of each client supports multi-modal input, including image, text or time series data.

3. The federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients according to claim 1, characterized in that, In step 2, the adaptive differential privacy noise applied to the local model updates includes: For model levels with high semantic relevance to the target task, low noise intensity is used to preserve key semantics; For model levels with low semantic relevance to the target task, high noise intensity is used to enhance privacy protection.

4. The federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients according to claim 3, characterized in that, The semantic relevance of the target task is measured by the average gradient amplitude, calculated as follows: ， In the formula, is the target task semantic relevance, is the local training step number, is the gradient of the first layer first step, and the value range is [1, n]. Setting threshold is the mean of the gradient magnitudes of all layers, if the layer semantics are high; if the layer semantics are low. the layer semantics are low. the layer semantics are high.

5. The federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients according to claim 1, characterized in that, In step 4, the modal perception weighted consensus fusion mechanism is used to fuse the knowledge of each client to update the global large model, including: Classify the clients according to the modal identification; Use a public dataset to obtain the prediction output of each client model on public samples; Calculate the prediction confidence of each client based on the variance of its prediction output; Determine the modal matching factor of each client according to the consistency of its modal identification and the task modal of the public dataset; Assign a fusion weight to each client based on the prediction confidence and the modal matching factor; Perform weighted averaging on the prediction output of each client based on the fusion weight to generate a weighted soft label; Update the global large model parameters to minimize the difference between the global large model output and the weighted soft label.

6. The federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients according to claim 5, characterized in that, The loss function used to update the global large model parameters also includes a diversity regularization term, which is constructed through a contrastive learning mechanism to encourage the preservation of unique representations of low-confidence clients.

7. The federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients according to claim 6, characterized in that, The contrastive learning mechanism includes: For any client, the representation layer output is pulled closer to the representation layer output of the positive sample client and pushed farther away from the representation layer output of the negative sample client; wherein the positive sample client is a client with the same modal as the client and high confidence, and the negative sample client is a client with different modal or the same modal but low confidence, and the calculation formula is as follows: ， In the formula, denotes the total number of clients participating in this round of fusion, is the client The presentation layer output, denotes the similarity calculation function, denotes the negative sample set of the client , the temperature coefficient, The positive sample denotes the client representation consistent with the client Modality and high confidence, negative sample denotes the client representation of the client Different modalities, or the same modalities but low confidence.

8. The federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients according to claim 7, characterized in that, The central server end updates the global large model by minimizing the following loss function: ， In the formula, represents the probability distribution of the soft label; represents the target probability distribution, that is, the consensus result after multi-client knowledge fusion; is used to measure the difference between two probability distributions, is a diversity regularization term loss, is a regularization coefficient.

9. A federated large model knowledge collaborative training system of a multi-modal heterogeneous client, characterized in that, The federated large model knowledge collaborative training system is used to implement the federated large model knowledge collaborative training method of any one of claims 1-8, comprising a plurality of clients and a central server end; The client is used for local training based on local private data through a heterogeneous lightweight model training module; the adaptive privacy processing module is used for applying differential privacy noise dynamically adjusted according to the importance of model layer functions to local model updates; the modal identification module is used for identifying the data modal of the client, and then the secure communication module is used for performing encrypted communication with the central server end; The central server end is used for realizing weighted consensus fusion based on the modal matching degree and the prediction confidence of the client through a modal perception fusion module. The global model update module is used for updating the global large model according to the fusion result; the representation layer distribution module is used for distributing the updated global large model parameters to the client; and then the secure communication interface is used for performing encrypted communication with the client.

10. A federated large model knowledge collaborative training device supporting multi-modal heterogeneous clients, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients according to any one of claims 1-8 when the computer program is executed.

Images