Federal large model knowledge collaborative training method supporting multi-modal heterogeneous clients

Abstract

The application discloses a federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients, comprising the following steps: each client receives model initialization parameters issued by a central server, independently trains a heterogeneous light model based on local private data and applies adaptive differential privacy noise; uploads the model update after applying the noise and its modal identifier to the central server; after receiving the model update and the modal identifier of each client, the global large model is updated by fusing the knowledge of each client based on a modal perception weighted consensus fusion mechanism; the central server issues the representation layer parameters of the updated global large model to the client, which is used for the initialization of the next round of local training. Without relying on public data sets or specific task settings, the application realizes the comprehensive compatibility of data heterogeneity, client dynamic participation, model diversity and privacy protection requirements, and significantly improves the adaptability, stability and knowledge utilization efficiency of federated training of large models.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and privacy computing technology, and particularly relates to a federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients. BACKGROUND

[0002] With the wide application of large language models (LLMs) and multi-modal large models in natural language processing, computer vision and other fields, the deployment demand of large models in high-sensitive scenarios such as medical diagnosis, financial risk control and intelligent government affairs is increasing. However, the training of large models relies on a large amount of high-quality data, and the data of various institutions in reality cannot be shared due to reasons such as privacy protection, compliance supervision or commercial competition, forming a serious "data island" problem.

[0003] Under this background, as a distributed machine learning paradigm, federated learning has become an important technical path to realize the collaborative training of large models across institutions, because of its core concept of "data not moving, model moving". In recent years, federated learning has gradually extended from the application of traditional small-scale models to large model scenarios, supporting multiple participants to jointly optimize a global large model under the premise of ensuring data privacy, and providing a feasible technical framework for the landing of large models in privacy-sensitive fields.

[0004] Although federated learning has shown application potential in the field of large models, due to the large parameter scale of large models, the limited capacity of client devices, the diversity of data modalities and the higher privacy requirements, there are still several key challenges in actual application.

[0005] Firstly, insufficient client resources: mainstream federated learning methods usually require clients and central servers to use the same model structure, i.e. model isomorphism constraint. However, large models usually have a large parameter scale and complex network architecture, which puts high requirements on computing resources, memory capacity and energy consumption level, while most client devices are limited by hardware conditions and are difficult to carry out local training and inference with such a large scale model, resulting in low training efficiency or even inability to run.

[0006] Secondly, lack of multi-modal heterogeneous data fusion mechanism: in real application scenarios, different clients may have different types of data modalities, such as images, texts or time series signals, and existing methods lack effective fusion mechanism for multi-modal heterogeneous data, resulting in low efficiency of knowledge integration.

[0007] Thirdly, balance between privacy protection and large model performance: in order to further enhance privacy protection, differential privacy technology is often introduced into the federated learning process, but traditional differential privacy mechanism usually adopts a unified noise addition strategy, which fails to fully consider the functional differences between different levels within the large model, and is prone to cause distortion of key layer information or increase of privacy leakage risk of non-key layer, making it difficult to achieve fine-grained privacy-utility trade-off.

[0008] To address the aforementioned issues, existing research has attempted to improve federated learning frameworks through methods such as knowledge distillation, meta-learning, or parameter aggregation. However, these methods still have significant shortcomings when facing the practical needs of large models. For example, while some solutions allow clients to use lightweight models, they fail to address the confidence assessment problem in cross-modal knowledge fusion, leading to low-quality or irrelevant modal knowledge interfering with global model updates. Other methods employ fixed-intensity noise perturbations for privacy protection, failing to dynamically adjust the perturbation level based on gradient importance, thus making it difficult to achieve a good balance between model performance and privacy protection. Furthermore, existing technologies generally lack system support for the diversity of client model structures and still rely heavily on the isomorphism assumption, limiting their applicability in heterogeneous device environments. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the purpose of this invention is to provide a federated large model knowledge collaborative training method that supports multimodal heterogeneous clients, allowing each client to use different lightweight models for local training. During upload, privacy noise is adaptively added based on the importance of each layer of the model. On the central server, the knowledge from each client is weighted and fused according to data modality and prediction confidence, while preserving diversity. Finally, the optimized representation layer knowledge is fed back to the client, achieving secure, efficient, and flexible collaborative learning.

[0010] The objective of this invention is achieved through the following technical solution:

[0011] To achieve the above-mentioned objectives, an embodiment provides a method for collaborative training of knowledge in a federated large model that supports multimodal heterogeneous clients, comprising the following steps:

[0012] Step 1: Each client receives the model initialization parameters from the central server, independently trains the heterogeneous lightweight model deployed on the client based on local private data, and obtains local model updates;

[0013] Step 2: During local training, each client applies adaptive differential privacy noise to the local model update, where the noise intensity is dynamically adjusted according to the hierarchical functional importance of the model.

[0014] Step 3: Each client uploads the model update after applying noise and its modal identifier to the central server;

[0015] Step 4: The central server receives model updates and modal identifiers from each client, and integrates the knowledge from each client to update the global big model based on the modality-aware weighted consensus fusion mechanism.

[0016] Step 5: The central server end sends the updated representation layer parameters of the global large model to the client end for initialization of the next round of local training.

[0017] In one embodiment, in step 1, the heterogeneous lightweight models of each client support multi-modal input, including image, text or time series data.

[0018] In one embodiment, in step 2, the adaptive differential privacy noise is applied to the local model update, including:

[0019] For model levels with high semantic relevance to the target task, a low noise intensity is used to retain key semantics;

[0020] For model levels with low semantic relevance to the target task, a high noise intensity is used to strengthen privacy protection.

[0021] In one embodiment, the semantic relevance of the target task is measured by the average gradient amplitude, calculated as follows:

[0022] ,

[0023] In the formula, is the semantic relevance of the target task, is the number of local training steps, is the gradient of the i-th layer in the j-th step, with a value range of [1, n]; a threshold value is set is the average gradient amplitude of all layers, if , the semantic relevance of this layer is high; if , the semantic relevance of this layer is low. Further, the corresponding noise intensity is obtained based on the following calculation formula:

[0024] In the formula,

[0025] ,

[0026] In the formula, is the reference noise scale, , The higher the semantic relevance of the target task is, the smaller the noise intensity is.

[0027] In one embodiment, in step 4, the modal perception-based weighted consensus fusion mechanism includes:

[0028] Classify the clients according to the modal identifier;

[0029] ​​​​Using a public dataset, obtain the prediction output of each client model for the public samples;

[0030] The prediction confidence level is calculated based on the variance of the prediction output of each client.

[0031] Based on the consistency between the client's modality identifier and the task modality of the public dataset, determine its modality matching factor;

[0032] Based on the predicted confidence level and the modality matching factor, fusion weights are assigned to each client;

[0033] Based on the fusion weights, the prediction outputs of each client are weighted and averaged to generate weighted soft labels;

[0034] The global model parameters are updated with the goal of minimizing the difference between the global large model output and the weighted soft label.

[0035] In one embodiment, the loss function used to update the global large model parameters also includes a diversity regularization term, which is constructed through a contrastive learning mechanism to encourage the preservation of unique representations of low-confidence clients.

[0036] In one embodiment, step 3, the contrastive learning mechanism includes:

[0037] For any client, its representation layer output is brought closer to the representation layer output of a positive sample client and pushed further away from the representation layer output of a negative sample client. Here, a positive sample client is a client with the same modality and high confidence, and a negative sample client is a client with a different modality or the same modality but low confidence. The calculation formula is as follows:

[0038] ,

[0039] In the formula, This indicates the total number of clients participating in this round of integration. For the client The output of the presentation layer, This represents the similarity calculation function. Indicates client The negative sample set, Temperature coefficient. Positive sample. Indicates with the client Clients with consistent modality and high confidence levels indicate that negative samples Indicates with the client Client representations with different modalities, or with the same modality but low confidence.

[0040] In one instance, the central server updates the global large model by minimizing the following loss function:

[0041] ,

[0042] wherein, denotes the probability distribution of the soft label; denotes the target probability distribution, i.e., the consensus result after multi-client knowledge fusion; is used to measure the difference between two probability distributions, is a diversity regularization term loss, is a regularization coefficient.

[0043] The application further provides a federated large model knowledge collaborative training system of multi-modal heterogeneous clients, which is used to implement the federated large model knowledge collaborative training method, and comprises a plurality of clients and a central server end.

[0044] The client is used to perform local training based on local private data through a heterogeneous lightweight model training module, to apply differential privacy noise dynamically adjusted according to model level function importance to local model updates through an adaptive privacy processing module, to identify the data modalities of the client through a modality identification module, and to perform encrypted communication with the central server end through a secure communication module.

[0045] The central server end is used to realize weighted consensus fusion based on the modality matching degree and the prediction confidence of the client through a modality perception fusion module, to update the global large model according to the fusion result through a global model update module, to distribute the updated global large model parameters to the client through a representation layer distribution module, and to perform encrypted communication with the client through a secure communication interface.

[0046] The application further provides a federated large model knowledge collaborative training device supporting multi-modal heterogeneous clients, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients when executing the computer program.

[0047] Compared with the prior art, the application has at least the following beneficial effects:

[0048] (1) The application constructs a federated large model knowledge collaborative training mechanism of multi-modal heterogeneous clients, and systematically solves the key bottlenecks faced by the prior art when applying model integration and knowledge distillation in federated learning. Compared with traditional methods, the application realizes comprehensive compatibility of data heterogeneity, client dynamic participation, model diversity and privacy protection requirements without relying on public data sets or specific task settings, significantly improving the adaptability, stability and knowledge utilization efficiency of large model federated training.

[0049] (2) The application allows clients to deploy lightweight models with different structures, and designs a weighted consensus fusion mechanism on the central server side, realizes effective cooperation of heterogeneous models under non-independent and identically distributed data, and improves the applicability of the system in complex edge environments.

[0050] (3) The application introduces a diversity regularization mechanism on the basis of weighted fusion, suppresses interference information while retaining the differentiated characteristics of low-weight clients, improves the learning ability of the global model for rare patterns, and enhances the robustness and generalization performance of the system.

[0051] (4) The application proposes a dual weighting strategy based on modal matching degree and prediction confidence, dynamically evaluates the knowledge contribution of each client, so that the output with high confidence and modal correlation obtains a higher fusion weight, effectively prevents false knowledge from dominating the optimization direction, and improves the knowledge integration accuracy and learning efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced.

[0053] Figure 1 The overall framework diagram of the federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients provided by the application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the application more clear and obvious, the following will be combined with the drawings and specific embodiments to further describe the application.

[0055] The embodiments further illustrate the application. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the protection scope of the application.

[0056] The existing federated learning method has significant limitations when combining model integration and knowledge distillation mechanism: traditional integration strategies usually assume that models are trained on homogeneous data and the participation process is stable, which is difficult to adapt to the asynchronous iteration characteristics of federated learning, such as client data heterogeneity, large device capability difference, and only part of the clients participating in each round; The existing fusion mechanism often ignores the potential useful representation contained in the low confidence or consensus deviating client, simply suppresses its output, causing knowledge waste; During the fusion process, the equal weight strategy is generally used, without considering the difference in prediction confidence and modal correlation of the client, which is easy to be disturbed by low-quality knowledge. The application provides a federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients. The technical innovations mainly have the following three points:

[0057] 1. Support clients to deploy lightweight models with different structures for local training, break through the model isomorphism constraint;

[0058] 2. Introduce gradient-relevance adaptive differential privacy mechanism, dynamically adjust the noise intensity according to the importance of model layer function;

[0059] 3. Construct a modal perception weighted consensus fusion mechanism, realize intelligent weight distribution based on data modal matching degree and prediction confidence;

[0060] Term explanation:

[0061] 1. Federated large model collaborative training: A technical paradigm in which multiple clients and central servers collaboratively train a large global model without sharing raw data.

[0062] 2. Heterogeneous clients: The state of client devices participating in federated learning, which differ in hardware capability, network condition, data modality or local model structure.

[0063] 3. Lightweight model: A neural network structure deployed on the client with significantly smaller parameter scale than the global large model.

[0064] 4. Heterogeneous data: The phenomenon of local data held by each client in federated learning, which differs in modality type, sample distribution, feature space or label structure.

[0065] 5. Gradient-relevance adaptive differential privacy: A privacy protection mechanism that dynamically adjusts the noise intensity according to the relevance of the model parameter level to the target task semantics.

[0066] 6. Modal perception weighted consensus fusion: A technique that dynamically allocates fusion weights when aggregating models on the server side of federated learning, according to the matching degree of client data modality and task (modal perception) and the reliability of its prediction output (confidence).

[0067] 7. Confidence: An evaluation index of the reliability of model prediction results, usually quantified by the distribution characteristics of output layer logits.

[0068] The core of the invention is to construct a large model federated collaborative training process for multi-modal heterogeneous clients: each client first uses a lightweight heterogeneous model to complete local training based on local private data; then, before uploading the model update, adaptive differential privacy noise is added according to the importance of each layer to the task; then, upload the processed model update along with the modal identifier to the central server; after receiving, the central server weights and fuses the knowledge of each client according to the modal matching degree and prediction confidence, generating an updated global large model; finally, the central server selectively distributes the representation layer parameters of the model to participating clients for initializing the next round of training.

[0069] The corresponding model framework is as followsFigure 1 As shown. The function of each part is analyzed in detail below:

[0070] S1. Each client receives the model initialization parameters sent by the central server, independently trains the heterogeneous lightweight model deployed on the client based on local private data, and obtains local model updates.

[0071] In this embodiment, each participating client deploys lightweight models with different structures for local training based on its own device capabilities and data modality. The parameter size of these lightweight models is significantly smaller than that of the global large model, making them suitable for resource-constrained environments such as mobile devices, medical terminals, or edge computing nodes, thus avoiding excessive computational load caused by model isomorphism. After deploying the lightweight models with different structures, training is conducted based on the local private dataset. Perform multiple rounds of local stochastic gradient descent (SGD) training, and save the local model parameters after training is complete. ,in, For the federated learning of global communication rounds, This represents the number of local iterations.

[0072] S2. During local training, each client applies adaptive differential privacy noise to the local model update, where the noise intensity is dynamically adjusted according to the hierarchical functional importance of the model.

[0073] In this embodiment, when the client uploads local model update information to the central server, a gradient-correlation adaptive differential privacy mechanism is adopted. Before the client uploads the local model update gradient, the noise intensity is dynamically adjusted according to the functional importance of the model level.

[0074] The functional importance of a model hierarchy is measured by the mean of its gradient magnitudes, calculated as follows:

[0075] ,

[0076] In the formula, The importance of functions at each model level. For local training steps, For the first Layer The gradient of the step is taken, with a value range of [1, n]; a threshold is set. The mean of the gradient magnitudes across all layers, if ,Should The higher the semantic relevance of the target task at each layer, the higher its functional importance; if ,Should Layer-specific target tasks have low semantic relevance and low functional importance.

[0077] The corresponding noise intensity is further obtained based on the following calculation formula:

[0078] ,

[0079] In the formula, As the reference noise scale, , The higher, The smaller the value, the lower the noise intensity.

[0080] Low noise intensity is added to key layers with strong task relevance (such as the classification head) to preserve semantic integrity, while high noise intensity is added to non-key layers with weak relevance (such as the shallow feature extraction module) to enhance privacy protection. This achieves a fine balance between privacy protection and model utility, avoiding the accuracy loss or privacy leakage caused by traditional static noise.

[0081] The gradient is clipped and noise is added to ensure that the overall performance meets the requirements. -Differential Privacy:

[0082] ,

[0083] In the formula, For the processed first The estimated value of the layer gradient, For the first The original model gradient of the layer, For the first The preset clipping threshold for the layer. For added noise, For noise scale, It is an identity matrix.

[0084] S3. Each client uploads the model update after applying noise and its modal identifier to the central server.

[0085] In this embodiment, the client uploads the gradient and modal identifiers processed by S2, such as "image" and "text", to the central server through an encrypted channel to ensure secure transmission.

[0086] S4. The central server receives model updates and modal identifiers from each client, and integrates the knowledge of each client to update the global big model based on the modality-aware weighted consensus fusion mechanism.

[0087] In this embodiment, after receiving the data (modality, gradient) uploaded by each client, the central server classifies the clients according to their modality and averages the representation layer outputs of models with the same modality:

[0088] ,

[0089] In the formula, This is the output of the model's representation layer. For modality A collection of clients, for federated learning global communication round, the first client in the first global communication round, after completing the local training, obtains the representation layer output of the local model.

[0090] The central server uses a public dataset to let each client model infer the sample and output . The logits output of the client to . At the same time, the confidence weight of each client is calculated:

[0091] ,

[0092] where is the variance of to measure the prediction confidence; is the modal matching factor: if the client modal is consistent with , then , otherwise ;

[0093] and generate weighted soft labels :

[0094] ;

[0095] In addition, a diversity regularization term is introduced to encourage low-confidence clients to retain unique representations. A contrastive learning mechanism is adopted:

[0096] ,

[0097] where denotes the total number of clients participating in the current round of fusion, is the representation layer output of the client , denotes the similarity calculation function, denotes the negative sample set of the client , is the temperature coefficient. The positive sample denotes the client representation consistent with the client modal and high confidence, and the negative sample denotes the client representation different from the client modal or the same modal but low confidence.

[0098] ​By giving higher fusion weight to the client with modal matching and prediction stability, and appropriately reducing the weight and retaining the diversity characteristics of the client with modal irrelevance or low confidence, the interference of false knowledge on the optimization direction of the global model is prevented.

[0099] Finally, the central server end updates the global large model by minimizing the following loss function:

[0100] ,

[0101] In the formula, represents the probability distribution of the soft label; represents the target probability distribution, i.e. the consensus result after multi-client knowledge fusion; is used to measure the difference between two probability distributions, is a diversity regularization term loss, is a regularization coefficient.

[0102] S5, the central server end distributes the updated representation layer parameters of the global large model to the client, which is used for initialization of the next round of local training.

[0103] The central server end distributes the representation layer parameters in the global large model to the client as the basis for initialization of the next round of local training. Form a knowledge closed loop, so that the client can continuously absorb global knowledge and improve the representation ability of the local model.

[0104] In order to clearly show a kind of federated large model knowledge collaborative training method supporting multi-modal heterogeneous client, the embodiment also provides a kind of federated large model knowledge collaborative training system of multi-modal heterogeneous client, the federated large model knowledge collaborative training system is used to realize the federated large model knowledge collaborative training method described, including multiple clients and a central server end;

[0105] The client is used to train locally based on local private data through a heterogeneous lightweight model training module; the adaptive privacy processing module is used to apply differential privacy noise dynamically adjusted according to the importance of model level function to local model updates; the modal identification module is used to identify the data modal of the client, and the secure communication module is used to perform encrypted communication with the central server end;

[0106] The central server end is used to realize weighted consensus fusion based on the modal matching degree and the prediction confidence of the client through the modal perception fusion module; the global model update module is used to update the global large model according to the fusion result; the representation layer distribution module is used to distribute the updated global large model parameters to the client; and the secure communication interface is used to perform encrypted communication with the client.

[0107] In addition, the embodiment further provides a federated large model knowledge collaborative training device supporting multi-modal heterogeneous clients, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the federated large model knowledge collaborative training method supporting multi-modal heterogeneous clients when executing the computer program.

[0108] The specific embodiments described above illustrate the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements, and equivalent replacements made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A method for collaborative training of knowledge in a federated large model that supports multimodal heterogeneous clients, characterized in that, Includes the following steps: Step 1: Each client receives the model initialization parameters from the central server, independently trains the heterogeneous lightweight model deployed on the client based on local private data, and obtains local model updates; Step 2: During local training, each client applies adaptive differential privacy noise to the local model update, where the noise intensity is dynamically adjusted according to the hierarchical functional importance of the model. Step 3: Each client uploads the model update after applying noise and its modal identifier to the central server; Step 4: The central server receives model updates and modality identifiers from each client. Based on a modality-aware weighted consensus fusion mechanism, it fuses the knowledge of each client to update the global model. Specifically, this includes: classifying clients according to their modality identifiers; using a public dataset, obtaining the prediction outputs of each client's model for the public samples; calculating the prediction confidence based on the variance of each client's prediction output; determining the modality matching factor based on the consistency between the client's modality identifier and the task modality in the public dataset; assigning fusion weights to each client based on the prediction confidence and the modality matching factor; performing a weighted average of the prediction outputs of each client based on the fusion weights to generate weighted soft labels; updating the global model parameters with the goal of minimizing the difference between the global model output and the weighted soft labels. The loss function used to update the global model parameters also includes a diversity regularization term, which is constructed through a contrastive learning mechanism to encourage the retention of unique representations of clients with low confidence. The contrastive learning mechanism includes: For any client, its representation layer output is brought closer to the representation layer output of a positive sample client and pushed further away from the representation layer output of a negative sample client. Here, a positive sample client is a client with the same modality and high confidence, and a negative sample client is a client with a different modality or the same modality but low confidence. The calculation formula is as follows: ， In the formula, This indicates the total number of clients participating in this round of integration. For the client The output of the presentation layer, This represents the similarity calculation function. Indicates client The negative sample set, Temperature coefficient, positive sample Indicates with the client Clients with consistent modality and high confidence levels indicate that negative samples Indicates with the client Client representations with different modalities, or with the same modality but low confidence; Step 5: The central server sends the updated global large model's representation layer parameters to the client for initialization of the next round of local training.

2. The federated large model knowledge collaborative training method supporting multimodal heterogeneous clients according to claim 1, characterized in that, In step 1, the heterogeneous lightweight models of each client support multimodal input, including images, text, or time-series data.

3. The federated large model knowledge collaborative training method supporting multimodal heterogeneous clients according to claim 1, characterized in that, In step 2, applying adaptive differential privacy noise to the local model update includes: For model layers with high semantic relevance to the target task, low noise intensity is used to preserve key semantics; For model layers with low semantic relevance to the target task, high noise intensity is used to enhance privacy protection.

4. The federated large model knowledge collaborative training method supporting multimodal heterogeneous clients according to claim 3, characterized in that, The semantic relevance of the target task is measured by the mean of the gradient magnitudes, calculated as follows: ， In the formula, For semantic relevance of the target task, For local training steps, For the first Layer The gradient of the step is in the range [1, n]; a threshold is set. The mean of the gradient magnitudes across all layers, if ,Should High semantic relevance of layers; if ,Should The semantic relevance of the layers is low.

5. The federated large model knowledge collaborative training method supporting multimodal heterogeneous clients according to claim 1, characterized in that, The central server updates the global large model by minimizing the following loss function: ， In the formula, This represents the probability distribution of soft tags; This represents the target probability distribution, i.e., the consensus result after the fusion of knowledge from multiple clients; Used to measure the difference between two probability distributions For the loss of diversity regularization, is the regularization coefficient.

6. A federated large-scale model knowledge collaborative training system with multimodal heterogeneous clients, characterized in that, The federated large model knowledge collaborative training system is used to implement the federated large model knowledge collaborative training method according to any one of claims 1-5, and includes multiple clients and a central server. The client is used to train locally based on local private data through a heterogeneous lightweight model training module; to apply differential privacy noise dynamically adjusted according to the functional importance of the model hierarchy to the local model update through an adaptive privacy processing module; to identify the client's data modality through a modality identification module; and to conduct encrypted communication with the central server through a secure communication module. The central server is used to achieve weighted consensus fusion based on the modality matching degree and prediction confidence of the client through the modality-aware fusion module. The global model update module updates the global model based on the fusion results; the updated global model parameters are sent to the client through the presentation layer distribution module; and encrypted communication with the client is then conducted through the secure communication interface.

7. A federated large model knowledge collaborative training device supporting multimodal heterogeneous clients, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the federated large model knowledge collaborative training method supporting multimodal heterogeneous clients as described in any one of claims 1-5.

Images