Asynchronous modality perception federated aggregation method and system for multi-modal large model

Abstract

The present application relates to the technical field of federated learning, and particularly relates to an asynchronous modality perception federated aggregation method and system for a multi-modal large model. The method comprises the following steps: a central server screens out clients participating in training, and issues current global model parameters to the screened target clients; each target client performs local training based on the global model parameters, generates a parameter increment, and asynchronously submits the parameter increment to the central server while carrying a training start timestamp; the central server calculates a modality perception staleness tolerance threshold, calculates an adaptive staleness weight based on the relationship between staleness and the modality perception staleness tolerance threshold, calculates a corrected global model update amount, dynamically adjusts a server learning rate, and updates the global model parameters; calculates a modality balance deviation between an actual participation ratio and a target participation ratio, reselects the clients until the global model converges, and outputs a multi-modal large language model; and the communication transmission overhead can be reduced, and the model training efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to an asynchronous modality-aware federated aggregation method and system for multimodal large models. Background Technology

[0002] Federated Learning (FL), as a distributed machine learning paradigm, allows for the collaborative training of a global model through an iterative process of local training on the client side and model aggregation on a central server, without requiring the collection of raw data from each participant. This paradigm demonstrates broad application prospects in privacy-sensitive fields such as medical image analysis, financial risk control, and social media content moderation because it can meet data compliance requirements while protecting user privacy. In recent years, deep learning models, represented by Multimodal Large Language Models (MLLM), have experienced rapid development. By fusing information from multiple modalities such as text, images, and videos, they achieve joint reasoning and understanding, demonstrating excellent performance in complex tasks such as visual question answering, image and text retrieval, and disaster information analysis. In practical deployments, high-quality multimodal training data is often scattered across different institutions or terminal devices, and is difficult to collect centrally due to limitations in data privacy regulations and communication bandwidth. Therefore, there is an urgent need to introduce the federated learning paradigm into the training process of multimodal large models.

[0003] In federated training of multimodal large models, different clients may have different modal configurations: some clients only hold text data, some clients have both image and text data, and some clients have missing modalities. This modal heterogeneity leads to significant differences in the local training computational cost for each client—processing image modalities requires additional visual encoders for forward and backward propagation, which is far more computationally intensive than pure text training. In the classic synchronous federated learning framework, all selected clients must complete local training within the same round before global aggregation can be performed. Therefore, slower clients will slow down the overall training progress, forming a serious synchronization waiting bottleneck. Currently, aggregation schemes for federated learning are mainly divided into two categories: synchronous aggregation and asynchronous aggregation. In synchronous aggregation, in the FedAvg algorithm, each client uploads its local model parameters to the server after each round of training. The server then broadcasts the global model after weighted averaging based on the number of samples. In terms of asynchronous aggregation, the FedAsync algorithm allows clients to upload model updates at different speeds. The server immediately aggregates the updates upon receipt without waiting for all clients to complete. It introduces a timestamp-based staleness weight decay mechanism, assigning lower aggregation weights to updates with larger delays, thereby maintaining a certain level of convergence stability in an asynchronous environment.

[0004] However, in the traditional federated training scenario of multimodal large language models, the heterogeneous configuration of different client modalities leads to significant differences in local training computational overhead. Existing synchronous and asynchronous aggregation schemes lack explicit correction mechanisms for differentiated obsolescence biases based on modal characteristics, often resulting in low training efficiency and difficulty in balancing accuracy and efficiency in asynchronous updates. Summary of the Invention

[0005] Based on this, in order to solve the above-mentioned technical problems, an asynchronous modality-aware federated aggregation method and system for multimodal large models is provided, which can reduce the communication transmission overhead across devices and improve the training efficiency of multimodal large model federation on the basis of asynchronous training.

[0006] An asynchronous modality-aware federated aggregation method for large multimodal models, the method comprising:

[0007] The central server calculates the modality-aware scheduling priority score based on the modality configuration tags, historical participation records, and expected training time in the modality computing profile database of each client. Based on the priority score, the central server selects the clients to participate in the training and sends the current global model parameters to the selected target clients.

[0008] Each target client independently performs local training based on the global model parameters and its own modality configuration, generates parameter increments, and asynchronously submits them to the central server along with the training start timestamp.

[0009] After receiving the parameter increment, the central server queries the baseline training time of the modality configuration to which the target client belongs, dynamically calculates the modality-aware staleness tolerance threshold based on the baseline training time, determines the staleness according to the current aggregation time step and the training start timestamp, and calculates the adaptive staleness weight based on the relationship between the staleness and the modality-aware staleness tolerance threshold.

[0010] The central server, based on the adaptive staleness weights, groups the parameter increments by modality configuration labels, calculates a weighted aggregate increment for each modality group, updates the momentum tracker of the corresponding modality using the weighted aggregate increments, and calculates the corrected global model update amount based on the weighted aggregate increments of each modality group, the modality momentum tracking value of the previous time step, and the modality participation ratio weights. At the same time, it dynamically adjusts the server learning rate according to the average staleness of all updating clients in the current aggregation window and updates the global model parameters.

[0011] At the end of each scheduling window containing a predetermined number of consecutive aggregation time steps, the central server calculates the actual participation ratio of each modality category within the scheduling window, calculates the modality balance deviation between the actual participation ratio and the target participation ratio, and corrects the cumulative participation count of each modality category in the next scheduling window based on the modality balance deviation. The corrected cumulative participation count is then fed back to the central server, and clients are re-selected until the global model converges, and the final multimodal large language model is output.

[0012] In one embodiment, the central server calculates a modality-aware scheduling priority score based on each client's modality configuration tags, historical participation records, and expected training time in the modality computation profile database. Based on the priority score, it selects clients to participate in training and sends the current global model parameters to the selected target clients, including:

[0013] The central server reads the modality configuration tags of each client, the expected training time of the modality category to which each client belongs, from the modality computing profile database, and reads the cumulative number of participations of each modality category up to the current aggregation time step, as well as the time step of each client last completing training and submitting an update.

[0014] Based on the read data, the central server calculates a modality-aware scheduling priority score for each candidate client. The priority score is a weighted synthesis of a rare modality compensation factor, an idle time normalization factor, and a random disturbance factor.

[0015] The candidate clients are sorted in descending order of their priority scores, and a predetermined number of clients at the top of the sort are selected as the target clients for this round of training.

[0016] The current global low-rank adaptive parameters are sent to the target clients so that each target client can perform local training based on its own modality configuration.

[0017] In one embodiment, each target client independently performs local training based on the global model parameters and its own modality configuration, generates parameter increments, and asynchronously submits them to the central server along with the training start timestamp, including:

[0018] Each target client receives the global model parameters sent by the central server and loads them into the local multimodal large language model as the initial weights for local training, and records the start timestamp of local training.

[0019] Each target client performs differentiated local training based on its own modality configuration label, calculates the difference between the trained local model parameters and the initial weights, and generates a low-rank adaptive parameter increment.

[0020] Each target client encapsulates the parameter increment, start timestamp, and modal configuration tag into an update data packet and submits it to the central server;

[0021] Each target client records the actual time spent in this local training and reports the actual time spent to the central server to update the expected training time in the modality computing profile database.

[0022] In one embodiment, after receiving the parameter increment, the central server queries the baseline training time of the modality configuration to which the target client belongs, dynamically calculates the modality-aware staleness tolerance threshold based on the baseline training time, determines the staleness based on the current aggregation time step and the training start timestamp, and calculates an adaptive staleness weight based on the relationship between the staleness and the modality-aware staleness tolerance threshold, including:

[0023] The central server receives the update data packet, parses out the parameter increment, start timestamp, and modal configuration tag from it, and records the current aggregation time step;

[0024] The central server queries the modality computing profile database based on the modality configuration tag to find the baseline training time of the modality category to which the target client belongs, as well as the current minimum baseline training time among all modality categories.

[0025] The central server dynamically calculates the modality perception obsolescence tolerance threshold by proportionally expanding the preset basic tolerance threshold based on the ratio of the baseline training time to the minimum baseline training time.

[0026] The central server obtains the obsolescence of the target client update based on the difference between the current aggregation time step and the starting timestamp, and calculates the adaptive obsolescence weight based on the numerical relationship between the obsolescence and the modality-aware obsolescence tolerance threshold.

[0027] In one embodiment, the central server, based on the adaptive stale weights, groups the parameter increments by modality configuration labels, calculates a weighted aggregate increment for each modality group, and updates the momentum tracker for the corresponding modality using the weighted aggregate increment, including:

[0028] The central server receives parameter increments, adaptive stale weights, and modality configuration tags from each target client, and categorizes and groups all parameter increments according to the modality configuration tags to obtain each modality group.

[0029] For each modality group, the parameter increments of each target client within the group are aggregated after being weighted by the adaptive stale weight and the number of local samples, and the weighted aggregate increment of the modality group at the current time step is calculated.

[0030] The momentum tracker of the corresponding mode is updated using the weighted aggregation increment. The modal momentum tracking value saved in the previous time step is fused with the current weighted aggregation increment by performing an exponential moving average according to the preset modal momentum coefficient to generate the modal momentum tracking value of the current time step.

[0031] Based on the total number of valid samples in each modality group within the current aggregation window and the adaptive aging weight, the participation ratio weight of each modality category is calculated.

[0032] In one embodiment, the global model update amount after correction is calculated based on the weighted aggregation increment of each modality group, the modal momentum tracking value of the previous time step, and the modal participation ratio weight. Simultaneously, the server learning rate is dynamically adjusted according to the average staleness of all updating clients within the current aggregation window, and the global model parameters are updated, including:

[0033] The basic update component is obtained by multiplying the weighted aggregate increment of each mode with the corresponding mode participation weight. Then, the modal momentum tracking value of each mode at the previous time step is multiplied by the corresponding mode participation weight and multiplied by the preset correction momentum coefficient to obtain the deviation compensation component.

[0034] The basic update component and the deviation compensation component are merged to obtain the corrected global model update amount.

[0035] The average obsolescence of all client updates within the current aggregation window is calculated and averaged to obtain the average obsolescence. The server learning rate is then dynamically adjusted based on the average obsolescence. When the average obsolescence increases, the server learning rate automatically decreases according to an exponential decay law.

[0036] The corrected global model update amount is multiplied by the dynamically adjusted server learning rate to obtain the global model parameter adjustment amount, and the parameter adjustment amount is added to the current global model parameters to generate the updated global model parameters.

[0037] In one embodiment, at the end of each scheduling window containing a predetermined number of consecutive aggregation time steps, the central server calculates the actual participation ratio of each modality category within the scheduling window, calculates the modality balance deviation between the actual participation ratio and the target participation ratio, and corrects the cumulative participation count of each modality category in the next scheduling window based on the modality balance deviation. The corrected cumulative participation count is then fed back, and clients are re-selected until the global model converges, outputting the final multimodal large language model, including:

[0038] The central server records each round of aggregation time steps in real time. When the cumulative number of aggregated time steps reaches the preset number of steps in the scheduling window, it determines that the current scheduling window ends and triggers window-level modal equilibrium statistics.

[0039] At the end of the scheduling window, the effective number of times each modality category actually participated in aggregation in all aggregation time steps within the scheduling window is counted, and the actual participation ratio of each modality category is calculated.

[0040] The central server retrieves the pre-set participation ratios of each modality target and calculates the modal equilibrium deviation of each modality category based on the actual participation ratios.

[0041] In one embodiment, the cumulative participation count of each modality category in the next scheduling window is corrected based on the modality equalization bias. The corrected cumulative participation count is then fed back to re-select clients until the global model converges, and the final multimodal large language model is output, including:

[0042] Obtain the cumulative participation count of each modality currently stored, calculate the correction value based on the modality balance deviation, and use the correction value to correct the cumulative participation count of each modality category at the start time of the next scheduling window;

[0043] The corrected cumulative participation count is written back to the modal computing profile database, which is used to automatically adjust the scheduling priority of each modal client during the client screening process, and to re-complete the processing of global parameter distribution, local training, asynchronous update upload, stale weight calculation, and momentum correction aggregation.

[0044] After each global model update, the validation accuracy of the global model is continuously monitored. When the fluctuation of the model validation accuracy is less than the preset threshold within multiple consecutive scheduling windows, the global model is determined to have converged, and the trained multimodal large language model is output.

[0045] An asynchronous modality-aware federated aggregation system for large multimodal models, the system comprising a central server and clients that are interconnected and communicate with each other:

[0046] The central server is used to calculate the modality-aware scheduling priority score based on the modality configuration tags, historical participation records, and expected training time in the modality computing profile database of each client, to select clients participating in training based on the priority score, and to send the current global model parameters to the selected target clients.

[0047] Each of the target clients is used to independently perform local training based on the global model parameters and its own modality configuration, generate parameter increments and asynchronously submit them to the central server with the training start timestamp;

[0048] The central server is also used to, after receiving the parameter increment, query the baseline training time of the modality configuration to which the target client belongs, dynamically calculate the modality-aware staleness tolerance threshold based on the baseline training time, determine the staleness according to the current aggregation time step and the training start timestamp, and calculate the adaptive staleness weight based on the relationship between the staleness and the modality-aware staleness tolerance threshold.

[0049] The central server is also used to group the parameter increments by modality configuration labels based on the adaptive stale weights, calculate the weighted aggregate increment for each modality group, update the momentum tracker of the corresponding modality using the weighted aggregate increment, and calculate the corrected global model update amount based on the weighted aggregate increment of each modality group, the modality momentum tracking value of the previous time step, and the modality participation ratio weights. At the same time, it dynamically adjusts the server learning rate according to the average staleness of all updating clients in the current aggregation window and updates the global model parameters.

[0050] At the end of each scheduling window containing a predetermined number of consecutive aggregation time steps, the central server is also used to count the actual participation ratio of each modality category within the scheduling window, calculate the modality balance deviation between the actual participation ratio and the target participation ratio, and correct the cumulative participation count of each modality category in the next scheduling window based on the modality balance deviation. The corrected cumulative participation count is fed back, and clients are re-selected until the global model converges, and the final multimodal large language model is output.

[0051] The aforementioned asynchronous modality-aware federated aggregation method and system for multimodal large models combines client modality configuration labels, historical participation records, and expected training time based on modality computation profiles to comprehensively calculate scheduling priorities. It also introduces a window-level modality balance bias feedback correction mechanism, which dynamically raises the scheduling priority of clients with insufficient participation frequency, ensuring balanced participation across various modalities during long-term training. Employing a mechanism of asynchronous local training on the client side and real-time parameter increment uploads, it eliminates the need to wait for heavy image and text clients to complete local computation, shortening the federated training time for multimodal large models, reducing idle computing power waste on lightweight clients, and improving heterogeneous edge computing capabilities. Overall training throughput efficiency across cross-institutional client clusters; setting staleness tolerance thresholds based on the differences in training time for each modality benchmark, and dynamically allocating aggregation contribution weights to updates with different lag levels through segmented adaptive staleness weights, which can suppress noise interference caused by high-lag updates from lightweight clients; through modality-based weighted aggregation, combined with independent modality momentum trackers and momentum bias correction terms, it can continuously capture the long-term update trends of each modality and explicitly compensate for gradient bias caused by global model version lag in asynchronous staleness updates; at the same time, it adaptively adjusts the server learning rate based on the window average staleness to ensure the convergence stability of the global model in multimodal asynchronous training scenarios. Attached Figure Description

[0052] Figure 1 This is a diagram illustrating the application environment of an asynchronous modality-aware federated aggregation method for multimodal large models in one embodiment.

[0053] Figure 2 This is a flowchart illustrating an asynchronous modality-aware federated aggregation method for a multimodal large model in one embodiment.

[0054] Figure 3 This is a schematic diagram of the overall architecture of an asynchronous modality-aware federated aggregation method and system for multimodal large models in one embodiment;

[0055] Figure 4 This is a schematic diagram of the internal structure of the momentum correction aggregation module in one embodiment;

[0056] Figure 5 This is a schematic diagram illustrating the complete interaction process of the AMAA method in a single-round asynchronous aggregation in one embodiment.

[0057] Figure 6 This is an application flowchart for a cross-hospital multimodal medical question-answering federated fine-tuning scenario in one embodiment;

[0058] Figure 7 This is an application flowchart for a disaster emergency multi-source heterogeneous data federated analysis scenario in one embodiment;

[0059] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] The asynchronous modality-aware federated aggregation method for large multimodal models provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1As shown, the application environment includes interconnected clients 110 and a central server 120. The central server 120 calculates a modality-aware scheduling priority score based on the modality configuration tags, historical participation records, and expected training time in the modality computation profile database of each client 110. Based on the priority score, it selects the clients 110 to participate in training and sends the current global model parameters to the selected target clients 110. Each target client 110 independently performs local training based on the global model parameters and its own modality configuration, generating parameter increments and asynchronously submitting them to the central server 120 along with the training start timestamp. After receiving the parameter increments, the central server 120 queries the baseline training time of the modality configuration of the target client 110, dynamically calculates the modality-aware staleness tolerance threshold based on the baseline training time, determines the staleness based on the current aggregation time step and the training start timestamp, and calculates adaptive staleness weights based on the relationship between staleness and the modality-aware staleness tolerance threshold. The central server 120 then calculates the adaptive staleness weights based on these weights. The parameter increments are grouped by modality configuration labels, and a weighted aggregation increment is calculated for each modality group. The momentum tracker of the corresponding modality is updated using the weighted aggregation increment. The corrected global model update is calculated based on the weighted aggregation increment of each modality group, the modality momentum tracking value of the previous time step, and the modality participation ratio weight. At the same time, the server learning rate is dynamically adjusted according to the average staleness of all updating clients 110 in the current aggregation window, and the global model parameters are updated. At the end of each scheduling window containing a predetermined number of consecutive aggregation time steps, the central server 120 counts the actual participation ratio of each modality category in the scheduling window, calculates the modality equilibrium deviation between the actual participation ratio and the target participation ratio, and corrects the cumulative participation count of each modality category in the next scheduling window based on the modality equilibrium deviation. The corrected cumulative participation count is fed back, and clients 110 are re-selected until the global model converges, and the final multimodal large language model is output.

[0062] This application provides an asynchronous modality-aware federated aggregation method (AMAA) for large multimodal models. The federated training system employs parameter-efficient fine-tuning (PEFT) technology to train the large multimodal language model locally, specifically using the low-rank adaptation (LoRA) method. In each round of asynchronous aggregation, the client... The local training output is the LoRA parameter increment. The server updates the global model by weighted aggregation of parameter increments submitted by each client using the AMAA method. Assume the federated system contains... There are 1 client, and the client set is 1 Each client Holding local datasets and modal configuration ,in Configure a set of all possible modalities (e.g., text only, image only, image + text, etc.); global model parameters are denoted as... , This indicates the aggregated time step on the server side.

[0063] In one embodiment, such as Figure 2 As shown, an asynchronous modality-aware federated aggregation method for large multimodal models is provided, including the following steps:

[0064] Step 202: The central server calculates the modality-aware scheduling priority score based on the modality configuration tags, historical participation records, and expected training time in the modality computing profile database of each client. Based on the priority score, the central server selects the clients to participate in the training and sends the current global model parameters to the selected target clients.

[0065] When the central server initiates the client scheduling process at each aggregation time step, it needs to read the various data required for scheduling priority calculation from the modal computing profile database in order to filter out the target clients.

[0066] In one embodiment, an asynchronous modality-aware federated aggregation method for multimodal large models may further include a process of constructing a modality computation profile and selecting target clients. Specifically, the process includes: a central server reading the modality configuration labels and expected training time of each client's modality category from the modality computation profile database, and reading the cumulative participation count of each modality category up to the current aggregation time step, as well as the time step of each client's last completed training and submitted an update; based on the read data, the central server calculating a modality-aware scheduling priority score for each candidate client, the priority score being a weighted composite of a rare modality compensation factor, an idle time normalization factor, and a random perturbation factor; arranging the candidate clients in descending order of priority score, and selecting a predetermined number of clients at the top of the ranking as target clients for this round of training; and issuing current global low-rank adaptive parameters to the target clients so that each target client can perform local training based on its own modality configuration.

[0067] The central server first queries the modality computation profile database to read the modality configuration tags of each candidate client in the client pool, the expected training time of their respective modality category, and the cumulative participation count for each modality category up to the current aggregation time step, as well as the time step when each client last completed training and submitted an update. The modality configuration tags identify the data modality type held by the client, such as plain text, image text, or plain image. The expected training time is estimated by an exponential moving average of the historical training times of each client under that modality category, reflecting the typical computational cost required for a single step of local training for that modality configuration. The cumulative participation count records the total frequency with which the corresponding modality category is scheduled to participate in aggregation during federated training. The time step when the client last completed training and submitted an update quantifies the idle waiting time of the client since its last participation.

[0068] Among them, the Modality Computational Profile is a fundamental data structure used to quantify the computational overhead characteristics of each modality configuration. For the client... Its modal portrait It includes three dimensions: expected single-step training time The expected value of the parameter update quantity in L2 and modal category labels .

[0069] Expected training time Estimate using the exponential moving average of the client's historical training time; assume the client... In the The actual training time for each participation is The updated expected training time is: (1); where, Indicates client In the Expected training time after each participation. As the smoothing coefficient, in this embodiment, we take... , For the first The actual training time for each iteration; initial value Use the default training time for this modality configuration category.

[0070] For those with the same modal configuration client collection The baseline training time for this modality category is defined as: (2); where, Modal configuration The baseline training time, This represents the number of clients in this modality category; the server periodically updates the baseline training time for each modality category as a reference for subsequent calculation of the obsolescence tolerance threshold.

[0071] Based on the data retrieved from the modality computing profile database, the central server can calculate a modality-aware scheduling priority score for each candidate client. The priority score is a weighted composite of three parts: a rare modality compensation factor, an idle time normalization factor, and a random perturbation factor. The rare modality compensation factor is determined based on the cumulative number of participations in the client's modality category. This rare modality compensation factor is inversely correlated with the cumulative number of participations; that is, the fewer participations a client has in a modality category, the higher its priority compensation. This prevents clients with high computational overhead and slow training speeds from being marginalized due to prolonged lack of scheduling. The idle time normalization factor is obtained by dividing the difference between the current aggregation time step and the client's last completed training time step by the expected training time for the client's modality category. By normalizing the absolute idle time relative to the typical training time of a modality, it ensures comparability in scheduling priority evaluation for clients with different computational capabilities, preventing lightweight clients from continuously monopolizing scheduling resources due to short absolute idle times. The random perturbation factor is generated through uniformly distributed random sampling within a preset interval, introducing randomness into the deterministic priority ranking to prevent the scheduling strategy from getting trapped in local optima. The three factors are multiplied by preset weight coefficients and then linearly added to obtain the final modality-aware scheduling priority score.

[0072] In this embodiment, the scheduling priority takes into account three factors: the first is rare modality compensation, where the client has a higher priority for modality categories that participate less often, thereby gradually leveling out the participation frequency of each modality; the second is idle time normalization, which normalizes the client's idle time by dividing it by its expected training time, ensuring that fast-computing clients are not ignored because of their short absolute idle time; and the third is random perturbation, which prevents local optima caused by completely deterministic scheduling.

[0073] After obtaining the modality-aware scheduling priority scores of each candidate client, the central server sorts all candidate clients in descending order of priority scores. Based on a preset number of concurrent clients or sampling ratio for each round, a predetermined number of clients are selected from the head of the queue as target clients for training in this round. If a selected client fails to submit an update after being triggered for more than a preset timeout threshold, the central server marks it as disconnected and selects a replacement client from the remaining candidate clients based on the same modality-aware scheduling priority scores to ensure effective participation within the current aggregation window.

[0074] After identifying the target clients, the central server distributes the current global low-rank adaptive (LoRA) parameters to each selected target client as the initial weights for local training. Upon receiving the global low-rank adaptive parameters, each target client can independently start local training according to its own modality configuration, generating parameter increments without waiting for other clients to complete training or become ready, thus formally entering the asynchronous local training phase.

[0075] Step 204: Each target client independently performs local training based on the global model parameters and its own modality configuration, generates parameter increments, and asynchronously submits them to the central server along with the training start timestamp.

[0076] After the global low-rank adaptive parameters are distributed, each selected target client can independently conduct local fine-tuning training and asynchronously upload training update data to the central server.

[0077] In one embodiment, the asynchronous modality-aware federated aggregation method for multimodal large models may further include a client-independent training process. Specifically, each target client receives global model parameters from the central server and loads them into its local multimodal large language model as initial weights for local training, and records the start timestamp of the local training. Each target client performs differentiated local training based on its own modality configuration tags, calculates the difference between the trained local model parameters and the initial weights, and generates a low-rank adaptive parameter increment. Each target client encapsulates the parameter increment, start timestamp, and modality configuration tags into an update data packet and submits it to the central server. Each target client records the actual time consumed in this local training and reports the actual time consumed to the central server to update the expected training time in the modality computation profile database.

[0078] After receiving the current global model parameters from the central server, each target client loads these parameters into its local multimodal large language model as the initial weights for this round of local training. Simultaneously, each target client independently records the start timestamp of this local training session. This start timestamp will be used by the central server to calculate the staleness of the client's update, in order to assess the time difference between the global model version on which the update was based and the current global model version.

[0079] After loading the initial weights, each target client performs differentiated local training based on its own modality configuration labels. Specifically, for plain text clients, local training only involves updating text-related modules such as the language model's self-attention projection matrix, resulting in relatively low computational overhead. For image-text clients, local training, in addition to text processing, also requires performing forward propagation of the visual encoder and cross-modal alignment calculations, including image slicing, visual feature extraction, and joint image-text representation learning, leading to significantly higher computational overhead than plain text clients. For pure image clients, local training primarily revolves around the visual encoder. During local training, each target client, based on its local private dataset, uses preset hyperparameters such as batch size, gradient accumulation steps, and local learning rate to perform forward propagation, loss calculation, backpropagation, and parameter updates.

[0080] After local training is complete, each target client extracts the difference between the trained local model parameters and the initial weights to generate low-rank adaptive (LoRA) parameter increments. Due to the use of efficient parameter fine-tuning techniques, the target client only needs to extract and upload the parameter increments in the form of a low-rank matrix, thus significantly reducing communication overhead.

[0081] Subsequently, each target client encapsulates the low-rank adaptive parameter increment, start timestamp, and its own modality configuration tag into an update data packet and submits it asynchronously to the central server. In this embodiment, each target client submits independently immediately after completing local training, without waiting for other clients to complete training or synchronize. Furthermore, each target client records the actual time taken for its local training while submitting the update and reports this actual time to the central server. Upon receiving this actual time, the central server uses it to update the expected training time of the client and the baseline training time of its modality category in the modality computation profile database.

[0082] Step 206: After receiving the parameter increment, the central server queries the baseline training time of the modality configuration of the target client, dynamically calculates the modality-aware staleness tolerance threshold based on the baseline training time, determines the staleness based on the current aggregation time step and the training start timestamp, and calculates the adaptive staleness weight based on the relationship between the staleness and the modality-aware staleness tolerance threshold.

[0083] To address the issue of uniformly penalizing delayed updates caused by differences in computational overhead among clients with different modal configurations in asynchronous federated aggregation, differentiated asynchronous fault tolerance can be achieved by providing a more lenient obsolescence tolerance window for computationally intensive modal configurations.

[0084] In one embodiment, the asynchronous modality-aware federated aggregation method for multimodal large models may further include a process for calculating adaptive staleness weights. Specifically, the process includes: a central server receiving an update data packet, parsing the parameter increment, start timestamp, and modality configuration tag from it, and recording the current aggregation time step; the central server querying the modality computation profile database based on the modality configuration tag for the baseline training time of the target client's modality category, and the current minimum baseline training time among all modality categories; the central server dynamically calculating the modality-aware staleness tolerance threshold by proportionally expanding a preset baseline tolerance threshold based on the ratio of the baseline training time to the minimum baseline training time; and the central server obtaining the staleness of the target client's update based on the difference between the current aggregation time step and the start timestamp, and calculating the adaptive staleness weights based on the numerical relationship between the staleness and the modality-aware staleness tolerance threshold.

[0085] After receiving the update data packets asynchronously submitted by each target client, the central server first parses out the parameter increments generated by the target client, the local training start timestamp, and the modality configuration tags, while simultaneously recording the current aggregation time step on the server side. The start timestamp indicates the version of the global model when the target client begins local training, and the current aggregation time step indicates the global iteration progress when the server receives the update. Together, they form the time reference for subsequent staleness calculations.

[0086] Subsequently, the central server, based on the parsed modality configuration tags, queries the modality computation profile database to obtain the baseline training time for the modality category to which the target client belongs, and simultaneously obtains the current minimum baseline training time among all modality categories. The baseline training time is calculated by averaging the historical expected training times of all clients with the same modality configuration, reflecting the typical computational cost of completing a single-step local training for that modality category. The minimum baseline training time serves as a normalization reference for the difference in computational cost between modalities, used to quantify the relative computational burden of each modality category relative to the fastest modality.

[0087] Based on the baseline training time and the minimum baseline training time obtained from the query, the central server dynamically calculates the modality-aware staleness tolerance threshold. Specifically, the central server uses a preset basic tolerance threshold as the lower limit and scales it proportionally according to the ratio of the baseline training time to the minimum baseline training time for that modality category. Modality categories with higher computational overhead have a higher ratio of baseline training time to minimum baseline training time, and a more lenient modality-aware staleness tolerance threshold; conversely, modality categories with lower computational overhead have a tolerance threshold closer to the basic tolerance threshold, maintaining relatively tight synchronization constraints.

[0088] After determining the modality-aware staleness tolerance threshold, the central server subtracts the current aggregation time step from the local training start timestamp to obtain the staleness of the target client's update. The staleness quantifies the time difference between the global model version on which the parameter increment is based and the current global model version.

[0089] Finally, the central server calculates the adaptive obsolescence weight for the client update based on the numerical relationship between obsolescence and modality-aware obsolescence tolerance threshold. If the obsolescence is within the modality-aware obsolescence tolerance threshold range, the adaptive obsolescence weight decreases linearly as the obsolescence increases, giving newer updates a higher aggregation weight. If the obsolescence exceeds the modality-aware obsolescence tolerance threshold range, the adaptive obsolescence weight is reduced to a preset minimum weight value to prevent extremely obsolete updates from being completely discarded and losing representativeness of the modality category.

[0090] In this embodiment, when the client At server time step Submission based on global model ( When updating the parameters during training, their staleness is defined as... The modality-aware staleness weight proposed in this application does not employ a uniform decay function, but rather dynamically adjusts it based on the calculated profile of the client's modality category. Firstly, the modality-aware tolerance threshold... Defined as: (3); among which, Modal configuration The outdated tolerance threshold, Based on the tolerance threshold, this embodiment takes , For modal expansion coefficients, in this embodiment, we take... , For modality The baseline training time, Minimizes the baseline training time across all modality categories, ensuring that modality configurations with higher computational overhead receive more lenient obsolescence tolerance.

[0091] Based on this, the client At time step The adaptive stale weight is calculated as follows: (4); among which, For the client At time step outdated weights, As for the obsolescence of this update, Configure the corresponding tolerance threshold for its mode. To ensure minimum safety margin, this embodiment takes... This is used to prevent extremely outdated updates from being completely discarded. When the obsolescence level... Within the tolerance range, the weight decreases linearly with age; beyond the tolerance range, it drops to the minimum value.

[0092] Step 208: The central server, based on adaptive stale weights, groups the parameter increments by modality configuration labels, calculates the weighted aggregate increment for each modality group, updates the momentum tracker of the corresponding modality using the weighted aggregate increment, and calculates the corrected global model update amount based on the weighted aggregate increment of each modality group, the modality momentum tracking value of the previous time step, and the modality participation ratio weight. At the same time, the server learning rate is dynamically adjusted according to the average staleness of all updating clients in the current aggregation window, and the global model parameters are updated.

[0093] After the central server obtains the parameter increments, adaptive stale weights and modal configuration tags corresponding to each client, it starts the modal grouping weighted aggregation operation and updates the momentum trackers specific to each modality simultaneously, while solving the participation ratio weights of each modality.

[0094] In one embodiment, the asynchronous modality-aware federated aggregation method for multimodal large models may further include a modality grouping and momentum update process. Specifically, the process includes: a central server receiving parameter increments, adaptive aging weights, and modality configuration labels from each target client; classifying and grouping all parameter increments according to the modality configuration labels to obtain modality groups; for each modality group, aggregating the parameter increments of each target client within the group after weighting with adaptive aging weights and the number of local samples, calculating the weighted aggregated increment of the modality group at the current time step; updating the momentum tracker of the corresponding modality using the weighted aggregated increment; fusing the modality momentum tracking value saved in the previous time step with the current weighted aggregated increment using an exponential moving average based on a preset modality momentum coefficient to generate the modality momentum tracking value for the current time step; and calculating the participation ratio weight of each modality category based on the total effective sample volume of each modality group within the current aggregation window and the adaptive aging weights.

[0095] The central server receives parameter increments, adaptive aging weights, and modality configuration labels from each target client. Based on the modality configuration labels, the central server categorizes and groups all client updates arriving within the current aggregation time step, resulting in modality groups. Each modality group corresponds to a modality configuration category, such as plain text, image-text, or plain image. For each modality group, the central server calculates the weighted aggregation increment for that modality group at the current time step. Specifically, for each client update within the group, its parameter increment is multiplied by the product of the client's adaptive aging weight and the number of local samples, which serves as the client's weighted contribution. The weighted contributions of all clients within the group are summed, and then divided by the sum of the products of the adaptive aging weights and the number of local samples for all clients within the group, yielding the normalized weighted aggregation increment. The number of local samples represents the scale of local training data held by the client, used to balance the impact of clients with different data volumes on the global model.

[0096] After obtaining the weighted aggregated increments for each modality group, the central server updates the momentum tracker for the corresponding modality using these increments. Specifically, for each modality category, the central server reads the momentum tracking value for that modality saved in the previous time step, fuses it with the weighted aggregated increment calculated in the current time step using an exponential moving average based on a preset modal momentum coefficient, and generates the modal momentum tracking value for the current time step. The modal momentum coefficient controls the fusion ratio between historical momentum and the current increment; a value close to 1 indicates that the momentum tracker focuses more on smoothly capturing the long-term trend of the modality's update direction, while a smaller value indicates a greater responsiveness to changes in the current increment. This momentum tracker is maintained independently for each modality, effectively accumulating the continuous update direction deviations of each modality type in an asynchronous environment, providing historical trend data for subsequent global model correction aggregation.

[0097] Simultaneously, the central server calculates the participation proportion weight for each modality category based on the total number of valid samples and adaptive aging weights for each modality group within the current aggregation window. Specifically, for each modality group, the sum of the products of the adaptive aging weights of all clients within the group and the number of local samples is used as the effective participation metric for that modality category. The normalized participation proportion weight is obtained by dividing the effective participation metric of each modality category by the sum of the effective participation metrics of all modality categories. The participation proportion weight satisfies the constraint that the sum is 1, ensuring that the relative contribution of each modality category is proportional to its effective participation within the current aggregation window, preventing a particular modality from dominating the global update direction due to a large number of clients or frequent submissions.

[0098] In this embodiment, for each modal configuration The central server maintains a momentum tracker. This is used to estimate the aggregation deviation direction for this modality category. At time step... Let the set of clients submitting updates be . The modal configuration is as follows The client subset is Then the mode The aggregation increment at this time step is: (5); among which, For modality At time step The weighted aggregate increment, For the client outdated weights, For the client The number of local samples, For the client Based on global model The parameter increments generated during training.

[0099] The update rule for the modal momentum tracker is: (6); among which, For modality At time step Momentum tracking value, For modal momentum coefficients, in this embodiment, we take... , This represents the current aggregation increment for this mode. The momentum tracker can smoothly capture the long-term trend of the update direction for each mode type, providing a basis for subsequent bias correction.

[0100] In one embodiment, the asynchronous modality-aware federated aggregation method for multimodal large models may further include a parameter update process. Specifically, this process includes: multiplying and summing the weighted aggregation increment of each modality with the corresponding modality participation weight to obtain the basic update component; then weighting the modal momentum tracking value of each modality at the previous time step with the corresponding modality participation weight and multiplying by a preset correction momentum coefficient to obtain the deviation compensation component; merging the basic update component and the deviation compensation component to obtain the corrected global model update amount; calculating the staleness of all client updates within the current aggregation window and taking the average value to obtain the average staleness; dynamically adjusting the server learning rate based on the average staleness, whereby the server learning rate automatically decreases according to an exponential decay law when the average staleness increases; multiplying the corrected global model update amount with the dynamically adjusted server learning rate to obtain the global model parameter adjustment amount; and then adding the parameter adjustment amount to the current global model parameters to generate the updated global model parameters.

[0101] The central server first calculates the corrected global model update, which is a weighted sum of two parts: a basic update component and a bias compensation component. The basic update component reflects the real-time update direction of each modality within the current aggregation window. Specifically, it is obtained by multiplying the weighted aggregation increment of each modality group by its corresponding modality participation weight and then summing the results. The basic update component aggregates the latest gradient information of each modality after filtering with outdated weights, forming the main part of the global update. The bias compensation component is used to mitigate the systematic directional bias caused by the client training based on an outdated global model version. Specifically, it is obtained by multiplying the modal momentum tracking value saved by each modality group at the previous time step by its corresponding modality participation weight, then applying a preset correction momentum coefficient for scaling, and finally summing the results.

[0102] In this embodiment, since the modal momentum tracker smoothly accumulates the long-term trends of each modal update direction using an exponential moving average, the deviation compensation component can inject historical trends into the current global update, thereby offsetting the deviation introduced by the old update deviating from the current optimal direction. The central server merges the basic update component and the deviation compensation component to obtain the corrected global model update amount.

[0103] After calculating the corrected global model update amount, the central server calculates the staleness of all clients submitting updates within the current aggregation window and takes the arithmetic mean of the staleness to obtain the average staleness of the current aggregation window. Based on the average staleness, the central server dynamically adjusts the server learning rate. Specifically, when the average staleness increases, it indicates that there are more client updates trained on older global model versions within the current aggregation window, increasing the potential impact of outdated gradient information on the global model. In this case, the server learning rate automatically decreases according to the exponential decay law, meaning the server learning rate is proportional to the negative exponential of the average staleness. When the average staleness is low, the server learning rate is maintained at a relatively high level to fully utilize timely updates and accelerate convergence.

[0104] Finally, the central server multiplies the corrected global model update by the dynamically adjusted server learning rate to obtain the final parameter adjustment applied to the global model. The central server then adds this parameter adjustment to the current global model parameters, generating updated global model parameters. These updated parameters include the current gradient information weighted and aggregated by modality grouping, incorporate bias compensation based on the modal momentum tracker, and are subject to adaptive learning rate adjustment that matches the system's asynchronicity. This effectively suppresses convergence fluctuations in a multimodal heterogeneous asynchronous environment, improving the training stability and final performance of the global model.

[0105] Specifically, in this embodiment, after obtaining the momentum tracking values ​​of each mode, the global model aggregation simultaneously considers the currently updated weighted sum and the historical momentum corrections of each mode. Global aggregation time step The corrected aggregation rule is: (7); among which, For the updated global model parameters, For server-side learning rate, For modality At time step The participation ratio weight, To correct the momentum coefficient, in this embodiment, we take... , This is the modal momentum tracking value from the previous time step.

[0106] Modal participation ratio weight The definition of is: (8); among which, For modality At time step The participation ratio weight, satisfying This weight takes into account the number of clients, local sample size, and stale weights for each modality within the current aggregation window, ensuring that the relative contribution of each modality during aggregation is proportional to its effective participation in the current update batch.

[0107] Momentum correction term Its purpose is to compensate for outdated gradient direction information carried by stale updates. When a client of a certain modality delays submitting updates for an extended period, its momentum tracking value... The continuous directional trend of this modality update is accumulated, and this trend is injected into the global update through the correction term, thereby mitigating the bias introduced by stale updates deviating from the current optimal direction. In this embodiment, an adaptive server learning rate mechanism is further introduced to dynamically adjust the step size based on the average staleness within the current aggregation window: (9); among which, For time steps Adaptive server learning rate, The initial server learning rate, For the attenuation coefficient, in this embodiment, we take... , This represents the average obsolescence of all updating clients within the current aggregation window. When the average obsolescence is high, the server learning rate is automatically reduced to suppress the impact of outdated information on the global model.

[0108] Step 210: At the end of each scheduling window containing a predetermined number of consecutive aggregation time steps, the central server counts the actual participation ratio of each modality category within the scheduling window, calculates the modal balance deviation between the actual participation ratio and the target participation ratio, and corrects the cumulative participation count of each modality category in the next scheduling window based on the modal balance deviation. The corrected cumulative participation count is fed back, and the clients are re-selected until the global model converges, and the final multimodal large language model is output.

[0109] During the multimodal asynchronous federated training iteration, the central server uses a time period containing a preset fixed number of continuous aggregation time steps as a scheduling window. After each scheduling window is completed, the window-level modality equalization correction mechanism is automatically triggered. By statistically analyzing the participation of each modality, calculating the equalization deviation, and feeding back the corrected scheduling parameters, the balanced scheduling of each modality client is achieved. This process is repeated until the model converges.

[0110] In one embodiment, the asynchronous modality-aware federated aggregation method for multimodal large models may further include a process of scheduling window modality balance correction and model convergence determination. The specific process includes: the central server records each aggregation time step in real time; when the cumulative number of aggregated time steps reaches the preset number of steps in the scheduling window, the current scheduling window is determined to end and window-level modality balance statistics are triggered; at the end of the scheduling window, the effective number of times each modality category actually participates in aggregation in all aggregation time steps within the scheduling window is counted, and the actual participation ratio of each modality category is calculated; the central server retrieves the preset target participation ratio of each modality and calculates the modality balance deviation of each modality category in combination with the actual participation ratio.

[0111] During federated training, the central server maintains an aggregated time step counter in real time. The counter increments with each global model update. When the cumulative aggregated time steps reach the preset scheduling window size, the central server determines the end of the current scheduling window and automatically triggers the window-level modality equalization statistical process. The scheduling window size can be preset based on the total number of clients, the number of modality categories, and training stability requirements.

[0112] At the end of the scheduling window, the central server retrospectively counts the effective participation counts of each modality category in aggregation across all aggregation time steps within that window. An effective count refers to the number of times a client within that modality category is selected for scheduling within the window, successfully completes local training, submits an update, and is ultimately included in the global aggregation. Based on these effective counts, the central server calculates the actual participation ratio for each modality category, which is the ratio of the effective participation count of that modality category within the window to the total effective participation count of all modalities within the window. The actual participation ratio objectively reflects the true participation level of each modality category within the current window.

[0113] Subsequently, the central server retrieves the pre-set target participation ratios for each modality. In this embodiment, the target participation ratio is determined based on the proportion of the number of clients in each modality category to the total number of clients. That is, if a modality category has 3 clients and there are a total of 10 clients, then the target participation ratio for that modality is 30%. The central server calculates the difference between the actual participation ratio of each modality category and the corresponding target participation ratio to obtain the modality balance deviation for each modality category. If the deviation is negative, it indicates that the modality category has insufficient participation in the previous window, and its clients are marginalized for a long time. If the deviation is positive, it indicates that the modality category has excessive participation, which may squeeze the participation space of other modalities.

[0114] Based on the modality balance bias, the central server corrects the cumulative participation count of each modality category at the start of the next scheduling window. Specifically, the central server multiplies the modality balance bias by the scheduling window size, maps it to a participation count bias, scales it according to a preset bias correction coefficient, and then adds it to the current cumulative participation count. For modality categories with insufficient participation, the corrected cumulative participation count decreases, resulting in a higher rare modality compensation score for that modality client in subsequent scheduling windows, thus increasing its probability of being selected for scheduling. For modality categories with excessive participation, the corrected cumulative participation count increases, correspondingly reducing its subsequent scheduling priority.

[0115] The central server writes the corrected cumulative participation count back to the modality computation profile database as the update input for historical participation records in the load balancing scheduling step. During the client selection process in the next scheduling window, the modality-aware scheduling priority score directly calls the corrected cumulative participation count to calculate the rare modality compensation item, thereby automatically adjusting the scheduling priority of each modality client. This forms a closed-loop control of aggregation statistics, deviation calculation, count correction, priority adjustment, and client selection, ensuring that the cumulative participation of each modality category gradually approaches the target proportion in the continuous scheduling window, preventing the global model from impairing its generalization ability due to long-term bias towards a specific modality distribution.

[0116] The central server iteratively executes load balancing scheduling, asynchronous local training, modality-aware staleness tolerance, momentum correction aggregation, and window-level modality balancing correction steps. During iteration, the central server continuously monitors the overall performance metrics of the global model on each validation set. When the fluctuation of the overall performance metrics within multiple consecutive scheduling windows is less than a preset convergence threshold, the federated training is deemed to have reached convergence, the iteration terminates, and the final trained global multimodal language model is output. The output model integrates the balanced contributions of each modality category, possessing stable inference capabilities and good generalization performance under heterogeneous modality configurations.

[0117] In this embodiment, at each asynchronous aggregation time step The scheduler needs to select a subset of clients from the available client pool to participate in the next round of local training. This application proposes a modality-aware participation scheduling strategy to ensure that the cumulative participation of each modality category tends to be balanced within consecutive aggregation windows. Define the client. At time step The scheduling priority score is: (10); among which, For the client The scheduling priority score, , , These are the coefficients for the rare mode weight, idle time weight, and random perturbation weight, respectively. In this embodiment, we take... , , , For modality Deadline step The cumulative number of participations, For the client The time step when the last training session was completed, The expected training time configured for this client modality. This is a random disturbance factor.

[0118] To ensure modal equilibrium at the macroscopic level, this application introduces a window-level modal equilibrium constraint mechanism, which will continuously... Each aggregation time step is defined as a scheduling window. At the end of each window, the actual participation rate of each modality category is calculated and compared with the target rate. Let the modality be... The target participation rate is In the scheduling window The actual participation rate within is Then the modal equalization deviation is defined as: (11); among which, For modality In the scheduling window The deviation in the proportion of participation within the organization To determine the target participation ratio for this modality, this embodiment uses the ratio of the number of clients in each modality to the total number of clients. , For window Internal modes The actual participation rate. This deviation will be fed back into the priority calculation of the next scheduling window, specifically by updating the cumulative participation count: (12); among which, For modality The cumulative number of participations at the start of the new scheduling window. The deviation correction coefficient is taken as [value missing] in this embodiment. , To map the proportional deviation to the participation count deviation, when a modality has insufficient participation in the previous window, its cumulative participation count correction value is reduced, which increases the scheduling priority of the client of that modality in subsequent windows, thereby automatically correcting the modality imbalance.

[0119] Furthermore, this application will set a timeout threshold based on the expected training time after the client triggers the event. If the client... Exceeding the trigger time The update has not yet been submitted, among which, The timeout factor is taken as the timeout multiple in this embodiment. The scheduler marks it as out of contact and triggers the selection of a replacement client. The selection of the replacement client also follows the modality-aware priority rules mentioned above.

[0120] In one embodiment, the asynchronous modality-aware federated aggregation method for multimodal large models may further include a model training process, specifically including: obtaining the current cumulative participation count of each modality, calculating a correction value based on the modality equilibrium deviation, and using the correction value to correct the cumulative participation count of each modality category at the start of the next scheduling window; writing the corrected cumulative participation count back to the modality computation profile database for automatically adjusting the scheduling priority of each modality client during the client screening process, and re-completing the processing of global parameter distribution, local training, asynchronous update upload, stale weight calculation, and momentum correction aggregation; after each global model update, continuously monitoring the validation accuracy of the global model, and when the fluctuation amplitude of the model validation accuracy within multiple consecutive scheduling windows is less than a preset threshold, determining that the global model has converged, and outputting the trained multimodal large language model.

[0121] After obtaining the modality equilibrium deviation for each modality category, the central server reads the currently stored cumulative participation count for each modality category from the modality computation profile database. Specifically, it multiplies the modality equilibrium deviation by the scheduling window size and maps it to a participation count deviation, then scales it according to a preset deviation correction coefficient to obtain a correction value for each modality category. The central server adds the correction value to the current cumulative participation count of the corresponding modality to generate the corrected cumulative participation count at the start of the next scheduling window. Specifically, for modality categories with negative modality equilibrium deviations, the correction value is negative, reducing their cumulative participation count; for modality categories with positive modality equilibrium deviations, the correction value is positive, increasing their cumulative participation count.

[0122] The central server can write the corrected cumulative participation count back to the modality computation profile database, directly updating the historical participation records for each modality category. When client filtering is initiated in the next scheduling window, the updated cumulative participation count can be read and substituted into the rare modality compensation term of the modality-aware scheduling priority score to calculate, thereby adjusting the scheduling priority of each modality client in real time. Based on the adjusted scheduling priority, the central server re-filters clients and sends the current global model parameters to the newly selected target clients. After receiving the parameters, the target clients independently execute local training according to their own modality configuration, generating parameter increments and asynchronously submitting them with the training start timestamp. The central server sequentially executes the aging weight calculation process, which includes parsing the update package, querying the baseline training time, calculating the modality-aware aging tolerance threshold, determining the aging degree, and calculating the adaptive aging weights. Then, it executes the momentum correction aggregation process, which includes grouping by modality, calculating the weighted aggregation increment, updating the modality momentum tracker, calculating the participation ratio weight, synthesizing and correcting the global model update amount, dynamically adjusting the server learning rate, and updating the global model parameters, completing a full round of federated aggregation.

[0123] After each global model update, the central server monitors the validation accuracy of the global model on the validation set and records the accuracy changes within consecutive scheduling windows. When the fluctuation range of the model validation accuracy within multiple consecutive scheduling windows is less than a preset threshold, the global model is determined to have converged, training is terminated, and the final multimodal large language model is output.

[0124] In one embodiment, the stale weights employ a piecewise linear decay function based on a tolerance threshold, or a continuous decay strategy based on a Hinge function can be used. (13); among which, This represents the global staleness decay rate. The continuous decay strategy based on the Hinge function eliminates the segmented structure, allowing stale weights to decay smoothly and continuously within a tolerable range, making it suitable for scenarios requiring high smoothness of aggregated weights. Furthermore, stale weights can be combined with the client-side LoRA learning rate adjustment, using a lower effective learning rate for updates with higher staleness, further suppressing the impact of outdated gradients from the client side.

[0125] In one embodiment, when using a modality-level momentum tracker for bias correction, the FedYogi adaptive optimization algorithm can be combined with modality-aware weights. Specifically, second-order momentum estimates are maintained on the server side. The aggregation rule becomes: (14); (15); among which, For aggregate gradient, First-order momentum, The second momentum coefficient, This is a numerically stable term. It utilizes an adaptive learning rate to automatically adjust the update step size for each parameter dimension, making it suitable for scenarios with high-dimensional model parameters and significant differences in the magnitude of gradients across dimensions.

[0126] In one embodiment, using a linear priority function with fixed weights, the scheduling problem can be modeled as a Markov decision process, and the optimal scheduling strategy can be dynamically learned using reinforcement learning. The state space includes the idle time of each client, modal distribution, and current model performance metrics; the action space is the combination of clients to select the next trigger; and the reward function is designed as a negative weighted sum of the improvement in global model validation accuracy after aggregation and the wall clock time. This approach is suitable for scenarios with a large number of clients, complex modal configurations, and high system dynamics, but training the scheduling strategy itself requires additional computational overhead.

[0127] In scenarios with limited network bandwidth, gradient compression techniques can be introduced into the asynchronous aggregation framework. Specifically, before submitting an update, the client increments the LoRA parameter. Apply Top- Sparsification or quantization operations: (16); among which, This is the parameter increment after compression. This indicates that the gradient with the largest magnitude is retained. One element, Indicates use Bits are quantized and encoded, and the server performs dequantization and aggregation after receiving the compressed update; this can further reduce communication overhead and is suitable for deployment environments where network bandwidth between the client and the central server is limited, such as cross-institutional collaborative training or edge device federated learning scenarios.

[0128] In one embodiment, the performance of AMAA (Asynchronous Modality-Aware Aggregation), an asynchronous modality-aware federated aggregation method for multimodal large models, was tested. Table 1 shows the performance comparison results of AMAA with traditional methods FedAvg, FedProx, FedYogi, and FedAdam on the Hateful-Memes dataset in four modal scenarios:

[0129] Table 1. Performance comparison of AMAA and comparative implementations on the Hateful-Memes dataset.

[0130]

[0131] As shown in Table 1, AMAA achieved the best AUC value in Aligned, Missing, and Mix scenarios, with the most significant advantage in the Missing scenario, improving by 1.8 percentage points compared to FedYogi. In the Single scenario, AMAA's AUC was slightly higher than that of the comparative implementation, indicating that its asynchronous aggregation mechanism can still maintain model quality when modal information is limited.

[0132] Table 2 shows the performance comparison results of AMAA with traditional methods FedAvg, FedProx, FedYogi, and FedAdam on the CrisisMMD dataset across four modal scenarios:

[0133] Table 2 Performance comparison of AMAA and comparative implementation methods on the CrisisMMD dataset.

[0134]

[0135] As shown in Table 2, in the CrisisMMD disaster information fusion analysis task, AMAA performed best in both the Aligned and Missing scenarios, with its F1 score in the Missing scenario being 1.8 percentage points higher than FedYogi. In the Single scenario, AMAA's F1 score was 69.0%, slightly lower than FedYogi's 69.2%, with a difference of only 0.2 percentage points, indicating that AMAA can still provide performance close to that of the synchronous aggregation method under extreme single-modal conditions.

[0136] Table 3 shows the performance comparison results of AMAA with traditional methods FedAvg, FedProx, FedYogi, and FedAdam on the VQA-RAD dataset across four modal scenarios:

[0137] Table 3. Performance comparison of AMAA and comparative implementation methods on the VQA-RAD dataset.

[0138]

[0139] As shown in Table 3, on the VQA-RAD medical visual question answering dataset, AMAA outperforms the comparative implementation in accuracy across all three modal scenarios. In the Missing scenario, AMAA improves accuracy by 1.8 percentage points compared to FedYogi, validating the effectiveness of the modal perception staleness tolerance mechanism in medical image delay scenarios.

[0140] In this embodiment, the values ​​of the core parameters are shown in Table 4:

[0141] Table 4 shows the core parameter configuration for this embodiment.

[0142]

[0143] This application presents an asynchronous modality-aware federated aggregation method for large multimodal models. By eliminating the synchronous waiting bottleneck through an asynchronous aggregation framework, lightweight clients do not need to wait for heavyweight clients to complete training. This effect is mainly attributed to the elastic update window provided by the modality-aware staleness tolerance module for computationally intensive modality configurations. By applying bias compensation to stale updates, the negative impact of outdated gradients on model accuracy in asynchronous aggregation is effectively mitigated. As shown in Tables 1 and 2, AMAA achieves an AUC of 71.3% in the Hateful-Memes Missing scenario and an F1 score of 73.9% in the CrisisMMD Missing scenario, representing improvements of 1.8 percentage points compared to FedYogi, thus verifying the effectiveness of the momentum correction mechanism under conditions of incomplete modal information.

[0144] A modality-aware participation scheduling strategy is employed, integrating rare modality compensation, idle time normalization, and random perturbations to determine client participation priority, preventing fast clients from dominating aggregation. When a modality category has insufficient participation within the scheduling window, a bias correction mechanism automatically increases the subsequent priority of clients with that modality, ensuring the global model does not favor a specific modality distribution. By configuring and maintaining an independent momentum tracker for each modality, the long-term trends of each modality's update direction are smoothly captured, and historical trends are injected into the global update by correcting the momentum term. Combined with an adaptive server learning rate mechanism based on average staleness, the update step size is automatically reduced when the system's asynchronicity is high, effectively suppressing the impact of outdated information on the global model and ensuring convergence quality.

[0145] This application employs LoRA parameter fine-tuning technology, where the client only needs to upload low-rank adaptive parameter increments instead of the complete model parameters, significantly reducing communication overhead. Furthermore, the asynchronous aggregation framework reduces the idle time of computing resources occupied by the client while waiting; this efficiency improvement is particularly critical in deployment environments with limited communication bandwidth or expensive client computing resources.

[0146] As shown in Table 3, AMAA outperforms the comparative implementations in accuracy across all three modalities on the VQA-RAD medical visual question answering dataset. Specifically, it achieves an accuracy of 61.6% in the Missing scenario, a 1.8 percentage point improvement over FedYogi. As shown in Table 1, AMAA also demonstrates excellent performance on the Hateful-Memes social media content moderation task; its cross-domain consistency advantage indicates that AMAA's modal perception mechanism has good task generalization capabilities.

[0147] By configuring and calculating independent tolerance thresholds for each modality, computationally intensive modalities obtain a more relaxed latency window, while lightweight modalities maintain tight synchronization constraints. As shown in Table 4, the basic tolerance threshold is set to 3, the modality expansion coefficient is 0.5, and the system dynamically adjusts the upper limit of tolerance based on the ratio of the baseline training time of the modality profile. This differentiated fault-tolerance strategy, compared to FedAsync's unified tolerance scheme, further shortens the training time by approximately 15% without sacrificing model quality. In extreme scenarios with only a single modality, asynchronous aggregation has a higher risk of bias. As shown in Table 2, AMAA achieves an F1 score of 69.0% in the Single scenario of CrisisMMD, only 0.2 percentage points lower than FedYogi's 69.2%, indicating that AMAA can still provide near-synchronous aggregation performance even under conditions of severely limited modality information; this effect is attributed to the guaranteed weight mechanism ( This prevents extremely outdated updates from being completely discarded, and the scheduler's balance constraints maintain the representativeness of the aggregation process.

[0148] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0149] In one embodiment, such as Figure 1 As shown, an asynchronous modality-aware federated aggregation system for large multimodal models is provided, including a central server 120 and clients 110 that are interconnected and communicate with each other.

[0150] The central server 120 is used to calculate the modality-aware scheduling priority score based on the modality configuration tags, historical participation records and expected training time in the modality computing profile database of each client 110, select the clients 110 participating in training based on the priority score, and send the current global model parameters to the selected target clients 110.

[0151] Each target client 110 is used to independently perform local training based on global model parameters and its own modality configuration, generate parameter increments and asynchronously submit them to the central server 120 with the training start timestamp.

[0152] The central server 120 is also used to query the baseline training time of the modality configuration of the target client 110 after receiving the parameter increment, dynamically calculate the modality-aware staleness tolerance threshold based on the baseline training time, determine the staleness based on the current aggregation time step and the training start timestamp, and calculate the adaptive staleness weight based on the relationship between the staleness and the modality-aware staleness tolerance threshold.

[0153] The central server 120 is also used to group parameter increments by modality configuration labels based on adaptive stale weights, calculate weighted aggregate increments for each modality group, update the momentum tracker of the corresponding modality using the weighted aggregate increments, and calculate the corrected global model update amount based on the weighted aggregate increments of each modality group, the modality momentum tracking value of the previous time step, and the modality participation ratio weights. At the same time, it dynamically adjusts the server learning rate according to the average staleness of all updating clients 110 in the current aggregation window and updates the global model parameters.

[0154] At the end of each scheduling window containing a predetermined number of consecutive aggregation time steps, the central server 120 is also used to count the actual participation ratio of each modality category within the scheduling window, calculate the modal balance deviation between the actual participation ratio and the target participation ratio, and correct the cumulative participation count of each modality category in the next scheduling window based on the modal balance deviation. The corrected cumulative participation count is fed back, and the client 110 is re-selected until the global model converges, and the final multimodal large language model is output.

[0155] In one embodiment, the central server 120 is further configured to read the modality configuration labels and expected training time of each client 110 from the modality computing profile database, and to read the cumulative number of participations of each modality category up to the current aggregation time step and the time step of the last time each client 110 completed training and submitted an update; based on the read data, calculate the modality-aware scheduling priority score for each candidate client 110, the priority score being a weighted synthesis of a rare modality compensation factor, an idle time normalization factor, and a random perturbation factor; sort the candidate clients 110 in descending order of priority score, and select a preset number of clients 110 at the top of the ranking as target clients 110 for this round of training; and issue the current global low-rank adaptive parameters to the target clients 110 so that each target client 110 can perform local training based on its own modality configuration.

[0156] In one embodiment, each target client 110 is further configured to receive global model parameters issued by the central server 120 and load them into the local multimodal large language model as initial weights for local training, and record the start timestamp of local training; perform differentiated local training according to its own modality configuration tags, calculate the difference between the trained local model parameters and the initial weights, and generate low-rank adaptive parameter increments; encapsulate the parameter increments, start timestamps, and modality configuration tags into an update data package and submit it to the central server 120; record the actual time consumed in this local training and report the actual time consumed to the central server 120 to update the expected training time in the modality computing profile database.

[0157] In one embodiment, the central server 120 is further configured to receive update data packets, parse parameter increments, start timestamps, and modality configuration tags from them, and record the current aggregation time step; based on the modality configuration tags, query the modality computation profile database for the baseline training time of the modality category to which the target client 110 belongs, and the current minimum baseline training time among all modality categories; based on the ratio of the baseline training time to the minimum baseline training time, proportionally expand the preset basic tolerance threshold and dynamically calculate the modality-aware staleness tolerance threshold; based on the difference between the current aggregation time step and the start timestamp, obtain the staleness of the update of the target client 110, and calculate the adaptive staleness weight based on the numerical relationship between the staleness and the modality-aware staleness tolerance threshold.

[0158] In one embodiment, the central server 120 is further configured to receive parameter increments, adaptive aging weights, and modality configuration labels from each target client 110, classify and group all parameter increments according to the modality configuration labels to obtain modality groups; for each modality group, aggregate the parameter increments of each target client 110 within the group after weighting with adaptive aging weights and the number of local samples, and calculate the weighted aggregate increment of the modality group at the current time step; update the momentum tracker of the corresponding modality using the weighted aggregate increment, and fuse the modality momentum tracking value saved in the previous time step with the current weighted aggregate increment according to a preset modality momentum coefficient using an exponential moving average to generate the modality momentum tracking value at the current time step; and calculate the participation ratio weight of each modality category based on the total number of effective samples and adaptive aging weights of each modality group within the current aggregation window.

[0159] In one embodiment, the central server 120 is further configured to multiply and sum the weighted aggregate increment of each modality with the corresponding modal participation weight to obtain the basic update component; then multiply the modal momentum tracking value of each modality at the previous time step with the corresponding modal participation weight by a preset correction momentum coefficient to obtain the deviation compensation component; merge the basic update component and the deviation compensation component to obtain the corrected global model update amount; calculate the staleness corresponding to the updates of all clients 110 in the current aggregation window and obtain the average staleness; dynamically adjust the server learning rate based on the average staleness, and automatically reduce the server learning rate according to the exponential decay law when the average staleness increases; multiply the corrected global model update amount with the dynamically adjusted server learning rate to obtain the parameter adjustment amount of the global model, and add the parameter adjustment amount to the current global model parameters to generate the updated global model parameters.

[0160] In one embodiment, the central server 120 is also used to record each round of aggregation time steps in real time. When the cumulative number of aggregated time steps reaches the preset number of steps in the scheduling window, it determines that the current scheduling window ends and triggers window-level modal equilibrium statistics. When the scheduling window ends, it counts the effective number of times each modal category actually participates in aggregation in all aggregation time steps within the scheduling window and calculates the actual participation ratio of each modal category. It retrieves the preset target participation ratio of each modality and calculates the modal equilibrium deviation of each modal category in combination with the actual participation ratio.

[0161] In one embodiment, the central server 120 is further configured to obtain the cumulative participation count of each modality currently stored, calculate the correction value in conjunction with the modality equilibrium deviation, and use the correction value to correct the cumulative participation count of each modality category at the start of the next scheduling window; write the corrected cumulative participation count back to the modality calculation profile database, which is used to automatically adjust the scheduling priority of each modality client 110 during the client 110 screening process, and re-complete the processing of global parameter distribution, local training, asynchronous update upload, stale weight calculation, and momentum correction aggregation; after each global model update, continuously monitor the verification accuracy of the global model, and when the fluctuation of the model verification accuracy within multiple consecutive scheduling windows is less than a preset threshold, determine that the global model has converged and output the trained multimodal large language model.

[0162] In another embodiment, an asynchronous modality-aware federated aggregation system for multimodal large models may include three core modules: a modality-aware staleness tolerance module that dynamically sets differentiated latency tolerance thresholds based on the modality computation profiles of each client, enabling computationally intensive modality configurations to obtain larger asynchronous update windows while maintaining tight synchronization constraints for lightweight configurations; a momentum correction aggregation module that explicitly eliminates systematic biases introduced by asynchronous updates by tracking the expected latency of each modality type and applying momentum compensation to stale gradients; and a load balancing scheduling module that coordinates the order and frequency of client participation within each asynchronous aggregation window to ensure balanced representativeness of each modality category during the aggregation process. These three modules work together to significantly shorten the wall-bell time of federated training while ensuring the convergence quality of the global model.

[0163] An asynchronous modality-aware federated aggregation method for large multimodal models is provided, along with the overall system architecture, as follows: Figure 3 As shown, the process revolves around the central aggregation server, which is responsible for receiving model updates from heterogeneous clients and maintaining global model parameters. The modality-aware staleness tolerance module is the first-level processing unit in the entire asynchronous aggregation process. When the client's LoRA parameter increment and timestamp arrive at the central server, the modality-aware staleness tolerance module first queries the modality computation profile database M7 to obtain the baseline training time and tolerance threshold of the modality category to which the client belongs, and then updates the staleness according to the updated parameters. and modality-aware tolerance threshold Calculate adaptive stale weights It provides a more lenient tolerance for delayed updates of computationally intensive modal configurations, while maintaining tight synchronization constraints for lightweight modal configurations. The momentum correction aggregation module receives the weighted parameter increments output by the modality-aware staleness tolerance module, groups and aggregates the contributions of each client according to modality category, and utilizes the modal momentum tracker. Capture the long-term trend of each modal update direction and correct the momentum term. The system compensates for the systematic biases introduced by outdated updates, ultimately generating corrected global model parameters and sending them back to the central server. Within each asynchronous aggregation window, the load balancing scheduling module reads the expected training time and modal labels of each client from the modal computing profile database, and calculates the scheduling priority score by combining three factors: rare modality compensation, idle time normalization, and random perturbation. Based on this, a subset of clients to participate in the next round of training is selected from the image-text client cluster and the plain text client cluster, and the window-level modal equalization bias is adjusted. The feedback is then used in the priority calculation for the next scheduling window. The deviation feedback connection between the momentum correction aggregation module and the load balancing scheduling module constitutes the core closed loop of the system: the modal distribution statistics of the aggregation results are used to correct subsequent scheduling decisions, ensuring that each modal category maintains balanced representativeness in continuous aggregation windows. This closed-loop mechanism is one of the key innovations that distinguishes AMAA from traditional unified asynchronous aggregation methods.

[0164] In this embodiment, the contradiction between synchronous waiting bottlenecks and asynchronous aggregation quality is addressed simultaneously from three dimensions: First, a modality-aware staleness tolerance module is designed to dynamically set differentiated latency tolerance thresholds based on the computational profiles of each client modality, allowing computationally intensive modality configurations to obtain larger asynchronous update windows, while lightweight configurations maintain tight synchronization constraints. Second, a momentum correction aggregation module is proposed to explicitly eliminate the bias introduced by asynchronous updates by tracking the expected latency of each modality type and applying momentum compensation to stale gradients. Third, a load balancing scheduling module is introduced to coordinate the client participation order within each asynchronous aggregation window, ensuring that each modality category maintains balanced representativeness in aggregation. This invention significantly shortens the wall-bell time of federated training while ensuring model convergence quality, enhances the system's robustness to client speed heterogeneity, and provides a feasible technical path for the practical deployment of multimodal large models in privacy-sensitive scenarios.

[0165] In one embodiment, the internal structure of the momentum correction aggregation module is as follows: Figure 4 As shown, Figure 4 The internal processing path and key state variables of the momentum correction aggregation module are illustrated. After receiving the weighted parameter increments from each client in the modality-aware staleness tolerance module, the modality grouping weighter divides the clients into plain text and image-text groups according to their modality configuration labels, and calculates the weighted aggregation increment for each group. The text modal momentum tracker and the image text modal tracker maintain the momentum state of the corresponding modal category, respectively. ,in The modal momentum coefficient is used to smooth and capture the long-term trend of each modality's update direction using an exponential moving average. The modal participation ratio calculator calculates the normalized participation ratio weights based on the effective sample size and stale weights of each modality client within the current aggregation window. ,satisfy The corrected momentum synthesizer combines the weighted aggregated increments of each current mode with the historical momentum tracking values ​​from the previous time step, and outputs the corrected total increment. Among them, the corrected momentum coefficient Control the strength of historical trend compensation. The adaptive learning rate adjuster adjusts the learning rate based on the average staleness of the current aggregation window. Dynamically calculate server learning rate When the system's asynchronous nature is high, the step size is automatically reduced. Simultaneously, the text modality momentum tracker feeds back momentum trend deviation to the adaptive learning rate regulator as an auxiliary signal to detect whether a modality exhibits an anomaly where the update direction deviates from the historical trend. The global model parameter updater ultimately applies the product of the corrected total increment and the adaptive learning rate to the current global model parameters, completing the asynchronous aggregation global update. The feedback connection from the text modality momentum tracker to the adaptive learning rate regulator is the module's anomaly handling mechanism: when the momentum tracking value of a certain modality fluctuates abnormally, the adaptive learning rate will further shrink to suppress the impact of outdated information on the global model.

[0166] In one embodiment, the complete interaction flow of the AMAA method in a single-round asynchronous aggregation is as follows: Figure 5 As shown, the entire process begins with the central aggregation server sending a client scheduling request to the load balancing scheduling module. The load balancing scheduling module first queries the modal computing profile database for the modal tags of each client. and expected training time After the profile database returns the corresponding profile data, the load balancing scheduling module integrates rare modality compensation items. Idle time normalization term and random disturbance term Calculate the scheduling priority score for each client. The selected clients and their priorities are returned after being sorted in descending order of score. The server then sends the current global LoRA parameters to both plain text clients and image-text clients. Plain text clients perform LoRA fine-tuning training locally, involving only updating the self-attention projection matrix of the language model, resulting in lower computational overhead. Image-text clients perform LoRA fine-tuning training locally, involving the visual encoder's forward propagation and text joint processing, including image slicing and cross-modal alignment, resulting in significantly higher computational overhead than plain text clients. Due to the introduction of the asynchronous mechanism, plain text clients and image-text clients do not need to complete training synchronously. After the plain text client completes training first, it immediately increments the LoRA parameters. With timestamp Submissions to the modality-aware stagnation tolerance module do not require waiting for the image-text client to complete. The image-text client submits its updates after a relatively long training period. Upon receiving submissions from each client, the modality-aware stagnation tolerance module queries the image database for the baseline training time for each modality category. With tolerance threshold According to the update status Calculate the adaptive aging weights based on their relationship with the corresponding modal tolerance threshold. :when The time weight decreases linearly with age, when When it drops to the minimum value The modality-aware stagnation-tolerant module passes the weighted parameter increments and modality labels to the momentum correction aggregation module. The momentum correction aggregation module then aggregates the weighted contributions from each client according to modality category, updating the momentum tracker for each modality. It also compensates for staleness bias by correcting the momentum term, while taking into account the average staleness of the current aggregation window. Adjusting the adaptive server learning rate After the momentum correction aggregation module completes aggregation, it returns the corrected global model parameters to the server. The central aggregation server updates the global model and writes the actual training time records of each client in this round to the profile database for subsequent rounds of modal profile updates. The entire time series diagram contains 14 message interactions and 4 self-processing steps (load balancing scheduling module calculates priority ranking, plain text client performs plain text local training, image and text client performs joint image and text training, and modality-aware staleness tolerance module calculates adaptive staleness weights), fully demonstrating the collaborative working mechanism of the three core modules of asynchronous aggregation, modality-aware tolerance, and momentum correction in the AMAA method.

[0167] In one embodiment, the provided asynchronous modality-aware federated aggregation method and system for multimodal large models can be applied to a cross-hospital multimodal medical question-answering federated fine-tuning scenario. In cross-hospital federated learning deployments, multiple medical institutions hold medical data with different modality configurations and need to jointly train a multimodal large language model without sharing the original data. Some hospitals only have text data of medical image reports, while others have both image images and text descriptions, and still others only have image data. Due to significant differences in GPU computing power among hospitals, the computational overhead of image encoding is much higher than that of pure text processing, causing pure text hospitals to wait a long time for image processing to be completed before entering the aggregation stage during synchronous federated training. This embodiment deploys the AMAA method on the VQA-RAD medical question-answering dataset, using LoRA rank... LoRA scaling factor The parameters are configured for efficient fine-tuning, and a basic obsolescence tolerance threshold is set on the server side. Modal momentum coefficient Corrected momentum coefficient Federated training is performed asynchronously by uniting 10 heterogeneous clients with different modalities. The specific process is as follows: Figure 6 As shown:

[0168] M1: Server Initialization and Parameter Distribution: The central aggregation server loads the base weights of the pre-trained multimodal large language model and initializes the global model parameters. Create a modal computation profile database and register modal tags for each client. The server calculates the initial scheduling priority score for each client through the load balancing scheduling module. The current global LoRA parameters are sent to the five selected clients, including two plain text clients and three image text clients.

[0169] M2: Client-side asynchronous local training: After receiving the global LoRA parameters, each client independently performs local fine-tuning based on its own modality configuration. The plain text client only updates the self-attention projection matrix of the language model, with a batch size of 1 and gradient accumulation steps of 16. The image-text client additionally performs forward propagation of the visual encoder and cross-modal alignment calculations, with a single-step training time approximately three times that of the plain text client. Each client uses a learning rate during local training. Local training produces LoRA parameter increments .

[0170] M3: Asynchronous Update Submission and Staleness Labeling: The plain text client completes local training first, and then increments the parameters. With training start timestamp The modality-aware stagnation tolerance module is submitted to the server. The server records the current global time step. Calculate the staleness of the update. and the client The actual training time is updated to the modal computational profiling database.

[0171] M4: Modality-aware staleness weight calculation: The modality-aware staleness tolerance module queries the profiling database to obtain the baseline training time for plain text modality categories. Benchmark training time for image and text modal categories Calculate the tolerance threshold for each mode. The image-text modality, due to its higher computational cost, receives a more lenient tolerance window. Adaptive stale weights are calculated for each modality to be aggregated and updated. When the obsolescence level is within the tolerance range, the weight decreases linearly; when it exceeds the tolerance range, it drops to the minimum value. .

[0172] M5: Momentum Correction Aggregation: The momentum correction aggregation module groups arriving updates by modality label and calculates the plain text modality aggregation increment for each group. Image and text modal aggregation increment Update the momentum tracker for each mode. By correcting the momentum term Compensate for systematic biases introduced by obsolete updates, while also taking into account the average obsoleteness of the current aggregation window. Calculate the adaptive server learning rate .

[0173] M6: Global Model Update and Writeback: The server updates and writes back the corrected global model parameters. Write back to the central aggregation server to synchronously update the actual training time records of each client in the modality computing profiling database, for use in the expected training time of the next round. The exponential moving average is updated.

[0174] M7: Load Balancing Scheduling and Client Triggering: The load balancing scheduling module calculates the actual participation rate of each modality at the end of each aggregation window. , relative to target participation rate Compare the calculated deviations The deviation is fed back into the cumulative participation correction for the next scheduling window. The scheduler integrates rare mode compensation terms. Idle time normalization term and random disturbance term Calculate the priority and send the updated global LoRA parameters to the next round of clients.

[0175] M8: Convergence Detection and Federated Training Termination: The server continuously monitors the overall validation accuracy of the global model across all modal clients. When the validation accuracy fluctuation is less than 0.5% within three consecutive aggregation windows, the federated training is considered converged, and the final global model is output. In the VQA-RAD medical question-answering scenario of this embodiment, compared to the synchronous federated training baseline, the AMAA method shortens the wall clock training time by approximately 35% while maintaining the model accuracy, and controls the deviation of the participation ratio of each modal client within 3%.

[0176] In one embodiment, the provided asynchronous modality-aware federated aggregation method and system for multimodal large models can be applied to a disaster emergency multi-source heterogeneous data federated analysis scenario. In a natural disaster emergency response scenario, multiple frontline rescue stations participate in federated learning through edge computing devices for CrisisMMD disaster multimodal data analysis. Each station is limited by on-site network conditions and equipment performance, resulting in significant differences in modality configuration and computing capabilities: some stations are equipped with portable GPUs that can simultaneously process disaster site images and text tweets, some stations only have mobile terminals that can only process text information, and some stations only upload image data via drones. Furthermore, network bandwidth in disaster areas is severely limited, and communication latency between stations fluctuates drastically. This embodiment adopts the AMAA method, sets the federated aggregation algorithm to FedYogi, and the total number of clients... The number of clients in each round of sampling is 2, and the LoRA rank is... The maximum sequence length of the model is 512, and the scheduling window size is... In the Missing modality scenario, the unavailability of some site image data is simulated with a missing rate of 0.5. The specific process is as follows: Figure 7 As shown:

[0177] M1: Edge Site Registration and Modal Profile Initialization: The central aggregation server assigns client IDs to 10 edge sites, records the modal configuration labels for each site (3 for image-text modalities, 4 for plain text modalities, and 3 for plain image modalities), and initializes the default training time for each modal category. Create a modal computational profiling database. The server loads a pre-trained multimodal large language model, extracts LoRA trainable parameters, and generates initial global parameters. .

[0178] M2: Scheduling Priority Calculation and Site Selection: The load balancing scheduling module reads the modality labels and expected training time of each site and calculates the scheduling priority score. Among them, pure image modality sites receive higher rare modality compensation because they are rare configurations in the federated system. The scheduler selects two sites to participate in this round of training after sorting them in descending order of priority.

[0179] M3: Global Parameter Deployment and Local Asynchronous Training: The server deploys the current global LoRA parameters to the two selected sites. The image-text site performs joint LoRA fine-tuning of the visual encoder and language model, and trains locally on the CrisisMMD dataset with a batch size of 1 and a gradient accumulation step of 16. The plain text site only performs LoRA fine-tuning of the language model's self-attention layer, resulting in significantly faster training speed than the image-text site. After each site completes its independent training, the LoRA parameters are incremented. With timestamp Asynchronous submission.

[0180] M4: Stale Weight Calculation and Modality Tolerance Determination: After receiving submissions from each site, the modality perception staleness tolerance module queries the profile database to obtain the baseline training time for each modality and calculates the modality perception tolerance threshold. In the Missing modality scenario, the training time for pure image sites is the longest due to the full execution of the image encoder, thus obtaining the tolerance threshold. Tolerance threshold for plain text websites Update and calculate outdated weights for each site. Assign a minimum weight to outdated updates that exceed the tolerance window. .

[0181] M5: FedYogi Adaptive Aggregation and Momentum Correction: The momentum correction aggregation module aggregates the weighted parameter increments of each site by modality and updates the momentum tracker for each modality. The aggregation employs the FedYogi adaptive optimization algorithm, with the server maintaining a second-order momentum estimate. Combined with modal participation ratio weights The corrected momentum term generates the global model update. Adaptive server learning rate. Based on the current average age of the aggregation window Dynamically adjust the step size; when network fluctuations in disaster areas cause an increase in average obsolescence, automatically reduce the step size.

[0182] M6: Global Model Update and Communication Compression Writeback: The server will update the corrected global parameters. Update to central storage, and simultaneously record the actual training time for each site in this round to the image database. Considering the bandwidth limitations in the disaster area, the server applies a Top-down constraint to the distributed global LoRA parameters. Sparse compression transmits only the most important parameter updates, compressing the amount of data in a single communication to 30% of the original size.

[0183] M7: Window-level Modal Equalization Correction and Site Rescheduling: At the end of each scheduling window consisting of 20 aggregate time steps, the scheduler calculates the actual participation ratio of each modality. and the target ratio Comparison. For modalities with insufficient participation, adjust the cumulative participation count. Increase the scheduling priority of its subsequent windows. Timeout multiplier. The site is marked as out of contact and a replacement site selection is triggered.

[0184] M8: Convergence Evaluation and Emergency Analysis Model Delivery: After 50 rounds of federated training, the server evaluates the overall performance of the global model on the CrisisMMD disaster classification task. Training terminates when the validation accuracy fluctuation of each modality client is less than 0.5% for three consecutive rounds, and the final emergency analysis model is output. In the disaster emergency scenario of this embodiment, the AMAA method improves accuracy by approximately 2.3% and reduces communication overhead by approximately 40% compared to the synchronous FedAvg baseline in the Missing scenario, while reducing the average idle waiting time of plain text sites by approximately 60%.

[0185] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements an asynchronous modality-aware federated aggregation method for multimodal large models. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0186] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0187] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement steps of an asynchronous modality-aware federated aggregation method for multimodal large models.

[0188] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of an asynchronous modality-aware federated aggregation method for multimodal large models.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0190] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0191] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An asynchronous modality-aware federated aggregation method for large multimodal models, characterized in that, The method includes: The central server calculates the modality-aware scheduling priority score based on the modality configuration tags, historical participation records, and expected training time in the modality computing profile database of each client. Based on the priority score, the central server selects the clients to participate in the training and sends the current global model parameters to the selected target clients. Each target client independently performs local training based on the global model parameters and its own modality configuration, generates parameter increments, and asynchronously submits them to the central server along with the training start timestamp. After receiving the parameter increment, the central server queries the baseline training time of the modality configuration to which the target client belongs, dynamically calculates the modality-aware staleness tolerance threshold based on the baseline training time, determines the staleness according to the current aggregation time step and the training start timestamp, and calculates the adaptive staleness weight based on the relationship between the staleness and the modality-aware staleness tolerance threshold. The central server, based on the adaptive staleness weights, groups the parameter increments by modality configuration labels, calculates a weighted aggregate increment for each modality group, updates the momentum tracker of the corresponding modality using the weighted aggregate increments, and calculates the corrected global model update amount based on the weighted aggregate increments of each modality group, the modality momentum tracking value of the previous time step, and the modality participation ratio weights. At the same time, it dynamically adjusts the server learning rate according to the average staleness of all updating clients in the current aggregation window and updates the global model parameters. At the end of each scheduling window containing a predetermined number of consecutive aggregation time steps, the central server calculates the actual participation ratio of each modality category within the scheduling window, calculates the modality balance deviation between the actual participation ratio and the target participation ratio, and corrects the cumulative participation count of each modality category in the next scheduling window based on the modality balance deviation. The corrected cumulative participation count is then fed back to the central server, and clients are re-selected until the global model converges, and the final multimodal large language model is output.

2. The asynchronous modality-aware federated aggregation method for multimodal large models according to claim 1, characterized in that, The central server calculates a modality-aware scheduling priority score based on each client's modality configuration tags, historical participation records, and expected training time in the modality computation profile database. Based on this priority score, it selects clients to participate in training and sends the current global model parameters, including: The central server reads the modality configuration tags of each client, the expected training time of the modality category to which each client belongs, from the modality computing profile database, and reads the cumulative number of participations of each modality category up to the current aggregation time step, as well as the time step of each client last completing training and submitting an update. Based on the read data, the central server calculates a modality-aware scheduling priority score for each candidate client. The priority score is a weighted synthesis of a rare modality compensation factor, an idle time normalization factor, and a random disturbance factor. The candidate clients are sorted in descending order of their priority scores, and a predetermined number of clients at the top of the sort are selected as the target clients for this round of training. The current global low-rank adaptive parameters are sent to the target clients so that each target client can perform local training based on its own modality configuration.

3. The asynchronous modality-aware federated aggregation method for multimodal large models according to claim 1, characterized in that, Each target client, based on the global model parameters and its own modality configuration, independently performs local training, generates parameter increments, and asynchronously submits them to the central server along with the training start timestamp, including: Each target client receives the global model parameters sent by the central server and loads them into the local multimodal large language model as the initial weights for local training, and records the start timestamp of local training. Each target client performs differentiated local training based on its own modality configuration label, calculates the difference between the trained local model parameters and the initial weights, and generates a low-rank adaptive parameter increment. Each target client encapsulates the parameter increment, start timestamp, and modal configuration tag into an update data packet and submits it to the central server; Each target client records the actual time spent in this local training and reports the actual time spent to the central server to update the expected training time in the modality computing profile database.

4. The asynchronous modality-aware federated aggregation method for multimodal large models according to claim 3, characterized in that, After receiving the parameter increment, the central server queries the baseline training time of the modality configuration to which the target client belongs, dynamically calculates the modality-aware staleness tolerance threshold based on the baseline training time, determines the staleness based on the current aggregation time step and the training start timestamp, and calculates adaptive staleness weights based on the relationship between the staleness and the modality-aware staleness tolerance threshold, including: The central server receives the update data packet, parses out the parameter increment, start timestamp, and modal configuration tag from it, and records the current aggregation time step; The central server queries the modality computing profile database based on the modality configuration tag to find the baseline training time of the modality category to which the target client belongs, as well as the current minimum baseline training time among all modality categories. The central server dynamically calculates the modality perception obsolescence tolerance threshold by proportionally expanding the preset basic tolerance threshold based on the ratio of the baseline training time to the minimum baseline training time. The central server obtains the obsolescence of the target client update based on the difference between the current aggregation time step and the starting timestamp, and calculates the adaptive obsolescence weight based on the numerical relationship between the obsolescence and the modality-aware obsolescence tolerance threshold.

5. The asynchronous modality-aware federated aggregation method for multimodal large models according to claim 1, characterized in that, The central server, based on the adaptive stale weights, groups the parameter increments by modality configuration labels, calculates a weighted aggregate increment for each modality group, and updates the momentum tracker for the corresponding modality using the weighted aggregate increment, including: The central server receives parameter increments, adaptive stale weights, and modality configuration tags from each target client, and categorizes and groups all parameter increments according to the modality configuration tags to obtain each modality group. For each modality group, the parameter increments of each target client within the group are aggregated after being weighted by the adaptive stale weight and the number of local samples, and the weighted aggregate increment of the modality group at the current time step is calculated. The momentum tracker of the corresponding mode is updated using the weighted aggregation increment. The modal momentum tracking value saved in the previous time step is fused with the current weighted aggregation increment by performing an exponential moving average according to the preset modal momentum coefficient to generate the modal momentum tracking value of the current time step. Based on the total number of valid samples in each modality group within the current aggregation window and the adaptive aging weight, the participation ratio weight of each modality category is calculated.

6. The asynchronous modality-aware federated aggregation method for multimodal large models according to claim 5, characterized in that, The global model update is calculated and corrected based on the weighted aggregation increment of each modality group, the modal momentum tracking value of the previous time step, and the modal participation ratio weight. Simultaneously, the server learning rate is dynamically adjusted according to the average staleness of all updating clients within the current aggregation window, and the global model parameters are updated, including: The basic update component is obtained by multiplying the weighted aggregate increment of each mode with the corresponding mode participation weight. Then, the modal momentum tracking value of each mode at the previous time step is multiplied by the corresponding mode participation weight and multiplied by the preset correction momentum coefficient to obtain the deviation compensation component. The basic update component and the deviation compensation component are merged to obtain the corrected global model update amount. The average obsolescence of all client updates within the current aggregation window is calculated and averaged to obtain the average obsolescence. The server learning rate is then dynamically adjusted based on the average obsolescence. When the average obsolescence increases, the server learning rate automatically decreases according to an exponential decay law. The corrected global model update amount is multiplied by the dynamically adjusted server learning rate to obtain the global model parameter adjustment amount, and the parameter adjustment amount is added to the current global model parameters to generate the updated global model parameters.

7. The asynchronous modality-aware federated aggregation method for multimodal large models according to claim 1, characterized in that, At the end of each scheduling window containing a predetermined number of consecutive aggregation time steps, the central server calculates the actual participation ratio of each modality category within the scheduling window, calculates the modality balance deviation between the actual participation ratio and the target participation ratio, and corrects the cumulative participation count of each modality category in the next scheduling window based on the modality balance deviation. The corrected cumulative participation count is then fed back to the central server, and clients are re-selected until the global model converges. The final multimodal large language model is then output, including: The central server records each round of aggregation time steps in real time. When the cumulative number of aggregated time steps reaches the preset number of steps in the scheduling window, it determines that the current scheduling window ends and triggers window-level modal equilibrium statistics. At the end of the scheduling window, the effective number of times each modality category actually participated in aggregation in all aggregation time steps within the scheduling window is counted, and the actual participation ratio of each modality category is calculated. The central server retrieves the pre-set participation ratios of each modality target and calculates the modal equilibrium deviation of each modality category based on the actual participation ratios.

8. The asynchronous modality-aware federated aggregation method for multimodal large models according to claim 7, characterized in that, Based on the modality equalization bias, the cumulative participation count of each modality category in the next scheduling window is corrected. The corrected cumulative participation count is fed back to re-select clients until the global model converges, and the final multimodal large language model is output, including: Obtain the cumulative participation count of each modality currently stored, calculate the correction value based on the modality balance deviation, and use the correction value to correct the cumulative participation count of each modality category at the start time of the next scheduling window; The corrected cumulative participation count is written back to the modal computing profile database, which is used to automatically adjust the scheduling priority of each modal client during the client screening process, and to re-complete the processing of global parameter distribution, local training, asynchronous update upload, old weight calculation, and momentum correction aggregation. After each global model update, the validation accuracy of the global model is continuously monitored. When the fluctuation of the model validation accuracy is less than the preset threshold within multiple consecutive scheduling windows, the global model is determined to have converged, and the trained multimodal large language model is output.

9. An asynchronous modality-aware federated aggregation system for large multimodal models, characterized in that, The system includes a central server and clients that are interconnected and communicate with each other. The central server is used to calculate the modality-aware scheduling priority score based on the modality configuration tags, historical participation records, and expected training time in the modality computing profile database of each client, to select clients participating in training based on the priority score, and to send the current global model parameters to the selected target clients. Each of the target clients is used to independently perform local training based on the global model parameters and its own modality configuration, generate parameter increments and asynchronously submit them to the central server with the training start timestamp; The central server is also used to, after receiving the parameter increment, query the baseline training time of the modality configuration to which the target client belongs, dynamically calculate the modality-aware staleness tolerance threshold based on the baseline training time, determine the staleness according to the current aggregation time step and the training start timestamp, and calculate the adaptive staleness weight based on the relationship between the staleness and the modality-aware staleness tolerance threshold. The central server is also used to group the parameter increments by modality configuration labels based on the adaptive stale weights, calculate the weighted aggregate increment for each modality group, update the momentum tracker of the corresponding modality using the weighted aggregate increment, and calculate the corrected global model update amount based on the weighted aggregate increment of each modality group, the modality momentum tracking value of the previous time step, and the modality participation ratio weights. At the same time, it dynamically adjusts the server learning rate according to the average staleness of all updating clients in the current aggregation window and updates the global model parameters. At the end of each scheduling window containing a predetermined number of consecutive aggregation time steps, the central server is also used to count the actual participation ratio of each modality category within the scheduling window, calculate the modality balance deviation between the actual participation ratio and the target participation ratio, and correct the cumulative participation count of each modality category in the next scheduling window based on the modality balance deviation. The corrected cumulative participation count is fed back, and clients are re-selected until the global model converges, and the final multimodal large language model is output.

Images