Gradient-weighted perception federated aggregation method and system for heterogeneous multi-modal data

Abstract

The present application relates to the technical field of federated learning, and particularly relates to a gradient weighting perception federated aggregation method and system for heterogeneous multi-modal data. The method comprises: a server obtaining modality availability information of each client, calculating an effective modality retention coefficient, calculating a modality completeness score based on the effective modality retention coefficient and the number of local samples, and normalizing the modality completeness score of each client to obtain a modality perception weight of each client; calculating a norm stability score, a direction consistency score and fusing to obtain a gradient quality score of each client; the server generates a round scheduling coefficient, and the modality perception weight and the gradient quality score are weighted and combined by using the round scheduling coefficient to generate a comprehensive aggregation weight of each client; the server generates updated global model parameters and distributes them to each client for the next round of federated training; the pertinence, stability and generalization ability of global aggregation can be improved.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a gradient-weighted sensing federated aggregation method and system for heterogeneous multimodal data. Background Technology

[0002] Federated learning (FL) provides a path for scenario-based applications such as terminal devices, medical institutions, and emergency platforms to balance privacy protection and collaborative modeling by exchanging only model parameters or gradients without aggregating the original data. As tasks such as visual question answering, hate content detection, and disaster information understanding increasingly rely on multimodal information such as images and text, multimodal federated learning is gradually becoming an important direction for combining privacy-preserving computing and intelligent perception. Compared with traditional unimodal federated tasks, clients in multimodal scenarios often exhibit problems such as inconsistent modality combinations, frequent modality loss, large differences in sampling scale, and varying local training stability. This necessitates that the global model integrate cross-client knowledge while avoiding being swayed by updates with insufficient information or high noise levels. Especially in heterogeneous terminal environments, how to efficiently identify more representative client updates and improve the quality of multimodal knowledge fusion while protecting privacy has become a critical issue that urgently needs to be addressed in this field. In existing technologies, the most basic approach is the parameter averaging method, represented by FedAvg. This involves the server weighting and summing local model updates based on the client sample size or participation ratio to complete global aggregation in each round. This method is simple to implement and has controllable communication costs, thus it is widely used as the foundational framework for various federated training systems. In the field of multimodal federated learning, researchers have further expanded on two scenarios: modality consistency and modality inconsistency. One type of work summarizes the task forms, fusion patterns, and heterogeneous sources of multimodal federated learning at the system level, providing a unified analytical framework for subsequent algorithm design. Another type of work addresses the issue of missing modalities on the client side by introducing missing modality compensation, cross-modal feature extraction, or robust fusion mechanisms. For example, temporal modules, local completion, and robust aggregation can mitigate the performance degradation caused by incomplete multimodal inputs. Overall, the main thread of existing solutions remains centered on a unified global model, completing multimodal representation learning locally, and then having the server perform aggregation according to preset rules.

[0003] Traditional aggregation rules primarily rely on sample size as the weighting criterion, making it difficult to distinguish between clients with many samples but a single modality and those with fewer samples but a more complete modality, potentially leading to an underestimation of the value of multimodal information. Secondly, existing methods typically assume that gradient or parameter updates uploaded by different clients are approximately comparable, lacking explicit evaluation of gradient stability, directional consistency, and information content. When local training is affected by noise, missing modalities, or small sample perturbations, the server-side struggles to promptly suppress low-quality updates. Many methods employ fixed aggregation strategies, using the same weighting logic in the early, middle, and convergence phases of training, failing to adapt to the gradual evolution of multimodal models from coarse-grained alignment to fine-grained optimization. Furthermore, when image-based single-modal clients, text-based single-modal clients, and complete multimodal clients coexist, existing solutions often lack dynamic coordination mechanisms for mixed participant structures, easily causing the global model to favor a particular modality or increasing convergence fluctuations. Additionally, when facing different datasets and application domains, existing robust mechanisms often emphasize local compensation but are insufficient to uniformly measure the true contributions of various clients from the server-side, thus affecting model generalization ability and training fairness.

[0004] Therefore, traditional federated aggregation methods fail to effectively distinguish the value of modal information because they only weight based on sample size. Furthermore, they lack dynamic evaluation and coordination mechanisms for gradient quality and training phase evolution. Consequently, they often fail to accurately reflect the true contributions of heterogeneous multimodal clients, resulting in insufficient global model knowledge fusion and poor convergence stability. Summary of the Invention

[0005] Based on this, in order to solve the above-mentioned technical problems, a gradient-weighted sensing federated aggregation method and system for heterogeneous multimodal data is provided, which can improve the targeting, stability and generalization ability of global aggregation.

[0006] A gradient-weighted, sensed federated aggregation method for heterogeneous multimodal data, the method comprising:

[0007] The server obtains modal availability information for each client, including modal indicator vectors, the missing rate of each modality, and the number of local samples. Based on the modal indicator vectors and the missing rate of each modality, the server calculates an effective modality retention coefficient, and based on the effective modality retention coefficients and the number of local samples, the server calculates a modality integrity score. The server then normalizes the modality integrity scores of each client to obtain the modality perception weight of each client.

[0008] The server receives the parameter increments uploaded by each client after completing local training, calculates the norm stability score based on the deviation between the second norm of the parameter increment and the historical smooth norm, and calculates the direction consistency score based on the direction cosine similarity between the parameter increment of the current round and the parameter increment of the previous round. The server then merges the norm stability score and the direction consistency score to obtain the gradient quality score of each client.

[0009] The server obtains the current training round and the preset total number of training rounds and generates round scheduling coefficients. It then uses the round scheduling coefficients to weight and combine the modality-aware weights and the gradient quality scores to generate a comprehensive aggregate weight for each client.

[0010] The server performs a weighted summation of the parameter increments uploaded by each client based on the comprehensive aggregation weight, generates updated global model parameters, and distributes them to each client for the next round of federated training.

[0011] In one embodiment, the server calculates an effective modality retention coefficient based on the modality indication vector and the missing rate of each modality, calculates a modality integrity score based on the effective modality retention coefficient and the number of local samples, and normalizes the modality integrity score of each client to obtain the modality-aware weight of each client, including:

[0012] The server establishes a binary modality indicator vector for each client, which includes an image modality availability identifier and a text modality availability identifier;

[0013] For each client, the availability flag of each modality is multiplied by the complement of the corresponding modality missing rate and then summed to obtain an accumulated value. Based on the accumulated value and the total number of modalities, the effective modality retention coefficient is obtained.

[0014] The server performs a modality integrity enhancement power operation on the effective modality retention coefficients, and multiplies the result by the logarithmically transformed value of the local sample count to obtain the original modality integrity score.

[0015] The server sums the original modal completeness scores of all participating clients in the current round to obtain a total score, and uses the proportion of each client's original modal completeness score to the total score as the client's modal perception weight.

[0016] In one embodiment, the server receives parameter increments uploaded by each client after completing local training, and calculates a norm stability score based on the deviation between the L2 norm of the parameter increments and the historical smoothing norm, including:

[0017] The server receives parameter increments uploaded by each client after completing local training. The parameter increments are the differences between the model parameters after the client's local training ends and the global model parameters at the beginning of the current training round.

[0018] For each client, the server calculates the L2 norm of the parameter increment in this round to represent the current update magnitude, and performs an exponential moving average process on the L2 norm and the historical smoothing norm to update the historical record; wherein, the exponential moving average process includes: weighting the historical smoothing norm of the previous round with a preset smoothing coefficient, weighting the L2 norm with the complement of the smoothing coefficient, and using the sum of the two as the historical smoothing norm after the current round update;

[0019] The server calculates the absolute difference between the L2 norm and the previous historical smoothing norm, calculates the relative deviation based on the absolute difference and the previous historical smoothing norm, and performs an exponential transformation after taking the negative value of the relative deviation to obtain the norm stability score.

[0020] In one embodiment, the server calculates a directional consistency score based on the directional cosine similarity between the parameter increment of the current round and the parameter increment of the previous round, and fuses the norm stability score and the directional consistency score to obtain the gradient quality score for each client, including:

[0021] For clients that have been sampled in two consecutive rounds, the server calculates the vector dot product of the parameter increment in the current round and the parameter increment in the previous round, calculates the L2 norm of the parameter increment in the current round and the parameter increment in the previous round respectively, and calculates the direction cosine similarity based on the vector dot product and the two L2 norms.

[0022] The server performs a linear mapping on the direction cosine similarity to obtain a direction consistency score.

[0023] The server performs a linear weighted fusion of the norm stability score and the directional consistency score using a preset balance coefficient to obtain the gradient quality score for each client.

[0024] In one embodiment, the method further includes:

[0025] The server determines whether the gradient quality score of each client is lower than a preset quality threshold, or whether the norm stability score deviates from the historical statistical range of the corresponding client by more than a preset multiple standard deviation, or whether the directional consistency score deviates from the historical statistical range of the corresponding client by more than a preset multiple standard deviation.

[0026] When any of the judgment conditions is met, the server marks the client as an abnormal update client and reduces the gradient quality score of the abnormal update client.

[0027] The reduced gradient quality score is used to calculate the overall aggregate weight, thereby reducing the proportion of the abnormal update client in the overall aggregate weight.

[0028] In one embodiment, the server obtains the current training round and the preset total number of training rounds and generates round scheduling coefficients. The server then uses these round scheduling coefficients to weight and combine the modality-aware weights and the gradient quality scores to generate a comprehensive aggregate weight for each client, including:

[0029] The server reads the current training round and the preset total number of training rounds from the round controller;

[0030] The server constructs a round scheduling coefficient based on the current training round and the total number of training rounds, and the round scheduling coefficient decreases as the number of training rounds increases;

[0031] For each client, the server multiplies the modality-aware weights by the round-based scheduling coefficients to obtain a modality-weighted term; multiplies the gradient quality score by the complement of the round-based scheduling coefficients to obtain a quality-weighted term; and adds the modality-weighted term and the quality-weighted term to obtain the original comprehensive score.

[0032] The server sums the original comprehensive scores of all participating clients in the current round to obtain the total score, and uses the proportion of each client's original comprehensive score to the total score as the client's comprehensive aggregation weight.

[0033] In one embodiment, the server performs a weighted summation of the parameter increments uploaded by each client based on the comprehensive aggregation weights, generates updated global model parameters, and distributes them to each client for the next round of federated training, including:

[0034] The server receives the parameter increment uploaded by each of the clients, and the parameter increment is the difference between the low-rank adapter LoRA parameter after the client's local training ends and the initial global LoRA parameter in this round.

[0035] For each client, the server multiplies the parameter increment element by the corresponding comprehensive aggregation weight to obtain the weighted parameter increment, and then sums the weighted parameter increments of all participating clients element by element to obtain the global parameter update amount.

[0036] The server adds the initial global model parameters of this round to the global parameter update amount element by element to obtain the updated global model parameters;

[0037] The server saves the updated global model parameters, the aggregated weights of each client in this round, and the training metrics to form training checkpoints;

[0038] The server distributes the updated global model parameters to the clients participating in the next round of training, and each client loads and starts the next round of local training, forming a federated training closed loop.

[0039] In one embodiment, the method further includes:

[0040] Each client performs multimodal model fine-tuning in the local low-rank adapter LoRA parameter space. The optimization objectives of the client's local training include a task loss term and a proximal regularization term. The proximal regularization term is the L2 distance between the client's local parameters and the global parameters issued by the server multiplied by the proximal regularization coefficient.

[0041] Each client uses a cosine annealing learning rate scheduling strategy to control the local update magnitude. The cosine annealing learning rate scheduling strategy includes a warm-up phase and a decay phase. During the warm-up phase, the local learning rate linearly increases from the initial value to a preset peak value. During the decay phase, the local learning rate gradually decays from the peak value to the minimum learning rate according to a cosine curve.

[0042] Each client only uploads the LoRA parameters of the low-rank adapter incrementally to the server.

[0043] A gradient-weighted, aware federated aggregation system for heterogeneous multimodal data, the system comprising a server and various heterogeneous clients, wherein:

[0044] The server is configured to acquire modal availability information for each client, the modal availability information including modal indicator vectors, the missing rate of each modality, and the number of local samples; calculate an effective modal retention coefficient based on the modal indicator vectors and the missing rate of each modality, calculate a modal integrity score based on the effective modal retention coefficients and the number of local samples, and normalize the modal integrity scores of each client to obtain the modal awareness weight of each client;

[0045] The server is also used to receive parameter increments uploaded by each client after completing local training, calculate norm stability score based on the deviation between the second norm of the parameter increment and the historical smooth norm, calculate directional consistency score based on the directional cosine similarity between the parameter increment of the current round and the parameter increment of the previous round, and fuse the norm stability score and the directional consistency score to obtain the gradient quality score of each client.

[0046] The server is also used to obtain the current training round and the preset total number of training rounds and generate round scheduling coefficients, and use the round scheduling coefficients to weight and combine the modality-aware weights and the gradient quality scores to generate the comprehensive aggregate weights of each client.

[0047] The server is also used to perform a weighted summation of the parameter increments uploaded by each client based on the comprehensive aggregation weight, generate updated global model parameters, and send them to each client for the next round of federated training.

[0048] The aforementioned gradient-weighted, perceptive federated aggregation method and system for heterogeneous multimodal data transforms the client's modality indicator vector and the missing rate of each modality into continuous modality-perceptive weights. This enables the server to distinguish the contribution differences between clients with complete modalities and those with single modalities or high missing rates, preventing the value of multimodal information from being diluted by the sheer number of samples. It comprehensively evaluates the reliability of client updates from two dimensions: norm stability and directional consistency. This allows it to identify and suppress low-quality updates caused by noise, missing modalities, or small sample perturbations, enhancing the robustness of global aggregation. By dynamically weighting and combining the modality-perceptive weights and gradient quality scores through round scheduling coefficients, it achieves an adaptive aggregation strategy that prioritizes modality coverage in the early stages of training and update quality in the later stages. Based on the comprehensive aggregation weights, it completes global parameter weighted aggregation and distributes the results to form a closed-loop iteration. This allows the global model to simultaneously integrate modality information value and gradient update quality, effectively improving task accuracy, convergence efficiency, and training fairness among clients in heterogeneous multimodal federated learning scenarios. Attached Figure Description

[0049] Figure 1 This is a diagram illustrating the application environment of a gradient-weighted, sensed federated aggregation method for heterogeneous multimodal data in one embodiment.

[0050] Figure 2 This is a flowchart illustrating a gradient-weighted, sensed federated aggregation method for heterogeneous multimodal data in one embodiment.

[0051] Figure 3 This is a schematic diagram of the overall structure of a gradient-weighted, sensed federated aggregation method for heterogeneous multimodal data in one embodiment;

[0052] Figure 4 This is a schematic diagram of the gradient quality estimation module structure in one embodiment;

[0053] Figure 5 This is a schematic diagram of the core interaction timing in a single round of federated training in one embodiment;

[0054] Figure 6 This is a flowchart of a multi-hospital collaborative medical imaging question-and-answer federated diagnosis and treatment scenario in one embodiment;

[0055] Figure 7 This is a flowchart illustrating a scenario of joint disaster assessment using images and text from multiple agencies, as shown in one embodiment.

[0056] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] The gradient-weighted, sensed federated aggregation method for heterogeneous multimodal data provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the application environment includes interconnected servers 110 and various heterogeneous clients 120. Server 110 can obtain modality availability information from each client 120, including modality indicator vectors, the missing rate of each modality, and the number of local samples. Based on the modality indicator vectors and the missing rate of each modality, it calculates the effective modality retention coefficient; based on the effective modality retention coefficient and the number of local samples, it calculates the modality completeness score; and it normalizes the modality completeness scores of each client 120 to obtain the modality-aware weights of each client 120. Server 110 receives parameter increments uploaded by each client 120 after completing local training, calculates the norm stability score based on the deviation between the L2 norm of the parameter increment and the historical smoothing norm, and calculates the norm stability score based on the current round. The direction cosine similarity between the parameter increment and the parameter increment of the previous round is used to calculate the direction consistency score. The norm stability score and the direction consistency score are then fused to obtain the gradient quality score for each client 120. The server 110 obtains the current training round and the preset total number of training rounds and generates round scheduling coefficients. The round scheduling coefficients are used to weight and combine the modality-aware weights and gradient quality scores to generate the comprehensive aggregate weights for each client 120. The server 110 performs a weighted summation of the parameter increments uploaded by each client 120 based on the comprehensive aggregate weights to generate updated global model parameters and distributes them to each client 120 for the next round of federated training.

[0059] In one embodiment, a gradient-weighted, aware federated aggregation method (GWMA) for heterogeneous multimodal data is provided. Deployed in a federated training system consisting of a server and multiple clients, it is designed for task environments where image and text modalities coexist but are distributed inconsistently across different clients. It is particularly suitable for scenarios involving complete bimodal clients, image monomodal clients, text monomodal clients, and clients with partially missing modalities. The GWMA method no longer determines aggregation strength solely based on sample size; instead, it integrates client modality completeness, gradient quality, and round scheduling into the server-side decision. Specifically, this application first constructs a modality availability encoding and completeness scoring mechanism to quantify the cross-modal knowledge potential that each client can provide in the current round. Then, after local training, it extracts the norm stability and directional consistency of gradient updates to form a gradient quality score, used to identify training noise, offsets caused by missing modalities, and unstable updates from clients with small sample sizes. Finally, the server dynamically adjusts the ratio between modality weights and gradient weights according to the training rounds and completes global aggregation and distribution within the low-rank adapter (LoRA) parameter space. The GWMA method can prioritize the absorption of more complete client knowledge in the early stages of training, and further emphasize update quality and convergence stability in the later stages of training, thereby taking into account the multimodal expressiveness of the global model, training fairness, and aggregation robustness within communication rounds.

[0060] In one embodiment, such as Figure 2 As shown, a gradient-weighted, aware federated aggregation method for heterogeneous multimodal data is provided, including the following steps:

[0061] Step 202: The server obtains modal availability information for each client, including modal indicator vectors, the missing rate of each modality, and the number of local samples. Based on the modal indicator vectors and the missing rate of each modality, the server calculates the effective modal retention coefficient, and based on the effective modal retention coefficient and the number of local samples, the server calculates the modal integrity score, and normalizes the modal integrity score of each client to obtain the modal perception weight of each client.

[0062] During the server initialization phase, a modal state description can be established for each client, and combined with the configurations in the federated data scenario such as aligned (standard multimodal scenario), missing (missing modal scenario), mix (mixed modal scenario), and single (single modal client) to form a unified scoring input.

[0063] In one embodiment, a gradient-weighted sensing federated aggregation method for heterogeneous multimodal data may further include a modality completeness assessment process. Specifically, the process includes: the server establishing a binary modality indicator vector for each client, the binary modality indicator vector including an image modality availability identifier and a text modality availability identifier; for each client, multiplying the availability identifier of each modality by the complement of the corresponding modality missing rate and summing the results to obtain an accumulated value; obtaining an effective modality retention coefficient based on the accumulated value and the total number of modalities; the server performing a modality completeness enhancement exponentiation operation on the effective modality retention coefficient and multiplying the result by the logarithmically transformed value of the local sample quantity to obtain the original modality completeness score; the server summing the original modality completeness scores of all participating clients in the current round to obtain a total score, using the proportion of each client's original modality completeness score to the total score as the client's modality sensing weight.

[0064] During the aggregation phase of each round of federated training, the server performs modal integrity assessment on each participating client to generate modal awareness weights for each client. This modal integrity assessment includes modal availability encoding, calculation of effective modal retention coefficients, calculation of raw modal integrity scores, and normalization of modal awareness weights.

[0065] Specifically, the server creates a binary modality indicator vector for each client participating in the training round. This vector contains two components: an image modality availability flag and a text modality availability flag. When a client has image modality data available for this training round, the image modality availability flag is set to 1; when the client has no image modality data or the image modality data is completely unavailable, the image modality availability flag is set to 0. Similarly, when a client has text modality data available for this training round, the text modality availability flag is set to 1; otherwise, it is set to 0.

[0066] After establishing the binary modality indicator vector, the server further obtains the missing rate for each client in the image and text modalities. The missing rate represents the proportion of missing valid samples in a given modality to the total number of samples that should exist in that modality, ranging from 0 to 1. The missing rate is 0 when all data for a modality is complete, and the more severe the data loss in a modality, the closer the missing rate is to 1. For each client, the server multiplies the availability flag of each modality by the complement of the corresponding modality's missing rate, and then sums the products of all modalities to obtain a cumulative value. The server divides the cumulative value by the total number of modalities to obtain the effective modality retention coefficient for that client.

[0067] After obtaining the effective modality retention coefficients for each client, the server further calculates the raw modality integrity score by combining the number of local samples for that client. The number of local samples is the total number of samples that the client can use for training in this round. Specifically, the server first performs a modality integrity enhancement power operation on the effective modality retention coefficients, that is, it raises the effective modality retention coefficients to the power of a preset modality integrity enhancement coefficient, obtaining the enhanced modality integrity factor. The modality integrity enhancement coefficient is a preset hyperparameter used to adjust the weight of modality integrity in the score, and can be set according to the importance of modal information in the specific application scenario. At the same time, the server performs a logarithmic transformation on the number of local samples for the client, obtaining the logarithmically transformed sample size value. Subsequently, the server multiplies the enhanced modality integrity factor by the logarithmically transformed sample size value to obtain the raw modality integrity score for that client.

[0068] After calculating the raw modal completeness scores for all participating clients in this round, the server normalizes these raw scores to obtain the modal awareness weight for each client. Specifically, the server sums the raw modal completeness scores of all participating clients in the current round to obtain the total score. The server uses the proportion of each client's raw modal completeness score to the total score as the modal awareness weight for that client.

[0069] In this embodiment, the server can first provide the first Each client establishes a modality indicator vector to characterize whether the image modality and text modality are available for training in the current round. For a bimodal scenario, the modality set can be defined as follows: And order: (1); where, Indicates the first Does each client have an image modality? Indicates the first Does each client have a text modal? This represents the modality availability encoding vector for the client. In the presence of missing modalities, an effective modality retention coefficient is further established by combining the missing rate parameter. (2); where, Indicates the effective modality retention coefficient of the client. Indicates the first A client in modal The missing rate is taken when all modalities are present. When modality loss is severe, a larger value is taken. This represents the total number of modes.

[0070] Based on the above encoding, the server can map the aligned (standard multimodal scene) scene to... The complete state maps the single (monomodal client) scenario to a single component. The state treats missing (missing modal scenario) and mix (mixed modal scenario) scenarios as a continuous state between the two, thereby avoiding the need to maintain isolated rules for different client types in the controller.

[0071] After obtaining the modal encoding, the server can calculate the client's modal contribution potential based on modal integrity and sample carrying capacity. The original modal integrity score for each client is defined as follows: (3); among which, This represents the raw score for client modal integrity. Represents the effective mode retention coefficient. Represents the modal integrity enhancement coefficient. This indicates the number of local samples on the client side. This is used to suppress the masking effect of pure sample size advantage on modality value. The server normalizes the raw scores of all participating clients to obtain the modality weights for the current round: (4); among which, Indicates the first Wheel Client Modality-aware weights, Indicates the first The set of clients sampled for training in each round. Represents the set of elements. The original modal integrity score for each client.

[0072] In this embodiment, It is not directly equivalent to the final aggregate weight, but rather participates in subsequent scheduling as a priori quantity reflecting the potential of multimodal knowledge. For clients with both image and text data, Because it is relatively large, it receives more attention in the early stages of training; for clients with only one modality or a high missing rate, It will be moderately reduced, but not completely eliminated, so as to avoid the global model losing local domain features while maintaining fair participation; through this normalization, modal differences are transformed into continuously adjustable server-side control variables.

[0073] Step 204: The server receives the parameter increments uploaded by each client after completing local training. It calculates the norm stability score based on the deviation between the L2 norm of the parameter increment and the historical smooth norm, and calculates the direction consistency score based on the direction cosine similarity between the parameter increment of the current round and the parameter increment of the previous round. The norm stability score and the direction consistency score are fused to obtain the gradient quality score of each client.

[0074] The server does not rely on the original sample content; instead, it directly utilizes the parameter increments or low-rank adapter (LoRA) increments obtained after each round of local training to construct gradient quality metrics on the server side. Specifically, after receiving the parameter increments uploaded by each client after completing local training, the server performs norm stability analysis on each client to assess whether the client's update magnitude in this round is within a reasonable range, and whether there are gradient surges or abnormal decays caused by noise, outliers, or missing modalities. After calculating the norm stability score, the server further performs directional consistency analysis on each client to measure whether the update direction is consistent between adjacent rounds, thereby identifying directional reversals caused by local training noise, missing modal interference, or local overfitting. Subsequently, the server fuses the norm stability score and the directional consistency score to obtain a comprehensive gradient quality score.

[0075] In one embodiment, a gradient-weighted, aware federated aggregation method for heterogeneous multimodal data may further include a process for calculating norm stability scores. Specifically, the process includes: the server receiving parameter increments uploaded by each client after completing local training; the parameter increment is the difference between the model parameters after the client's local training and the global model parameters at the start of the current training round; for each client, the server calculates the L2 norm of the current parameter increment to characterize the current update magnitude, and performs an exponential moving average process between the L2 norm and the historical smoothing norm to update the historical record; wherein the exponential moving average process includes: weighting the previous historical smoothing norm with a preset smoothing coefficient, weighting the L2 norm with the complement of the smoothing coefficient, and using the sum of the two as the updated historical smoothing norm; the server calculates the absolute difference between the L2 norm and the previous historical smoothing norm, calculates the relative deviation based on the absolute difference and the previous historical smoothing norm, takes the negative value of the relative deviation, and performs an exponential transformation to obtain the norm stability score.

[0076] In each round of federated training, the server first broadcasts the current global model parameters to each selected client. Upon receiving these parameters, each client performs local training using its own multimodal or unimodal data. After completing a preset number of rounds of local training, the client compares its model parameters at the end of local training with the global model parameters received at the start of the current round, calculating the parameter increment. The parameter increment is the difference between the client's model parameters at the end of local training and the global model parameters at the start of the current round—that is, the actual change in model parameters during local training. Each client uploads the parameter increment to the server, which stores it upon receiving it.

[0077] For each client participating in this training round, the server calculates the L2 norm of the uploaded parameter increment vector. The L2 norm is calculated by summing the squares of each element in the parameter increment vector and then taking the square root of the sum. The L2 norm reflects the overall magnitude of the parameter increment for that client in this round; a larger L2 norm indicates a larger adjustment to the model parameters, and a smaller L2 norm indicates a smaller adjustment. The server maintains a historical smoothing norm for each client to record the smoothing trend of the parameter increment L2 norm in past rounds. After calculating the current L2 norm in each round, the server updates the historical smoothing norm using an exponential moving average to include the current round's L2 norm in the historical record while maintaining smooth tracking of historical trends.

[0078] In this embodiment, the exponential moving average processing specifically includes: the server weights the previous round's historical smoothing norm with a preset smoothing coefficient, and weights the current round's L2 norm with the complement of the smoothing coefficient. The sum of these two weights is then used as the updated historical smoothing norm for this round. The complement of the smoothing coefficient is 1 minus the smoothing coefficient. The smoothing coefficient is a preset hyperparameter, ranging from 0 to 1, and can be set according to the specific application scenario to emphasize historical trends and current changes. When the smoothing coefficient is larger, the previous round's historical smoothing norm has a higher proportion in the update result, and the updated historical smoothing norm retains more of the historical trend, showing lower sensitivity to instantaneous changes in the current round and stronger resistance to disturbances.

[0079] By using exponential moving average processing, the server can obtain a smooth estimate of the long-term trend of client update magnitude without storing the L2 norm of all historical rounds, thus providing a stable reference benchmark for judging whether the current round update is abnormal. For clients participating in training for the first time, since there is no historical smoothing norm from the previous round, the server directly uses the L2 norm of the current round as the initial historical smoothing norm, and performs exponential moving average updates in subsequent rounds. After completing the exponential moving average update, the server compares the L2 norm of the parameter increment of the current round with the historical smoothing norm of the previous round to determine whether the current update magnitude deviates from the historical trend. Specifically, the server first calculates the absolute difference between the L2 norm of the current round and the historical smoothing norm of the previous round, i.e., takes the absolute value of the difference. The absolute difference reflects the degree of deviation between the current update magnitude and the historical smoothing trend.

[0080] Subsequently, the server calculates the relative deviation based on the absolute difference and the previous historical smoothing norm. The relative deviation, the ratio of the absolute difference to the previous historical smoothing norm, normalizes the degree of absolute deviation to a proportional value relative to the historical scale. By calculating the relative deviation, the server eliminates the incomparability caused by differences in the overall magnitude of update amplitudes between different clients, enabling norm stability assessment to be performed across clients on a uniform relative scale. To prevent the anomaly of a zero denominator due to a zero previous historical smoothing norm, the server can introduce a very small positive number as a smoothing term when calculating the relative deviation, ensuring numerical stability. After calculating the relative deviation, the server takes a negative value for it and then performs an exponential transformation to obtain the norm stability score. The norm stability score ranges from 0 to 1.

[0081] In this embodiment, in the first In the round, the client Receive global parameters sent by the server After completing local training, the parameters are obtained. Define its local update as: (5); among which, Indicates the first The client in the first Increment of local parameters of the wheel, This represents the model parameters after local training is complete. This represents the global model parameters at the start of this round.

[0082] To characterize the information content and stability of this update, its L2 norm is calculated and smoothed using an exponential moving average: (6); among which, Indicates the client In the The smooth gradient norm of the wheel, Represents the smoothing coefficient. This represents the L2 norm of the current update round.

[0083] Furthermore, the current norm is compared with the historical smoothing norm to obtain a stability score: (7); among which, The norm stability score is represented. To prevent extremely small positive numbers with a denominator of zero, when the client's local training is affected by noise or outliers, its update norm deviates significantly from the historical scale. It will automatically decrease; when the client continuously generates effective updates with stable amplitude. It remains at a high level.

[0084] In one embodiment, a gradient-weighted, aware federated aggregation method for heterogeneous multimodal data may further include a process for obtaining gradient quality scores. Specifically, for clients sampled in two consecutive rounds, the server calculates the vector dot product of the parameter increment in the current round and the parameter increment in the previous round, calculates the L2 norm of both the current and previous round parameter increments, and calculates the direction cosine similarity based on the vector dot product and the two L2 norms. The server then performs a linear mapping of the direction cosine similarity to obtain a direction consistency score. Finally, the server performs a linear weighted fusion of the norm stability score and the direction consistency score using a preset balance coefficient to obtain the gradient quality score for each client.

[0085] Directional consistency analysis requires comparing the parameter increment directions of the client in two adjacent training rounds, therefore it is only applicable to clients that were sampled and participated in training in both the current and previous rounds. For clients meeting the applicable conditions, the server first calculates the dot product between the parameter increment vector of the current round and the parameter increment vector of the previous round. The dot product is calculated by multiplying corresponding elements of the two vectors one by one, and then summing all the products. Simultaneously, the server calculates the L2 norm for both the current and previous round parameter increment vectors. The L2 norm is calculated by summing the squares of each element in the vectors and then taking the square root of the sum. The L2 norm of the current round parameter increment reflects the overall magnitude of the update in this round, and the L2 norm of the previous round parameter increment reflects the overall magnitude of the update in the previous round.

[0086] After obtaining the vector dot product and two L2 norms, the server can calculate the direction cosine similarity. The direction cosine similarity is calculated by dividing the vector dot product by the product of the current round's parameter increment L2 norm and the previous round's parameter increment L2 norm. To prevent calculation errors due to a zero denominator, the server can introduce a very small positive number as a smoothing term to ensure numerical stability. The direction cosine similarity ranges from -1 to 1. When the direction cosine similarity is close to 1, it indicates that the current round's update direction is highly consistent with the previous round, and the client is continuously optimizing along the same direction. When the direction cosine similarity is close to 0, it indicates that the update directions of the two rounds are nearly orthogonal, lacking consistency and potentially leading to training instability. When the direction cosine similarity is close to -1, it indicates that the update directions of the two rounds are opposite, suggesting a reversal in the client's direction and potentially causing training oscillations or local overfitting.

[0087] After calculating the direction cosine similarity, the server linearly maps it from the [-1,1] interval to the [0,1] interval to obtain the direction consistency score. The specific method of linear mapping is as follows: add 1 to the direction cosine similarity and divide by 2, so that the direction consistency score is 1 when the directions are completely consistent, 0 when the directions are completely opposite, and 0.5 when the directions are orthogonal.

[0088] After obtaining the norm stability score and the directional consistency score separately, the server performs a linear weighted fusion of the two scores to obtain a comprehensive gradient quality score. Specifically, the server sets a preset balance coefficient, using the balance coefficient as the weight for the norm stability score and the complement of the balance coefficient as the weight for the directional consistency score. The norm stability score and the directional consistency score are then weighted and summed to obtain the gradient quality score for the client.

[0089] In this embodiment, relying solely on the norm is insufficient to determine whether an update truly facilitates convergence; therefore, the consistency of update directions in adjacent rounds is further compared. For clients sampled in two consecutive rounds, directional consistency is defined as: (8); among which, This represents the update direction cosine similarity. Represents the dot product of vectors. This indicates the previous local update. To prevent extremely small positive numbers from being divided by zero.

[0090] Based on norm stability and directional consistency, a comprehensive gradient quality score is constructed: (9); among which, Indicates the client In the The gradient quality score of the wheel, This represents the balance coefficient between the stability term and the orientation consistency term. Used to linearly map directional consistency to The interval. When the client exhibits frequent direction reversals, gradient oscillations, or offsets driven by missing modes during local training, and The frequency will decrease synchronously. Based on this, the server can distinguish between clients with "rich modalities but high update noise" and clients with "few modalities but stable training," thus providing a more refined decision-making basis for subsequent adaptive scheduling.

[0091] In one embodiment, a gradient-weighted, aware federated aggregation method for heterogeneous multimodal data may further include a process of identifying and adjusting abnormal update clients. Specifically, the server determines whether the gradient quality score of each client is lower than a preset quality threshold, or whether the norm stability score deviates from the historical statistical range of the corresponding client by more than a preset multiple of standard deviation, or whether the directional consistency score deviates from the historical statistical range of the corresponding client by more than a preset multiple of standard deviation. When any of these conditions is met, the server marks the client as an abnormal update client and reduces its gradient quality score. The reduced gradient quality score is used to calculate the overall aggregation weight, thereby reducing the proportion of abnormal update clients in the overall aggregation weight.

[0092] After obtaining the gradient quality scores of each client, the server can compare each client's gradient quality score with a preset quality threshold, compare each client's norm stability score with its historical statistical range, and compare each client's directional consistency score with its historical statistical range. Specifically, when a client's gradient quality score is lower than the preset quality threshold, it indicates that the client's overall update quality is unacceptable, and its update's contribution to the global model is riskier than beneficial. The server maintains a historical sequence of norm stability scores for each client across multiple training rounds, calculating the statistical mean and standard deviation of this historical sequence. When a client's norm stability score in the current round deviates from this historical statistical range by more than a preset multiple of the standard deviation, it indicates an abnormal abrupt change in the client's update magnitude in this round, possibly caused by local data contamination, abnormal training batches, or severe modality loss. Similarly, the server maintains a historical sequence of directional consistency scores for each client, calculating the corresponding statistical mean and standard deviation. When a client's directional consistency score in the current round deviates from this historical statistical range by more than a preset multiple of the standard deviation, it indicates a significant reversal or oscillation in the client's update direction in this round, possibly caused by local overfitting, noise-driven processes, or adversarial perturbations.

[0093] The three criteria mentioned above are independent of each other. The occurrence of any one criterion triggers an anomaly flag, and the server marks the client as an anomalous update client. For the flagged anomalous update client, the server reduces its gradient quality score. Then, the reduced gradient quality score can be used in the subsequent calculation of the overall aggregate weight. Since the overall aggregate weight is obtained by weighting and normalizing the modality-aware weight and the gradient quality score according to the round-based scheduling coefficient, the reduction in the gradient quality score of the anomalous client will directly lower its original overall score, thus significantly reducing its proportion in the normalized overall aggregate weight, or even effectively marginalizing it.

[0094] Step 206: The server obtains the current training round and the preset total number of training rounds and generates round scheduling coefficients. The round scheduling coefficients are used to weight and combine the modality-aware weights and gradient quality scores to generate the comprehensive aggregate weights for each client.

[0095] After obtaining the modality-aware weights and gradient quality scores of each client, the server enters the adaptive aggregation scheduling process to dynamically adjust the proportion of the two types of weights according to the current training stage, and generate the comprehensive aggregate weights for each client.

[0096] In one embodiment, a gradient-weighted, aware federated aggregation method for heterogeneous multimodal data may further include a process for generating comprehensive aggregation weights. Specifically, the process includes: the server reading the current training round and the preset total number of training rounds from the round controller; the server constructing round scheduling coefficients based on the current training round and the total number of training rounds, with the round scheduling coefficients decreasing as the number of training rounds increases; for each client, the server multiplying the modality-aware weights by the round scheduling coefficients to obtain a modality-weighted term; multiplying the gradient quality score by the complement of the round scheduling coefficients to obtain a quality-weighted term; adding the modality-weighted term and the quality-weighted term to obtain the original comprehensive score; and summing the original comprehensive scores of all participating clients in the current round to obtain the total score, using the proportion of each client's original comprehensive score to the total score as the client's comprehensive aggregation weight.

[0097] During the aggregation phase of each round of federated training, the server first reads the current training round and the preset total number of training rounds from its maintained round controller. The current training round indicates the iteration stage of the current round within the entire federated training process; the total number of training rounds is the preset total number of iterations in the training plan when the federated task is initialized, representing the total number of rounds expected to be executed in the entire federated training process. The round controller continuously maintains an incrementing count of the current round throughout the federated training process, incrementing the current round by 1 after each aggregation round is completed, until the total number of training rounds is reached, at which point training terminates.

[0098] After obtaining the current training epoch and the total number of training epochs, the server constructs an epoch scheduling coefficient based on the ratio of the two. The epoch scheduling coefficient is a continuous value between 0 and 1. In the early stages of training, the epoch scheduling coefficient takes a larger value close to 1; as training progresses, the epoch scheduling coefficient gradually decreases; in the later stages of training, the epoch scheduling coefficient takes a smaller value close to 0. The epoch scheduling coefficient is used to control the relative proportions of modality-aware weights and gradient quality scores in the overall aggregation weights: when the epoch scheduling coefficient is large, modality-aware weights dominate in the overall aggregation, and the server prioritizes absorbing client updates with more complete modalities and richer cross-modal knowledge to help the global model quickly establish multimodal representation capabilities; when the epoch scheduling coefficient is small, gradient quality scores dominate in the overall aggregation, and the server emphasizes the stability of the update direction and the reliability of convergence to suppress the perturbation of the global model by low-quality updates.

[0099] After generating the round scheduling coefficients, the server performs a weighted combination operation on a client-by-client basis for each client participating in the current training round, generating the client's original comprehensive score. Specifically, for each client, the server first multiplies the client's modality-aware weights by the round scheduling coefficients to obtain a modality-weighted term. The modality-weighted term reflects the client's potential contribution to global aggregation in the modality information dimension, and its magnitude depends on both the client's own modality completeness and the emphasis on modality information in the current training phase. Subsequently, the server multiplies the client's gradient quality score by the complement of the round scheduling coefficients to obtain a quality-weighted term. The quality-weighted term reflects the reliability of the client's contribution to global aggregation in the gradient update quality dimension. Finally, the server adds the modality-weighted term and the quality-weighted term to obtain the client's original comprehensive score. The original comprehensive score integrates two types of heterogeneous information: the client's modality information value and gradient update quality, and uses the round scheduling coefficients to achieve dynamic weight allocation of these two types of information at different training phases.

[0100] Through the aforementioned weighted combination, the server integrates the modality-aware weights and gradient quality scores to obtain a total score. Then, the server uses the proportion of each client's original composite score to the total score as the composite aggregation weight for that client. After normalization, the sum of the composite aggregation weights of all participating clients is 1, with each weight value ranging from 0 to 1.

[0101] In this embodiment, to reflect the idea of ​​"emphasizing modality in the early stage and quality in the later stage", a scheduling coefficient related to the total number of rounds is constructed: (10); among which, Indicates the first The scheduling coefficient of the round, Indicates the index of the current round. This indicates the total number of training rounds. At the beginning of training... near This indicates a greater emphasis on modal integrity; as training progresses, Gradually decreasing indicates a gradual increase in dependence on gradient quality.

[0102] The server constructs the client's overall aggregate weight accordingly: (11); among which, Indicates the client In the The final aggregate weight of the round, Represents modality-aware weights. Indicates the gradient quality score. This indicates the set of participants in the current round.

[0103] Step 208: The server performs a weighted summation of the parameter increments uploaded by each client based on the comprehensive aggregation weights, generates updated global model parameters, and distributes them to each client for the next round of federated training.

[0104] After generating the overall aggregation weights for each client, the server enters the global parameter aggregation and broadcasting phase. The aggregation operation is performed within the low-rank adapter (LoRA) parameter space, meaning that the objects participating in federated communication and aggregation are not all the parameters of the multimodal large model, but only the low-rank adaptation parameters updated by each client during local fine-tuning.

[0105] In one embodiment, the gradient-weighted aware federated aggregation method for heterogeneous multimodal data may further include parameter updates and iterative training processes. Specifically, the process includes: the server receiving parameter increments uploaded by each client, where the parameter increment is the difference between the low-rank adapter LoRA parameters after local training on the client and the initial global LoRA parameters for the current round; for each client, the server multiplies the parameter increment element-wise by the corresponding comprehensive aggregation weight to obtain the weighted parameter increment, and then sums the weighted parameter increments of all participating clients element-wise to obtain the global parameter update amount; the server adds the initial global model parameters for the current round to the global parameter update amount element-wise to obtain the updated global model parameters; the server saves the updated global model parameters, the comprehensive aggregation weights of each client in the current round, and the training metrics to form training checkpoints; the server distributes the updated global model parameters to the clients participating in the next round of training, and each client loads and starts the next round of local training, forming a federated training closed loop.

[0106] In each round of federated training, the server first broadcasts the current global low-rank adapter parameters to each selected client in that round, serving as the initial parameters for each client's local training. Upon receiving the global LoRA parameters, each client loads them into the low-rank adapter module of its local multimodal model and then performs local fine-tuning training using its locally available multimodal or unimodal data. After completing a preset number of rounds of local training, the client calculates the parameter increment and uploads it to the server. The parameter increment is the difference between the low-rank adapter LoRA parameters after the client's local training and the initial global LoRA parameters for that round; that is, the actual change in the low-rank adapter parameters during the client's local training.

[0107] After obtaining the parameter increments and corresponding aggregate weights for each client, the server performs a weighting operation on a client-by-client basis. Then, it accumulates the weighted results for all clients to generate the global parameter update. Specifically, for each client participating in this training round, the server multiplies each element of that client's parameter increment vector by its corresponding aggregate weight to obtain the client's weighted parameter increment. Subsequently, the server accumulates the weighted parameter increments of all participating clients element-wise; that is, for each element in the parameter increment vector, the server adds the weighted parameter increment values ​​of all clients at that position to obtain the global parameter update at that position.

[0108] After calculating the global parameter update amount, the server adds it element-wise to the initial global model parameters for this round to generate the updated global model parameters. Specifically, for each element position in the global model parameter vector, the server adds the value of the initial global model parameter for this round at that position to the value of the global parameter update amount at that position to obtain the updated global model parameter value at that position. The updated values ​​at all positions together constitute the updated global model parameters.

[0109] After generating the updated global model parameters, the server performs a training checkpoint saving operation to persistently store the key state information of this round of training. The training checkpoints include: the updated global model parameters, the comprehensive aggregate weights of each client in this round, and the training metrics.

[0110] After saving the training checkpoints, the server first selects the next round of training clients from all clients based on a preset sampling ratio. Then, the server broadcasts the updated global model parameters to the selected clients, along with the training configuration information for this round, including the current round number, local training round number, local batch size, local learning rate scheduling parameters, and near-end regularization coefficients. Upon receiving the global model parameters, each client loads them into the low-rank adapter module of its local multimodal model, replacing the previous round's global parameters as the initial parameters for this round of local training. After loading the global parameters, each client performs local fine-tuning training using its local multimodal or unimodal data, generating the parameter increment and local training loss for this round. After training is complete, the client sends the parameter increment, modality availability information, and local loss back to the server. Based on this, the server re-executes modality integrity assessment, gradient quality estimation, and adaptive aggregation scheduling, generating a new round of comprehensive aggregation weights and performing global parameter aggregation.

[0111] In this embodiment, to coordinate with the local training process, a round-related learning rate can also be used to control the local update magnitude: (12); among which, Indicates the first Round local learning rate, This represents the initial learning rate. This represents the minimum learning rate. Indicates the number of preheating cycles. This indicates the total number of rounds.

[0112] In a preferred embodiment, the parameters participating in federated communication are not all the parameters of the entire multimodal large model, but rather the trainable low-rank adapted parameters obtained after fine-tuning by the low-rank adapter (LoRA). The server aggregates the low-rank adapter (LoRA) parameter states uploaded by each client according to a comprehensive weight to obtain the global parameters for the next round: (13); among which, Indicates the first Round global low-rank adapter (LoRA) parameters, Indicates the client The overall aggregate weight, Indicates the client In the The local low-rank adapter (LoRA) parameters uploaded in rounds.

[0113] When the system uses FedProx as the underlying local optimizer, the local target can be written as: (14); among which, Indicates the client In the The local training objective of the round Represents a local batch data set. This represents the task loss function. The parameter is The multimodal model, Represents the proximal regularity coefficient. This represents the current global parameters issued by the server.

[0114] Building upon this, this application combines server-side weighted aggregation with client-side proximal constraints. On one hand, the low-rank adapter (LoRA) parameters reduce communication overhead, enabling the server to save the training loss in each round and periodically distribute checkpoints; on the other hand, the combined weights... This allows global updates to no longer be solely driven by sample size, but to simultaneously reflect modality completeness and gradient quality. Therefore, this application significantly enhances the aggregation specificity in heterogeneous multimodal scenarios while maintaining existing federated training entry points, data partitioning, and compatibility with multiple federated baselines.

[0115] In one embodiment, the gradient-weighted, aware federated aggregation method for heterogeneous multimodal data may further include a client-side local training process. Specifically, each client fine-tunes its multimodal model within its local low-rank adapter LoRA parameter space. The optimization objectives of the client-side local training include a task loss term and a proximal regularization term. The proximal regularization term is the L2 distance between the client's local parameters and the global parameters issued by the server multiplied by the proximal regularization coefficient. Each client uses a cosine annealing learning rate scheduling strategy to control the local update magnitude. The cosine annealing learning rate scheduling strategy includes a warm-up phase and a decay phase. During the warm-up phase, the local learning rate linearly increases from its initial value to a preset peak value. During the decay phase, the local learning rate gradually decays from the peak value to the minimum learning rate according to a cosine curve. Each client only uploads the incremental low-rank adapter LoRA parameters to the server.

[0116] After receiving the global LoRA parameters from the server, each client loads them into the low-rank adapter module of its local multimodal model. The multimodal model consists of a pre-trained base model and an additional low-rank adapter module. The parameters of the base model remain frozen throughout the federated training process; only the parameters of the low-rank adapter module are updated. By training only the low-rank adapter parameters instead of all model parameters, the computational cost of local training on the client is significantly reduced, while effectively adapting to the fine-tuning requirements of multimodal tasks. During local training, the client employs an optimization objective with a proximal regularization term. This objective includes a task loss term and a proximal regularization term. The task loss term is the task loss calculated by the client on the local multimodal training data, used to measure the model's predictive performance on the current task; the task loss can be cross-entropy loss, mean squared error loss, or other loss functions suitable for multimodal tasks. The proximal regularization term is the L2 distance between the client's local parameters and the global parameters sent by the server, multiplied by a proximal regularization coefficient. Specifically, the client calculates the L2 distance between the current local LoRA parameters and the initial global LoRA parameters issued by the server, and then multiplies this L2 distance by a preset proximal regularization coefficient to obtain the proximal regularization term. The proximal regularization coefficient is a preset hyperparameter used to adjust the weight ratio of the proximal regularization term in the optimization objective.

[0117] During local training, each client employs a cosine annealing learning rate scheduling strategy to dynamically adjust its local learning rate. This controls the magnitude of local updates and prevents excessively large local updates in the later stages of training from weakening the server-side weight control effect. The cosine annealing learning rate scheduling strategy consists of two consecutive phases: a warm-up phase and a decay phase. During the warm-up phase, the local learning rate linearly increases from its initial value to a preset peak. At the beginning of the warm-up phase, the local learning rate is set to a relatively small initial value to prevent gradient explosion or training instability caused by random parameter initialization or distribution shifts in the early stages of training. During the decay phase, the local learning rate gradually decreases from its peak according to a cosine curve to a minimum learning rate. The local learning rate gradually decreases from its peak according to the curve shape of a cosine function, eventually reaching the preset minimum learning rate.

[0118] After completing local training, each client calculates the LoRA parameter increment and uploads it to the server. The parameter increment is the difference between the LoRA parameters at the end of the client's local training and the initial global LoRA parameters received from the server at the start of this training round; that is, the actual change in the low-rank adapter parameters during this round of local training. Each client only uploads the LoRA parameter increment, not the complete parameters of the multimodal model, nor the client's original training data. When uploading the LoRA parameter increment, the client can also upload structured state information such as local training loss, modality availability information, and missing rate, so that the server can perform modality completeness assessment and gradient quality estimation.

[0119] In one embodiment, regarding modal completeness modeling, if the client includes speech, structured sensor data, or other modalities in addition to images and text, the modality set can be... This is expanded to a multidimensional set, and the binary modality indicator vector in the formula is replaced with a multimodal sparse vector; when certain application scenarios place more emphasis on specific modalities, it can also be further... Modality priority coefficients are introduced to adapt to different tasks such as medical imaging, disaster mapping, or industrial sensing. For gradient quality estimation, norm stability and directional consistency can be replaced with updated variance, historical loss reduction rate, client confidence interval, or robust statistics. For malicious clients or severely noisy scenarios, outlier truncation or median filtering mechanisms can be superimposed to enhance attack resistance. Regarding aggregation strategies, the server is not limited to weighted FedAvg; other aggregation methods can also be used. It can be embedded in FedAdam, FedYogi, FedAdagrad, or the momentum-driven FedAvgM to replace the original equal-weight or sample-size weight update terms; on the client side, it can be combined with different low-rank adapter (LoRA) orders. Scaling factor Dropout ratio and near-end coefficient This allows communication overhead and adaptability to be adjusted according to the device's computing power. Regarding training scheduling, the round scheduling coefficients do not need to be limited to cosine functions; piecewise linear, exponential decay, or adaptive strategies based on validation performance feedback can also be used. As long as the overall principle of emphasizing modality coverage in the early stages of training and emphasizing update quality in the later stages of training is met, the desired technical effect can be achieved.

[0120] In one embodiment, the technical effectiveness of GWMA is summarized and verified around modal integrity assessment, gradient quality estimation, and adaptive scheduling aggregation mechanism. Addressing common issues in heterogeneous multimodal federated learning such as missing modalities, convergence oscillations, uneven client quality, and limited communication, a unified evaluation criterion is used to compare and summarize aggregation accuracy, convergence efficiency, and deployment adaptability. The results are shown in Tables 1 to 3.

[0121] Table 1. Overall Performance Comparison in Modality Missing Scenarios

[0122]

[0123] Table 1 shows that GWMA improves both classification accuracy and tolerance under modality missing conditions, especially maintaining a higher level in the missing modality tolerance index, indicating that modality-aware weights can more effectively retain cross-modal information provided by the complete client.

[0124] Table 2 Comparison of Aggregation Stability and Convergence Efficiency

[0125]

[0126] Table 2 shows that GWMA, by incorporating gradient quality scores into the aggregation process, can suppress the perturbation of unstable updates on the global model, resulting in faster descent in the early stages of training and greater stability in the later stages, demonstrating better convergence continuity.

[0127] Table 3. Correspondence between technical modules and deployment benefits

[0128]

[0129] Table 3 shows that the three core modules of GWMA work together with the federated fine-tuning link to form a closed loop from client quality judgment to server aggregation control and lightweight parameter transmission, thus possessing comprehensive advantages of being deployable, scalable, and balancing accuracy and efficiency.

[0130] This application provides a gradient-weighted, aware federated aggregation method for heterogeneous multimodal data. It prioritizes clients with richer cross-modal information by constructing a modality completeness score; evaluates the stability and effectiveness of local updates in multiple training rounds by building a gradient quality estimation mechanism; and emphasizes modality coverage value in the early stages of training and gradually increases attention to high-quality gradients in the later stages through an adaptive aggregation scheduling strategy. Rather than simply replacing sample-weighted aggregation, this method introduces a contribution modeling mechanism that better suits the characteristics of multimodal heterogeneous scenarios on top of the existing federated training framework, thereby improving the targeting, stability, and generalization ability of global aggregation. This method has clear engineering necessity for applications such as hate content detection, emergency multimedia analysis, and medical visual question answering.

[0131] This application provides a gradient-weighted, perceptual federated aggregation method for heterogeneous multimodal data. By prioritizing updates from clients possessing both image and text information through modality integrity scoring, it reduces the dilution effect of single-modality samples on global aggregation. As shown in Table 1, GWMA achieves an accuracy of 74.3%, higher than FedAvg's 71.4% and FedYogi's 72.8%, effectively addressing the technical problem of inaccurate information contribution differentiation in heterogeneous multimodal federated learning. GWMA jointly models gradient quality scores and modality-aware weights, ensuring that server aggregation focuses not only on sample size but also on the stability and consistency of update directions. As shown in Table 1, GWMA achieves an F1 score of 73.8%, superior to FedProx's 71.5%, indicating a reduction in class boundary shifts under multi-client, multimodal conditions and enhanced global model discriminative consistency. By introducing a modality completeness factor into the aggregation weights, a more refined contribution assessment is adopted for clients with missing modalities, avoiding their simple amplification or complete neglect. As shown in Table 1, GWMA's tolerance for missing modalities reaches 71.6%, higher than FedAdam's 69.1%, indicating that this application can maintain more stable task performance under conditions of incomplete modalities. The adaptive scheduling aggregation mechanism emphasizes modality coverage in the early stage of training and gradient quality in the later stage, thus making the aggregation objectives at different stages clearer. As shown in Table 2, GWMA achieves stable accuracy in 69 rounds, better than FedAvgM's 76 rounds and FedAvg's 88 rounds, indicating that this application shortens the training time caused by repeated oscillations and the accumulation of invalid updates. By estimating the stability of uploaded gradients and suppressing the impact of low-quality updates on the server side, GWMA can reduce the large fluctuations in the global loss curve. GWMA improves the proportion of effective updates through quality-aware weighting, indicating that this application can simultaneously improve the performance of tail clients and reduce the gap between clients. Gradient quality estimation is used to identify client updates with directional deviations and norm anomalies, preventing the server from being pulled by low-confidence parameters during aggregation. This demonstrates stronger robustness to noisy data, distribution skew, and local training instability. GWMA operates on the federated aggregation link of trainable parameters of the low-rank adapter (LoRA), eliminating the need for full synchronization of the complete multimodal model. This reduces communication overhead while retaining modality-aware and quality-aware control capabilities. As shown in Tables 2 and 3, this application achieves training benefits of fast initial speed and stable subsequent performance, as well as deployment benefits of maintaining high performance with low overhead. It is suitable for widespread application in scenarios with heterogeneous terminals, limited bandwidth, or fluctuating number of participating clients.

[0132] In one embodiment, the overall structure of a gradient-weighted, sensed federated aggregation method for heterogeneous multimodal data is as follows: Figure 3As shown, the server controller is responsible for maintaining the current round, global low-rank adapter (LoRA) parameters, and training configuration, and drives the entire federated training cycle. The client sampler selects participants from all terminals according to a preset sampling ratio, ensuring that heterogeneous image clients, text clients, and bimodal clients can enter the unified scheduling link. The multimodal client cluster performs low-rank adapter (LoRA) fine-tuning of the multimodal base model locally on each client, uploading only parameter increments without exposing the original data. The modality completeness estimator calculates modality weights alpha based on the client's image modality availability, text modality availability, missing rate, and sample size to identify nodes that provide more cross-modal knowledge. The gradient quality estimator analyzes the norm stability and directional consistency of the increments uploaded by each client, obtaining a gradient quality score beta. The round-aware aggregation scheduler dynamically adjusts the ratio of alpha to beta according to the current training stage, emphasizing modality coverage in the early stages of training and update stability in the later stages. The global low-rank adapter (LoRA) aggregator completes server-side parameter fusion based on the comprehensive weight omega. The model state storage module is used to save checkpoints, loss curves, and historical weights, and provides recovery information to the server controller. The loop in the diagram, from the global low-rank adapter (LoRA) aggregator back to the round-aware aggregation scheduler, and from the model state storage module back to the server controller, embodies the feedback loop of this application, enabling the server to continuously revise subsequent aggregation decisions based on the convergence trend.

[0133] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0134] In one embodiment, such as Figure 1 As shown, a gradient-weighted, aware federated aggregation system for heterogeneous multimodal data is provided, including a server 110 and various heterogeneous clients 120, wherein:

[0135] Server 110 is used to obtain modal availability information of each client 120. The modal availability information includes modal indicator vectors, the missing rate of each modality, and the number of local samples. Based on the modal indicator vectors and the missing rate of each modality, the effective modal retention coefficient is calculated. Based on the effective modal retention coefficients and the number of local samples, the modal integrity score is calculated. The modal integrity score of each client 120 is normalized to obtain the modal perception weight of each client 120.

[0136] Server 110 is also used to receive parameter increments uploaded by each client 120 after completing local training, calculate norm stability score based on the deviation between the second norm of the parameter increment and the historical smooth norm, and calculate direction consistency score based on the direction cosine similarity between the parameter increment of the current round and the parameter increment of the previous round, and fuse the norm stability score and direction consistency score to obtain the gradient quality score of each client 120.

[0137] Server 110 is also used to obtain the current training round and the preset total number of training rounds and generate round scheduling coefficients. The round scheduling coefficients are used to weight and combine the modality-aware weights and gradient quality scores to generate the comprehensive aggregate weights of each client 120.

[0138] Server 110 is also used to perform weighted summation of parameter increments uploaded by each client 120 based on comprehensive aggregation weights, generate updated global model parameters, and distribute them to each client 120 for the next round of federated training.

[0139] In one embodiment, server 110 is further configured to establish a binary modality indicator vector for each client 120, the binary modality indicator vector including an image modality availability identifier and a text modality availability identifier; for each client 120, server 110 is further configured to multiply the availability identifier of each modality by the complement of the corresponding modality missing rate and then accumulate the results to obtain an accumulated value; based on the accumulated value and the total number of modalities, an effective modality retention coefficient is obtained; the effective modality retention coefficient is subjected to a modality integrity enhancement power operation, and the result is multiplied by the logarithmically transformed value of the local sample number to obtain the original modality integrity score; the original modality integrity scores of all participating clients 120 in the current round are summed to obtain a total score, and the proportion of the original modality integrity score of each client 120 to the total score is used as the modality perception weight of client 120.

[0140] In one embodiment, server 110 is further configured to receive parameter increments uploaded by each client 120 after completing local training. The parameter increment is the difference between the model parameters after the client 120's local training ends and the global model parameters at the start of the current training round. For each client 120, server 110 is further configured to calculate the L2 norm of the parameter increment in the current round to represent the current update magnitude, and perform exponential moving average processing on the L2 norm and the historical smoothing norm to update the historical record. The exponential moving average processing includes: weighting the historical smoothing norm of the previous round with a preset smoothing coefficient, weighting the L2 norm with the complement of the smoothing coefficient, and using the sum of the two as the historical smoothing norm after the current round update. Server 110 is further configured to calculate the absolute difference between the L2 norm and the historical smoothing norm of the previous round, calculate the relative deviation based on the absolute difference and the historical smoothing norm of the previous round, take the negative value of the relative deviation and perform exponential transformation to obtain the norm stability score.

[0141] In one embodiment, for clients 120 that have been sampled in two consecutive rounds, server 110 is further configured to calculate the vector dot product of the parameter increment in the current round and the parameter increment in the previous round, calculate the L2 norm of the parameter increment in the current round and the parameter increment in the previous round respectively, and calculate the direction cosine similarity based on the vector dot product and the two L2 norms; server 110 is further configured to perform linear mapping on the direction cosine similarity to obtain the direction consistency score; and perform linear weighted fusion of the norm stability score and the direction consistency score with a preset balance coefficient to obtain the gradient quality score of each client 120.

[0142] In one embodiment, server 110 is further configured to determine whether the gradient quality score of each client 120 is lower than a preset quality threshold, or whether the norm stability score deviates from the historical statistical range of the corresponding client 120 by more than a preset multiple standard deviation, or whether the directional consistency score deviates from the historical statistical range of the corresponding client 120 by more than a preset multiple standard deviation; when any judgment condition is met, server 110 is further configured to mark client 120 as an abnormal update client 120 and reduce the gradient quality score of the abnormal update client 120; the reduced gradient quality score is used for the calculation of the comprehensive aggregation weight, so that the proportion of the abnormal update client 120 in the comprehensive aggregation weight is reduced.

[0143] In one embodiment, server 110 is further configured to read the current training round and the preset total number of training rounds from the round controller; construct a round scheduling coefficient based on the current training round and the total number of training rounds, wherein the round scheduling coefficient decreases as the number of training rounds increases; for each client 120, server 110 is further configured to multiply the modality-aware weight by the round scheduling coefficient to obtain a modality-weighted term; multiply the gradient quality score by the complement of the round scheduling coefficient to obtain a quality-weighted term; add the modality-weighted term and the quality-weighted term to obtain the original comprehensive score; sum the original comprehensive scores of all participating clients 120 in the current round to obtain the total score, and use the proportion of the original comprehensive score of each client 120 to the total score as the comprehensive aggregation weight of client 120.

[0144] In one embodiment, server 110 is further configured to receive parameter increments uploaded by each client 120, wherein the parameter increment is the difference between the low-rank adapter LoRA parameters after the local training of client 120 and the initial global LoRA parameters of this round; for each client 120, server 110 is further configured to multiply the parameter increment element-wise by the corresponding comprehensive aggregation weight to obtain the weighted parameter increment, and accumulate the weighted parameter increments of all participating clients 120 element-wise to obtain the global parameter update amount; add the initial global model parameters of this round and the global parameter update amount element-wise to obtain the updated global model parameters; save the updated global model parameters, the comprehensive aggregation weights of each client 120 in this round and the training metrics to form a training checkpoint; and send the updated global model parameters to the clients 120 participating in the next round of training, so that each client 120 can load and start the next round of local training, forming a federated training closed loop.

[0145] In one embodiment, each client 120 performs multimodal model fine-tuning within its local low-rank adapter LoRA parameter space. The optimization objectives of the local training by the client 120 include a task loss term and a proximal regularization term. The proximal regularization term is the L2 distance between the local parameters of the client 120 and the global parameters issued by the server 110 multiplied by the proximal regularization coefficient. Each client 120 uses a cosine annealing learning rate scheduling strategy to control the local update magnitude. The cosine annealing learning rate scheduling strategy includes a warm-up phase and a decay phase. During the warm-up phase, the local learning rate linearly increases from the initial value to a preset peak value. During the decay phase, the local learning rate gradually decays from the peak value to the minimum learning rate according to a cosine curve. Each client 120 only uploads the incremental low-rank adapter LoRA parameters to the server 110.

[0146] In one embodiment, a gradient-weighted, sensed, federated aggregation system for heterogeneous multimodal data may include a gradient quality estimation module, the structure of which is as follows: Figure 4As shown, the system first receives the low-rank adapter (LoRA) parameter increment Delta uploaded by the client and sends updates from adjacent rounds of the same client into the analysis process. The norm stability analysis unit calculates the L2 norm of the current increment and compares it with the historical smoothing norm to determine if the client is experiencing gradient spikes, oscillations, or abnormal decay. The direction consistency analysis unit uses cosine similarity to measure whether the update direction in this round is consistent with the previous round, thereby identifying direction reversals caused by local training noise, missing modal interference, or local overfitting. The quality score fusion unit combines the stability score qi and the direction consistency score si according to a preset balance coefficient, outputting a unified gradient quality score Beta I. When Beta I is found to be too low or a single indicator deviates significantly from the historical range, the abnormal update discriminator sends a re-suppression signal to the previous stage, prompting the system to re-examine the client's credibility instead of directly sending low-quality updates into global aggregation. The weight normalizer converts the retained Beta I into quality weights that can participate in server scheduling, and then passes these quality weights to the server aggregation interface, where they, along with the modal weights, form the final aggregation coefficients. The module saves historical rounds and caches them to provide historical mean, threshold and trend references for norm stability analysis unit, forming a feedback loop for anomaly identification and weight correction. This enables the module to maintain a stable preference for high-quality client updates as federated training continues.

[0147] In one embodiment, the core interaction timing of a gradient-weighted, aware federated aggregation method and system for heterogeneous multimodal data in a single round of federated training is as follows: Figure 5As shown in the diagram, firstly, the server controller sends the current round number and sampling quantity to the client sampler. The client sampler then selects participating nodes based on the fixed seed of the round, ensuring reproducible client sampling results for each training round. Subsequently, the server broadcasts the current global low-rank adapter (LoRA) parameters to the selected client nodes. Each client first reads its own image modalities, text modalities, and their missing states, then enters the local low-rank adapter (LoRA) trainer to perform multimodal fine-tuning. After training, the client obtains the parameter increment Delta i and the local loss, and sends the structured states such as modality availability, missing rate, and sample size to the modality estimator, forming the modality weight Alpha I. Simultaneously, the client also sends the current round's parameter increment and historical update index to the gradient quality estimator, which internally performs norm stability analysis and direction consistency analysis to obtain the gradient quality score Beta I. The aggregation scheduler receives Alpha I, Beta I, and the round scheduling coefficients, generates the comprehensive aggregation weight Omega I, and sends it back to the server to perform global low-rank adapter (LoRA) aggregation. After the server completes the aggregation, it synchronously saves the checkpoints and logs, and then broadcasts the new global parameters to the client to proceed to the next round. The time series shows that this application does not simply average the data after local training on the client, but rather filters and reweights client contributions layer by layer through three stages: modality value evaluation, gradient quality evaluation, and round-aware scheduling, thereby achieving more robust server-side federated aggregation in heterogeneous multimodal data environments.

[0148] In one embodiment, a gradient-weighted sensing federated aggregation method and system for heterogeneous multimodal data can be deployed in a collaborative medical intranet environment consisting of tertiary hospitals, specialized hospitals, and regional imaging centers. Each medical institution retains its local image and medical record question-and-answer data, sharing only the low-rank adapter (LoRA) parameter increments to avoid patient privacy leaks. Since some institutions only have image reading capabilities while others have complete text and image medical records, the system needs to complete the joint training of the medical question-and-answer model under conditions of inconsistent original data formats, and improve the global diagnostic and treatment assistance accuracy and convergence stability under limited bandwidth and strict compliance constraints; the specific process is as follows... Figure 6 As shown:

[0149] M1: Server Initialization Federated Task: The server loads the multimodal base model and low-rank adapter (LoRA) adaptation parameters on the trusted intranet node, configures the total number of clients, sampling ratio, number of training rounds and learning rate corresponding to the medical data, establishes a global parameter cache, round log and client index, and outputs the initial low-rank adapter (LoRA) parameters to be broadcast and the training schedule table.

[0150] M2: Hospital clients report modal status: Each hospital locally compiles statistics on image modalities, text medical record modalities, missing rates, and sample size, and defines the generated modal availability code and effective modal retention coefficient; the server summarizes these and forms the modal completeness input, in which bimodal hospitals are given priority to enter the high contribution candidate set, and the relevant completeness advantage can be seen in the multimodal benefit conclusion in Table 1.

[0151] M3: Server Distributes Shared Parameters: Based on the sampling results, the server distributes the current global low-rank adapter (LoRA) parameters, round number, and local training configuration to the selected hospitals, enabling the complete bimodal hospital, the imaging monomodal hospital, and the text monomodal hospital to start training under a unified interface, and outputting local fine-tuning tasks that can be executed by each client.

[0152] M4: The client performs local medical question-answering training: Each hospital uses local image question-answering samples to complete several steps of low-rank adapter (LoRA) fine-tuning, generating this round of parameter increments, local losses, and modal coverage statistics; for hospitals with only single-modal data, only the corresponding coded pathways are updated and the missing identifiers are retained, thereby outputting local update results that can be used for subsequent quality analysis.

[0153] M5: Server-side gradient quality estimation: The server receives parameter increments uploaded by each hospital, calculates norm stability and directional consistency, obtains gradient quality scores, and identifies anomalous updates caused by small samples or modal missing values, which are used to output candidate quality weight values ​​for each hospital.

[0154] M6: Perform gradient-weighted modality-aware aggregation: The server combines modality integrity weights, gradient quality scores, and current round scheduling coefficients to generate comprehensive aggregation weights. In the early stages of training, it prioritizes absorbing complete hospital knowledge in terms of text and images, and in the later stages, it increases the proportion of stable updates. Combined with the convergence round indicators in Table 2, the aggregation coefficients are adjusted for round awareness, and new global low-rank adapter (LoRA) parameters are output.

[0155] M7: Result Writeback and Next Round Broadcast: The server saves the updated global parameters, round indicators, and hospital-level weight statistics, and broadcasts the new parameters to the hospitals selected in the next round; each hospital continues to perform local training accordingly, forming a closed-loop iterative link of "local update - quality assessment - server aggregation".

[0156] M8: Global Model Deployment and Diagnosis Invocation: When the preset number of rounds or convergence threshold is reached, the server freezes the final low-rank adapter (LoRA) parameters and distributes them to each hospital inference node for medical image question answering and assisted diagnosis. During the deployment phase, the modal state registration interface is retained to provide a scalable operating foundation for subsequent access of new hospitals and periodic retraining output.

[0157] In this multi-hospital collaborative medicine scenario, GWMA prioritizes the use of cross-modal knowledge from hospitals with complete text and image data and suppresses unstable updates, enabling the global model to maintain high accuracy in diagnosis and treatment question answering even in environments with missing modalities. As shown in Tables 1 and 2, this scheme can improve the tolerance for missing modalities to 71.6% while reducing the number of stable convergence rounds to 69. This demonstrates that this embodiment can achieve more robust cross-institutional joint modeling results while taking into account medical privacy, bandwidth limitations, and differences in institutional capabilities.

[0158] In one embodiment, a gradient-weighted sensing federated aggregation method and system for heterogeneous multimodal data can be deployed in a cross-domain collaborative network composed of emergency management departments, public welfare organizations, and frontline mobile terminals. Different organizations continuously upload disaster images, on-site text descriptions, and short messages. However, due to significant differences in terminal capabilities, some nodes can only provide image or text-based single-modal data, and the wireless link bandwidth fluctuates considerably. The system aims to complete the federated update of the disaster identification model without centrally aggregating raw disaster data, and to reduce the interference of sudden noise, cold-start terminals, and nodes with high missing rates on the global judgment results through a gradient-weighted mechanism; the specific process is as follows: Figure 7 As shown:

[0159] M1: Emergency server establishes round control center: The central node creates a federated session, configures the data scenario, client sampling number, total rounds and low-rank adapter (LoRA) fine-tuning hyperparameters for disaster identification tasks, and establishes a registry for mobile terminals, command vehicles and fixed sites, outputting initial global parameters with round control information.

[0160] M2: Terminal registration modal availability and link status: When accessing the network, each terminal reports the availability, missing rate and number of local samples of image and text modalities. The server generates a modal completeness score based on this. Command nodes with complete image and text modalities are given higher prior weights, while single-modal nodes retain their eligibility to participate. The priority of this step can be determined by referring to the overall accuracy and missing modal tolerance indicators in Table 1.

[0161] M3: The server selects participating nodes and sends parameters: The server selects participating terminals in this round based on the sampling strategy and link status, and sends the current global low-rank adapter (LoRA) parameters, batch size and gradient accumulation configuration, so that high-bandwidth nodes and edge lightweight nodes can enter the training process under a unified protocol and output local training start instructions.

[0162] M4: Terminal performs local disaster sample fine-tuning: Participating terminals use local image and text samples or single-modal samples to perform short-cycle low-rank adapter (LoRA) training, generating parameter increments, local losses and training duration statistics; for cold-start terminals, only a small number of gradient updates are performed and historical missing identifiers are retained, and lightweight update packages that can be sent back are output.

[0163] M5: Server performs abnormal update identification: The server compares the norm change and directional similarity of the current increment with the historical increment, filters out abnormal updates caused by network jitter, sample skew, or sudden noise, and generates a gradient quality score, which can be used to stabilize the aggregation results after edge terminal access.

[0164] M6: The server completes adaptive aggregation scheduling: The server integrates the modal integrity weights and gradient quality scores according to the round coefficients, prioritizes absorbing complete terminal knowledge in the early stage of training, and emphasizes directional stability and reliable quality updates in the later stage of training; combined with the 69 rounds of convergence index in Table 2, it outputs global parameters that take into account both speed and stability.

[0165] M7: Global parameters are written back to frontline nodes: After aggregation, the server writes back the new low-rank adapter (LoRA) parameters to the frontline terminals and command nodes, and simultaneously records the contribution weight, link latency and model performance of various terminals, providing a basis for decision-making on the next round of client selection and cold start node compensation output; combined with the communication benefit indicators in Table 3, the participation frequency of weak bandwidth nodes can be controlled accordingly.

[0166] M8: Disaster assessment results are continuously updated in a closed loop: Frontline nodes use the latest models to perform disaster classification, image and text matching, and risk level assessment. New samples and inference feedback continue to flow back to the server, driving the next round of modality registration and federated training, forming a continuous adaptive closed loop for emergency response.

[0167] In this multi-agency disaster assessment scenario, GWMA maintains high robustness in identification even with uneven edge terminal capabilities and significant link fluctuations. As shown in Tables 1 and 2, the system maintains a tolerance level of 71.6% even under missing modal conditions, and keeps the number of stable convergence rounds of the global model at 69 rounds, maintaining a stable accuracy of 74.1%. This demonstrates that this embodiment is more suitable for emergency collaborative deployment environments with frequent cold-start terminal access and unstable network conditions.

[0168] In one embodiment, a computer device is provided, which may be a server or a client, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a gradient-weighted sensing federated aggregation method for heterogeneous multimodal data. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0169] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0170] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement steps of a gradient-weighted, sensed federated aggregation method for heterogeneous multimodal data.

[0171] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being executed by a processor to implement the steps of a gradient-weighted, aware federated aggregation method for heterogeneous multimodal data.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A gradient-weighted, aware federated aggregation method for heterogeneous multimodal data, applied to a federated learning system consisting of a server and various heterogeneous clients, characterized in that... The method includes: The server obtains modal availability information for each client, including modal indicator vectors, the missing rate of each modality, and the number of local samples. Based on the modal indicator vectors and the missing rate of each modality, the server calculates an effective modality retention coefficient, and based on the effective modality retention coefficients and the number of local samples, the server calculates a modality integrity score. The server then normalizes the modality integrity scores of each client to obtain the modality perception weight of each client. The server receives the parameter increments uploaded by each client after completing local training, calculates the norm stability score based on the deviation between the second norm of the parameter increment and the historical smooth norm, and calculates the direction consistency score based on the direction cosine similarity between the parameter increment of the current round and the parameter increment of the previous round. The server then merges the norm stability score and the direction consistency score to obtain the gradient quality score of each client. The server obtains the current training round and the preset total number of training rounds and generates round scheduling coefficients. It then uses the round scheduling coefficients to weight and combine the modality-aware weights and the gradient quality scores to generate a comprehensive aggregate weight for each client. The server performs a weighted summation of the parameter increments uploaded by each client based on the comprehensive aggregation weight, generates updated global model parameters, and distributes them to each client for the next round of federated training.

2. The gradient-weighted sensing federated aggregation method for heterogeneous multimodal data according to claim 1, characterized in that, The server calculates an effective modality retention coefficient based on the modality indication vector and the missing rate of each modality, calculates a modality integrity score based on the effective modality retention coefficient and the number of local samples, and normalizes the modality integrity score of each client to obtain the modality-aware weight of each client, including: The server establishes a binary modality indicator vector for each client, which includes an image modality availability identifier and a text modality availability identifier; For each client, the availability flag of each modality is multiplied by the complement of the corresponding modality missing rate and then summed to obtain an accumulated value. Based on the accumulated value and the total number of modalities, the effective modality retention coefficient is obtained. The server performs a modality integrity enhancement power operation on the effective modality retention coefficients, and multiplies the result by the logarithmically transformed value of the local sample count to obtain the original modality integrity score. The server sums the original modal completeness scores of all participating clients in the current round to obtain a total score, and uses the proportion of each client's original modal completeness score to the total score as the client's modal perception weight.

3. The gradient-weighted sensing federated aggregation method for heterogeneous multimodal data according to claim 1, characterized in that, The server receives parameter increments uploaded by each client after completing local training, and calculates a norm stability score based on the deviation between the L2 norm of the parameter increments and the historical smoothing norm, including: The server receives parameter increments uploaded by each client after completing local training. The parameter increments are the differences between the model parameters after the client's local training ends and the global model parameters at the beginning of the current training round. For each client, the server calculates the L2 norm of the parameter increment in this round to represent the current update magnitude, and performs an exponential moving average process on the L2 norm and the historical smoothing norm to update the historical record; wherein, the exponential moving average process includes: weighting the historical smoothing norm of the previous round with a preset smoothing coefficient, weighting the L2 norm with the complement of the smoothing coefficient, and using the sum of the two as the historical smoothing norm after the current round update; The server calculates the absolute difference between the L2 norm and the previous historical smoothing norm, calculates the relative deviation based on the absolute difference and the previous historical smoothing norm, and performs an exponential transformation after taking the negative value of the relative deviation to obtain the norm stability score.

4. The gradient-weighted sensing federated aggregation method for heterogeneous multimodal data according to claim 3, characterized in that, The server calculates a directional consistency score based on the directional cosine similarity between the parameter increment of the current round and the parameter increment of the previous round, and fuses the norm stability score and the directional consistency score to obtain the gradient quality score for each client, including: For clients that have been sampled in two consecutive rounds, the server calculates the vector dot product of the parameter increment in the current round and the parameter increment in the previous round, calculates the L2 norm of the parameter increment in the current round and the parameter increment in the previous round respectively, and calculates the direction cosine similarity based on the vector dot product and the two L2 norms. The server performs a linear mapping on the direction cosine similarity to obtain a direction consistency score. The server performs a linear weighted fusion of the norm stability score and the directional consistency score using a preset balance coefficient to obtain the gradient quality score for each client.

5. The gradient-weighted sensing federated aggregation method for heterogeneous multimodal data according to claim 1, characterized in that, The method further includes: The server determines whether the gradient quality score of each client is lower than a preset quality threshold, or whether the norm stability score deviates from the historical statistical range of the corresponding client by more than a preset multiple standard deviation, or whether the directional consistency score deviates from the historical statistical range of the corresponding client by more than a preset multiple standard deviation. When any of the judgment conditions are met, the server marks the client as an abnormal update client and reduces the gradient quality score of the abnormal update client. The reduced gradient quality score is used to calculate the overall aggregate weight, thereby reducing the proportion of the abnormal update client in the overall aggregate weight.

6. The gradient-weighted sensing federated aggregation method for heterogeneous multimodal data according to claim 1, characterized in that, The server obtains the current training round and the preset total number of training rounds and generates round scheduling coefficients. It then uses these round scheduling coefficients to weight and combine the modality-aware weights and the gradient quality scores to generate a comprehensive aggregate weight for each client, including: The server reads the current training round and the preset total number of training rounds from the round controller; The server constructs a round scheduling coefficient based on the current training round and the total number of training rounds, and the round scheduling coefficient decreases as the number of training rounds increases; For each client, the server multiplies the modality-aware weights by the round-based scheduling coefficients to obtain a modality-weighted term; multiplies the gradient quality score by the complement of the round-based scheduling coefficients to obtain a quality-weighted term; and adds the modality-weighted term and the quality-weighted term to obtain the original comprehensive score. The server sums the original comprehensive scores of all participating clients in the current round to obtain the total score, and uses the proportion of each client's original comprehensive score to the total score as the client's comprehensive aggregation weight.

7. The gradient-weighted sensing federated aggregation method for heterogeneous multimodal data according to claim 1, characterized in that, The server performs a weighted summation of the parameter increments uploaded by each client based on the comprehensive aggregation weights, generates updated global model parameters, and distributes them to each client for the next round of federated training, including: The server receives the parameter increment uploaded by each of the clients, and the parameter increment is the difference between the low-rank adapter LoRA parameter after the client's local training ends and the initial global LoRA parameter in this round. For each client, the server multiplies the parameter increment element by the corresponding comprehensive aggregation weight to obtain the weighted parameter increment, and then sums the weighted parameter increments of all participating clients element by element to obtain the global parameter update amount. The server adds the initial global model parameters of this round to the global parameter update amount element by element to obtain the updated global model parameters; The server saves the updated global model parameters, the aggregated weights of each client in this round, and the training metrics to form training checkpoints; The server distributes the updated global model parameters to the clients participating in the next round of training, and each client loads and starts the next round of local training, forming a federated training closed loop.

8. The gradient-weighted sensing federated aggregation method for heterogeneous multimodal data according to claim 7, characterized in that, The method further includes: Each client performs multimodal model fine-tuning in the local low-rank adapter LoRA parameter space. The optimization objectives of the client's local training include a task loss term and a proximal regularization term. The proximal regularization term is the L2 distance between the client's local parameters and the global parameters issued by the server multiplied by the proximal regularization coefficient. Each client uses a cosine annealing learning rate scheduling strategy to control the local update magnitude. The cosine annealing learning rate scheduling strategy includes a warm-up phase and a decay phase. During the warm-up phase, the local learning rate linearly increases from the initial value to a preset peak value. During the decay phase, the local learning rate gradually decays from the peak value to the minimum learning rate according to a cosine curve. Each client only uploads the LoRA parameters of the low-rank adapter incrementally to the server.

9. A gradient-weighted sensing federated aggregation system for heterogeneous multimodal data, characterized in that, The system includes a server and various heterogeneous clients, wherein: The server is configured to acquire modal availability information for each client, the modal availability information including modal indicator vectors, the missing rate of each modality, and the number of local samples; calculate an effective modal retention coefficient based on the modal indicator vectors and the missing rate of each modality, calculate a modal integrity score based on the effective modal retention coefficients and the number of local samples, and normalize the modal integrity scores of each client to obtain the modal awareness weight of each client; The server is also used to receive parameter increments uploaded by each client after completing local training, calculate norm stability score based on the deviation between the second norm of the parameter increment and the historical smooth norm, calculate directional consistency score based on the directional cosine similarity between the parameter increment of the current round and the parameter increment of the previous round, and fuse the norm stability score and the directional consistency score to obtain the gradient quality score of each client. The server is also used to obtain the current training round and the preset total number of training rounds and generate round scheduling coefficients, and use the round scheduling coefficients to weight and combine the modality-aware weights and the gradient quality scores to generate the comprehensive aggregate weights of each client. The server is also used to perform a weighted summation of the parameter increments uploaded by each client based on the comprehensive aggregation weight, generate updated global model parameters, and send them to each client for the next round of federated training.

Images