A method and system for collaborative training of cross-modal models with communication efficiency

Abstract

The application discloses a kind of computing and communication efficiency cross-modal model collaborative training method and system, training method includes the following steps: using the hybrid network structure of fusion local perception and global semantic association, the model size is compressed by combining structure reparameterization technology to construct light cross-modal basic model architecture;Constructing a minimalist feature adaptation module, adding a minimalist feature adaptation module at the output end of the cross-modal basic model architecture, by gradually reducing its parameter quantity to a minimalist linear transformation form, local feature differentiation tuning is realized, while reducing the amount of communication transmission data;In local training, by comparing the feature distribution difference of public reference data set and local data, the distribution distance loss function is minimized, to promote the feature space of each participant to gradually align;Server side maintains global feature adaptation module parameters, each participant uploads updates after fine-tuning the module based on local data, and the global model is optimized by weighted average.

Description

Technical Field

[0001] This invention relates to the intersection of distributed machine learning and edge computing, and in particular to a method and system for cross-modal model collaborative training with high computational and communication efficiency. Background Technology

[0002] In recent years, Federated Learning (FL) has shown significant potential in image classification due to its distributed collaboration and privacy protection mechanisms, but it still faces multiple challenges. First, existing solutions rely on the aggregation of all parameters of the global model, resulting in enormous pressure on communication bandwidth, especially in scenarios where edge devices have limited resources and it is difficult to meet real-time requirements. Second, although traditional federated frameworks (such as FedCPA and AugHFL) have improved robustness against poisoning attacks and heterogeneous data through parameter filtering or data augmentation, they have not solved the problem of deep feature alignment for non-independent identically distributed (Non-IID) data, which can easily lead to client drift and model performance degradation. In addition, although lightweight communication technologies (such as single-step feature compression in 3SFC and convolution kernel-level parameter selection in SmartIdx) have reduced the amount of data transmitted, they are still limited by module redundancy or insufficient sparsity efficiency, making it difficult to achieve extreme compression in ultra-low bandwidth scenarios.

[0003] Research based on Contrastive Language-Image Pretraining (CLIP) provides a new approach to multimodal federated learning. CLIP achieves strong correlation between image and text features through cross-modal contrastive learning, and its derivative techniques (such as the CLIP denoising network by Cheng et al. and the CODER feature alignment by Yi et al.) have performed well in image classification and retrieval tasks. However, existing cross-modal models have a large number of parameters, which leads to computational and storage bottlenecks when directly deployed to edge devices, and the synchronous transmission of multimodal features during federated aggregation further increases communication overhead.

[0004] Research on the Internet of Things (IoT) and communication optimization provides crucial support for distributed learning. For example, Federated Generative Models achieve multi-source data synthesis through a federated generation framework, while RED-New and FedNew methods utilize lightweight transmission and quantization techniques to reduce communication costs. However, existing lightweight models (such as MobileSAM and EfficientSAM) are not optimized for the multimodal characteristics of federated scenarios, making it difficult to balance computational efficiency with preserving cross-modal semantic relationships. Furthermore, non-IID data alignment often relies on complex domain adaptive modules (such as MMD loss and adversarial training), and such methods are difficult to implement in low-bandwidth scenarios due to excessive computational load.

[0005] The core contradiction in current federated learning lies in the imbalance between performance, efficiency, and privacy:

[0006] Insufficient model lightweighting: Although the hybrid architecture reduces some computational complexity, traditional heavy parameterization techniques still suffer from problems such as high inference latency and parameter redundancy, limiting the deployment efficiency of edge devices;

[0007] Communication compression bottleneck: Existing solutions (such as dynamic sparse training and Top-k sparsity) have failed to achieve extreme simplification at the module level, and the amount of communication data in a single round still accounts for more than 10% of the original model;

[0008] Cross-modal alignment is inefficient: In multimodal federated learning such as text-image, the heterogeneity of the feature space makes it difficult to achieve global aggregation convergence, and there is a lack of lightweight semantic guidance mechanisms;

[0009] Security versus efficiency conflict: Privacy enhancement techniques (such as differential privacy) are often accompanied by a decrease in model accuracy, while efficient communication strategies (such as gradient quantization) can easily amplify the risk of malicious attacks.

[0010] Therefore, there is an urgent need for a lightweight, low-communication-overhead federated learning scheme that can deeply integrate cross-modal alignment capabilities to solve the collaborative optimization challenges of performance degradation, resource constraints, and privacy leaks in edge scenarios. Summary of the Invention

[0011] This solution addresses the problems and needs raised above by proposing a computationally and communicationally efficient cross-modal model collaborative training method and system. It achieves the aforementioned technical objectives and brings about several other technical benefits due to the adoption of the following technical features.

[0012] One objective of this invention is to propose a method for cross-modal model co-training with computational and communication efficiency, comprising the following steps:

[0013] S10: Constructing a lightweight cross-modal basic model architecture: Adopting a hybrid network structure that integrates local perception and global semantic association, and combining structural reparameterization technology to compress the model size to construct a lightweight cross-modal basic model architecture;

[0014] S20: Construct a simplified feature adaptation module: Add a simplified feature adaptation module to the output of the cross-modal basic model architecture. By gradually reducing its parameter count to a simplified linear transformation form, local feature differentiation optimization is achieved, while reducing the amount of communication transmission data.

[0015] S30: Introducing a distributed feature alignment strategy: During local training, by comparing the feature distribution differences between the public reference dataset and the local data, the distribution distance loss function is minimized, which promotes the gradual alignment of the feature spaces of each participant.

[0016] S40: Distributed parameter aggregation and update: The server maintains the parameters of the global feature adaptation module. Each participant fine-tunes the module based on local data and then uploads the update. Global model optimization is achieved through weighted averaging.

[0017] Furthermore, the computational and communication efficiency cross-modal model collaborative training method according to the present invention may also have the following technical features:

[0018] In one example of the present invention, step S10 specifically includes the following steps:

[0019] In the image encoder, convolutional networks and self-attention mechanisms are integrated to enhance computational efficiency and expressive power through local perception and global semantic association.

[0020] In the training and inference phases, equivalent reparameterization techniques are used to transform complex network structures into compact forms, thereby reducing floating-point operations and parameter size.

[0021] Lightweight operations are used instead of fully connected layers to reduce computational complexity.

[0022] In one example of the present invention, step S10, which combines structural reparameterization technology to compress the model size, specifically includes the following steps:

[0023] Transform the multi-branch network in the training phase into a single-path structure in the inference phase;

[0024] Replacing some fully connected layers with 1×1 convolutions reduces parameter redundancy and computational complexity.

[0025] In one example of the present invention, step S20, constructing the minimalist feature adaptation module specifically includes the following steps:

[0026] S21: Generate feature attention masks by scaling or non-linear mapping through channels, and then perform weighted adjustments on the original features;

[0027] S22: A linear transformation layer with fewer than a preset threshold of parameters is used as an extremely simplified form of the adaptation module, while retaining key discrimination information in conjunction with the information bottleneck theory.

[0028] In one example of the present invention, the training process of the minimalist feature adaptation module satisfies the following conditions:

[0029] 1) By constraining the feature compression process through information bottleneck theory, the amount of mutual information related to the task is maximized.

[0030] 2) An end-to-end joint optimization strategy is adopted to synchronously update the parameters of the adaptation module and the cross-modal comparative learning target.

[0031] In one example of the present invention, step S30 specifically includes the following steps:

[0032] S31: Select an unlabeled public dataset and calculate the difference in distribution between it and the local data in the feature space.

[0033] S32: Simultaneously optimize cross-modal contrastive loss and distribution alignment loss during local training, constraining features from each client to align to the shared space.

[0034] S33: Dynamically balance the weight coefficients of cross-modal tasks and alignment objectives to improve the model convergence stability in non-IID scenarios.

[0035] In one example of the present invention, step S40 further includes:

[0036] S41: Freeze the backbone parameters of the cross-modal base model and perform local training and global aggregation only on the feature adaptation module;

[0037] S42: Combining gradient quantization, parameter truncation, or low-rank approximation techniques to further compress communication transmission volume.

[0038] Another objective of this invention is to propose a computationally and communication-efficient cross-modal model collaborative training system, comprising:

[0039] The lightweight model building module is configured to use a hybrid network structure that integrates local perception and global semantic association, and combines structural reparameterization technology to compress the model size in order to build a lightweight cross-modal basic model architecture.

[0040] The simplified model building module is configured to build a simplified feature adaptation module. A simplified feature adaptation module is added to the output of the cross-modal basic model architecture. It is configured to achieve local feature differentiation optimization by gradually reducing its parameter count to a simplified linear transformation form, while reducing the amount of communication transmission data.

[0041] The feature alignment module is configured to minimize the distribution distance loss function by comparing the feature distribution differences between the public reference dataset and the local data during local training, thereby promoting the gradual alignment of the feature spaces of each participant.

[0042] The parameter aggregation and update module is configured to adapt the global feature parameters maintained on the server side to the module. Each participant fine-tunes the module based on local data and then uploads the updated data. Global model optimization is achieved through weighted averaging.

[0043] In one example of the present invention, the lightweight model building module includes:

[0044] The computational balancing unit is configured to fuse convolutional networks and self-attention mechanisms in the image encoder, thereby balancing computational efficiency and expressive power through enhanced local perception and global semantic association.

[0045] The computation simplification unit is configured to use equivalent reparameterization techniques during the training and inference phases to transform complex network structures into a compact form, thereby reducing floating-point operations and parameter size.

[0046] Lightweight units are configured to use lightweight operations instead of fully connected layers to reduce computational complexity.

[0047] In one example of the present invention, the minimalist model building module includes:

[0048] The feature weighting unit is configured to generate a feature attention mask by channel-wise scaling or non-linear mapping, and to perform weighted adjustments on the original features;

[0049] The extremely simplified unit is configured to use a linear transformation layer with fewer than a preset threshold of parameters as an extremely simplified form of the adaptation module, while retaining key discrimination information in conjunction with the information bottleneck theory.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] Reduce local computing resource consumption: Through structural reparameterization technology and lightweight module design, the floating-point operation volume and memory usage of edge devices are significantly reduced, making it suitable for low-computing-power devices.

[0052] Enhanced privacy protection: Only sparse parameters of the minimal feature adaptation module are transmitted, avoiding exposure of raw data or complete model gradients and reducing the risk of sensitive information leakage.

[0053] Improve model generalization and adaptability: By combining domain adaptive alignment and contrastive learning, the feature shift problem of non-independent identically distributed (Non-IID) data can be effectively alleviated, and the cross-scene generalization ability can be enhanced.

[0054] Supports collaborative training of heterogeneous devices: The minimalist architecture design is compatible with edge nodes with different hardware capabilities (such as IoT devices and mobile terminals), enabling seamless collaboration of large-scale heterogeneous devices.

[0055] Dynamically optimize training stability: By dynamically adjusting the weight coefficients, the goal of cross-modal tasks and distribution alignment is balanced, client drift is suppressed, and convergence efficiency and robustness are improved.

[0056] Efficient end-to-end deployment: The lightweight backbone and modular design support rapid inference and real-time updates, meeting the application requirements of industrial-grade low latency and high responsiveness.

[0057] The preferred embodiments of the invention will be described in more detail below with reference to the accompanying drawings, so as to facilitate an understanding of the features and advantages of the invention. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. The drawings are merely illustrative of some embodiments of the present invention and are not intended to limit the scope of the present invention to all embodiments.

[0059] Figure 1 A flowchart illustrating a cross-modal model collaborative training method for computational and communication efficiency according to an embodiment of the present invention;

[0060] Figure 2 This is a structural framework diagram of Mobiflip according to an embodiment of the present invention;

[0061] Figure 3 This is an architecture diagram of a lightweight adapter module according to an embodiment of the present invention;

[0062] Figure 4 Example images extracted from CIFAR-10, CIFAR-100, and brain tumor classification according to embodiments of the present invention (shown in the first, second, and third rows, respectively);

[0063] Figure 5 The figure shows the experimental results of key factors according to an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The same reference numerals in the drawings represent the same components. It should be noted that the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0065] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, “an” or “a” and similar terms do not necessarily indicate a quantity limitation. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships; these relative positional relationships may change accordingly when the absolute position of the described object changes.

[0066] According to a first aspect of the present invention, a computational and communication-efficient cross-modal model collaborative training method is provided, such as... Figure 1 As shown, it includes the following steps:

[0067] S10: Constructing a lightweight cross-modal basic model architecture: Adopting a hybrid network structure that integrates local perception and global semantic association, and combining structural reparameterization technology to compress the model size to construct a lightweight cross-modal basic model architecture; that is, deploying the hybrid architecture model on edge devices, and using structural reparameterization technology to convert the multi-branch network in the training stage into a single-path structure in the inference stage, reducing memory consumption; using dynamic token hybridization technology to replace the fully connected layer, reducing the floating-point operation of the image encoder and improving the real-time inference speed.

[0068] S20: Constructing a simplified feature adaptation module: A simplified feature adaptation module is added to the output of the cross-modal basic model architecture. By gradually reducing its parameter count to a simplified linear transformation form, local feature differentiation optimization is achieved, while reducing the amount of data transmitted in communication. A linear transformation layer is added to the output of the cross-modal model to generate channel attention masks and perform element-wise weighting of features. An end-to-end training strategy is adopted to jointly optimize the parameters of the adaptation module and the cross-modal contrast loss, while introducing information bottleneck constraints to prevent overfitting.

[0069] S30: Introducing a distributed feature alignment strategy: During local training, by comparing the feature distribution differences between the public reference dataset and the local data, the distribution distance loss function is minimized, prompting the feature spaces of each participant to gradually align; Each client downloads the global adaptation module parameters, fine-tunes them on the local data, calculates the distribution difference loss with the public reference data, updates the module parameters, and uploads them;

[0070] S40: Distributed Parameter Aggregation and Update: The server maintains global feature adaptation module parameters. Each participant fine-tunes the module based on local data and uploads the updated parameters. Global model optimization is achieved through weighted averaging. The server performs weighted averaging on client parameters and, combined with gradient quantization technology, compresses the amount of data in a single round of communication to less than 1% of the original model parameters.

[0071] This method reduces local computing resource consumption: through structural reparameterization technology and lightweight module design, it significantly reduces the floating-point operation volume and memory usage of edge devices, making it suitable for low-computing-power devices.

[0072] This method enhances privacy protection by transmitting only the sparse parameters of the minimal feature adaptation module, avoiding the exposure of the original data or the gradient of the complete model, and reducing the risk of sensitive information leakage.

[0073] This method improves the model's generalization and adaptability: by combining domain adaptive alignment and contrastive learning, it effectively alleviates the feature shift problem of non-independent identically distributed (Non-IID) data and enhances cross-scene generalization ability.

[0074] This method supports collaborative training of heterogeneous devices: the minimalist architecture design is compatible with edge nodes with different hardware capabilities (such as IoT devices and mobile terminals), enabling seamless collaboration of large-scale heterogeneous devices.

[0075] This method dynamically optimizes training stability: by dynamically adjusting the weight coefficients, it balances cross-modal tasks and distribution alignment goals, suppresses client drift, and improves convergence efficiency and robustness.

[0076] This method enables efficient end-to-end deployment: the lightweight backbone and modular design support rapid inference and real-time updates, meeting the application requirements of industrial-grade low latency and high responsiveness.

[0077] In one example of the present invention, step S10 specifically includes the following steps:

[0078] In the image encoder, convolutional networks and self-attention mechanisms are integrated to enhance computational efficiency and expressive power through local perception and global semantic association.

[0079] In the training and inference phases, equivalent reparameterization techniques are used to transform complex network structures into compact forms, thereby reducing floating-point operations and parameter size.

[0080] Lightweight operations are used instead of fully connected layers to reduce computational complexity.

[0081] Specifically, such as Figure 2As shown, the lightweight cross-modal base model architecture is the efficient MobileCLIP model architecture, and the basic backbone network used by Mobiflip is the lightweight MobileCLIP architecture. Traditional CLIP models often use large-scale ViT or pure Transformers. Although they have achieved excellent performance in multimodal contrastive learning, their computational and storage overhead is too high for mobile scenarios. To address this, MobileCLIP introduces a CNN-Transformer hybrid structure in the image and text encoders: CNN provides local receptive fields and fewer parameters, while Transformer is responsible for global semantic association. The combination of the two avoids the excessive computational burden of pure self-attention while maintaining good expressive power.

[0082] Furthermore, to accelerate inference and reduce redundant computation, MobileCLIP employs an equivalent but more compact reparameterized structure in both the training and inference phases, replacing some fully connected transformations with 1×1 convolutional token hybrids, thereby reducing floating-point operations (FLOPs) and parameter size. Experiments show that MobileCLIP-S0 achieves approximately 5 times faster inference speed than the standard ViT-B / 16, while having only about one-third the number of model parameters. By combining CNN-Transformer and structural reparameterization, MobileCLIP significantly reduces training and inference overhead, giving Mobiflip a clear advantage when deployed on edge devices.

[0083] In one example of the present invention, step S10, which combines structural reparameterization technology to compress the model size, specifically includes the following steps:

[0084] Transform the multi-branch network in the training phase into a single-path structure in the inference phase;

[0085] Replacing some fully connected layers with 1×1 convolutions reduces parameter redundancy and computational complexity.

[0086] In one example of the present invention, step S20, constructing the minimalist feature adaptation module specifically includes the following steps:

[0087] S21: Generate feature attention masks by scaling or non-linear mapping through channels, and then perform weighted adjustments on the original features;

[0088] S22: A linear transformation layer with fewer than a preset threshold of parameters is used as an extremely simplified form of the adaptation module, while retaining key discrimination information in conjunction with the information bottleneck theory.

[0089] In federated learning, some studies have proposed "fine-tuning only a small sub-module (such as attention, prompt, adapter, etc.)" to reduce communication costs, rather than transmitting the entire model. However, such solutions may still be "not small enough" in some scenarios, or fail to adequately remove redundancy. To further reduce the client's communication and computational burden in federated learning, Mobiflip introduces a minimalist adaptation module at the output of the MobileCLIP image encoder to capture personalized feature differences in local client data. In Mobiflip, the structure of this lightweight module is progressively reduced, even simplified to an extreme 1×1 convolution, thus maintaining good expressive power under extremely limited edge bandwidth and computing power conditions. Figure 3 The simplified process from two linear layers → one linear layer → 1×1 convolution is demonstrated. To clarify the theoretical basis, the Information Bottleneck (IB) theory is introduced to explain why a "smaller" IIB can still "retain key information".

[0090] Information theory (IB) theory reveals the balance between "compression" and "preservation of task-related information" from an information theory perspective. Its classic objective function is shown in the following formula:

[0091]

[0092] In the formula, I(X;T) measures the mutual information between the input X and the intermediate representation T, I(T;Y) represents the discriminative power of T for the predicted target Y, and β controls the trade-off between information retention and compression. Intuitively, if task-irrelevant noise or redundant information can be effectively "squeezed out," the same multimodal tasks such as classification / retrieval can be completed with fewer parameters and less communication.

[0093] Suppose that the MobileCLIP image encoder outputs D-dimensional features f. We add an adaptation module to this, obtaining an attention mask m and performing element-wise weighting, as shown in the following formula:

[0094] f ′ =f⊙m

[0095] In the formula, ⊙ represents the Hadamard product.

[0096] like Figure 3As shown, the initial design consisted of two linear layers (including Batch Normalization, activation, and Softmax), which were then gradually simplified to a single linear layer, and finally refined to a 1×1 convolution with comparable parameter count but more direct control over the channels. During this process, we found that even when the parameter scale of a single linear layer and a 1×1 convolution were similar, the 1×1 convolution still significantly outperformed the linear layer. This indicates that while providing strong compression, the convolutional channel scaling method better preserves the core discriminative information in multimodal features. We can generate m using the following two structures:

[0097] Small fully connected network: Includes Batch Normalization (BN) and activation functions, and finally uses Softmax to output an attention mask. Similar to existing adapters, this design has a much smaller number of parameters than the MobileCLIP backbone, but still retains some nonlinearity and channel mixing capabilities.

[0098] 1×1 convolution (extremely lightweight version): Multiply each feature channel by a learnable coefficient, which is equivalent to scaling each dimension of f individually. The number of parameters is only D (2D if bias is included), making it one of the "smallest" submodules that can be done.

[0099] By gradually reducing the number of adaptation modules, it was found that good performance could still be achieved in multimodal FL using a minimal 1×1 convolution. In this way, each client can achieve differentiated tuning of local features without transmitting large-scale updates.

[0100] When there is excessive redundancy in the adaptation modules, some parameters will contribute very little to the global optimization objective during federated training, and can be regarded as "zero space" W. null As shown in the formula below:

[0101]

[0102] By removing this redundancy, the input features are effectively "compressed" within the IB theoretical framework, retaining only the most critical channels for cross-modal contrastive learning and classification. This significantly reduces communication and computational costs without sacrificing accuracy. Furthermore, this minimalist design reduces overfitting and noise perturbations in non-IID scenarios. Ablation experiments also demonstrate that even with only a 1×1 convolutional kernel, Mobiflip maintains performance close to or better than the original backbone in multimodal retrieval and classification tasks, significantly reducing transmission and training overhead in federated learning environments.

[0103] In one example of the present invention, the training process of the minimalist feature adaptation module satisfies the following conditions:

[0104] 1) By constraining the feature compression process through information bottleneck theory, the amount of mutual information related to the task is maximized.

[0105] 2) An end-to-end joint optimization strategy is adopted to synchronously update the parameters of the adaptation module and the cross-modal comparative learning target.

[0106] In one example of the present invention, step S30 specifically includes the following steps:

[0107] S31: Select an unlabeled public dataset and calculate the difference in distribution between it and the local data in the feature space.

[0108] S32: Simultaneously optimize cross-modal contrastive loss and distribution alignment loss during local training, constraining features from each client to align to the shared space.

[0109] S33: Dynamically balance the weight coefficients of cross-modal tasks and alignment objectives to improve the model convergence stability in non-IID scenarios.

[0110] Specifically, this step introduces a distributed feature alignment strategy as the federated training strategy for Mobiflip. The federated training process of the Mobiflip framework consists of a central server and N local clients {C1, C2, ..., C...}. N The server first initializes the weights (θ) of the MobileCLIP backbone model. enc ), and keep it frozen, while simultaneously issuing a minimalist adaptation module (θ) a This allows for local fine-tuning on each client, significantly reducing communication and computation costs during training.

[0111] During the local training phase, each client C i Independently optimize the weights of its adaptation modules The overall loss function L for local training i It consists of three parts, namely, contrastive learning loss L contr Classification loss L cls Domain adaptation loss L DA They are combined through weight parameters α and β, and are collectively defined as follows:

[0112] L i =L contr +αL cls +βL DA

[0113] α and β can be used to flexibly control the proportion of different loss terms.

[0114] Comparison loss L contr This is used to bring matching image-text feature pairs closer together and separate mismatched pairs, thus achieving multimodal feature alignment. Specifically, in the calculation, let the batch size be B, and the image feature of the j-th sample be represented as... Text features are represented as T j′ The cosine similarity between the two is defined by the following formula:

[0115]

[0116] Then, construct the image-to-text matching probability matrix P = softmax(S / τ) and the text-to-image matching probability matrix Q = softmax(S / τ). T / τ), where Let be the similarity matrix among all samples, and τ be the temperature parameter. Then the contrast loss can be expressed as follows:

[0117]

[0118] In the formula, p j,j and q j,j Let represent the matching probability of the j-th sample under the two perspectives, respectively.

[0119] When there are clear category labels, the client can also introduce classification loss L. cls As an additional supervisory signal. Given the predicted probability distribution. Compared with the real one-hot label vector y j,c The classification loss is calculated using the cross-entropy function, as shown below:

[0120]

[0121] In the formula, K represents the total number of categories in the classification task.

[0122] In addition, to further mitigate the differences in data distribution among clients, clients can add domain adaptation loss locally. DA The source domain D is the data from the local client. i The unlabeled public dataset is the target domain D. t Based on the Local Maximum Mean Difference (LMMD) method, the domain adaptation loss is defined as follows:

[0123]

[0124] Where φ(·) represents the Hilbert space mapping of the feature, and It represents the membership weight of samples to category c in each domain.

[0125] After completing several rounds of local training, the client will update the adaptation module parameters. The data is uploaded to the server for aggregation. The server's aggregation is based on the training data size n of each client. i The weighted average is calculated as shown in the following formula:

[0126] In the formula,

[0127] Subsequently, the server will update the global parameter θ a The data is then redistributed to all clients for the next iteration. This aggregation method, based on a minimalist adaptation module, effectively reduces communication overhead and achieves good generalization capability for multi-source data after multiple iterations.

[0128] In other words, the maximum mean difference between local data and public reference data in the regenerated kernel Hilbert space is calculated as an additional loss function constraint; by jointly optimizing the cross-modal contrastive loss and the distribution alignment loss, the convergence stability of the model in non-independent and identically distributed scenarios is improved.

[0129] In one example of the present invention, step S40 specifically includes:

[0130] During FL training, the server maintains the globally adaptable module parameters φ. (t) In round t, each client i downloads φ (t) And in local data D i Several updates were performed to obtain The data is then uploaded to the server for weighted averaging, as shown in the formula:

[0131]

[0132] In the formula, n i N = ∑ (This is the number of client samples) i n i Because only φ is transmitted while the MobileCLIP backbone (θ) is frozen, the communication overhead is significantly reduced. Further compression can be achieved by truncating or quantizing gradients or weight differences before transmission, or by reducing the global aggregation frequency, thereby reducing communication overhead to an even lower order of magnitude.

[0133] Since simply reducing the amount of data transmitted cannot resolve the performance loss caused by significant data distribution differences (non-IID) between clients, a Local Maximum Mean Difference (LMMD) loss is incorporated into local training. The client's objective function is given by the formula:

[0134]

[0135] In the formula, l CL It is the image-text contrast loss, D g Indicates unlabeled public reference data, l LMMD Used to measure client data D i and D g In the regenerating nucleus Hilbert space The distribution differences within the region are shown in the formula:

[0136]

[0137] By minimizing this difference, the features learned by each client can be aligned to a shared distribution, thereby mitigating the bias caused by differences between devices or populations. Combined with the communication advantages of small adaptation modules, client update differences are smoother, and local oscillations and mutual cancellations are reduced during FedAvg aggregation.

[0138] It should be noted that the above content explores how to perform communication compression and add domain alignment in federated scenarios based on small or even extremely small (1×1 convolution) adapter modules in order to maintain high accuracy under non-IID data.

[0139] In one example of the present invention, step S40 further includes:

[0140] S41: Freeze the backbone parameters of the cross-modal base model and perform local training and global aggregation only on the feature adaptation module;

[0141] S42: Combining gradient quantization, parameter truncation, or low-rank approximation techniques to further compress communication transmission volume.

[0142] According to a second aspect of the present invention, a computationally and communication-efficient cross-modal model collaborative training system comprises:

[0143] The lightweight model building module is configured to use a hybrid network structure that integrates local perception and global semantic association, and combines structural reparameterization technology to compress the model size in order to build a lightweight cross-modal basic model architecture.

[0144] The simplified model building module is configured to add a simplified feature adaptation module to the output of the cross-modal basic model architecture. It is configured to achieve local feature differentiation optimization by gradually reducing its parameter count to a simplified linear transformation form, while reducing the amount of communication transmission data.

[0145] The feature alignment module is configured to minimize the distribution distance loss function by comparing the feature distribution differences between the public reference dataset and the local data during local training, thereby promoting the gradual alignment of the feature spaces of each participant.

[0146] The parameter aggregation and update module is configured to adapt the global feature parameters maintained on the server side to the module. Each participant fine-tunes the module based on local data and then uploads the updated data. Global model optimization is achieved through weighted averaging.

[0147] This system reduces local computing resource consumption: through structural reparameterization technology and lightweight module design, it significantly reduces the floating-point operation volume and memory usage of edge devices, making it suitable for low-computing-power devices.

[0148] The system enhances privacy protection by transmitting only the sparse parameters of the minimal feature adaptation module, avoiding the exposure of raw data or complete model gradients and reducing the risk of sensitive information leakage.

[0149] This system improves the generalization and adaptability of the model: by combining domain adaptive alignment and contrastive learning, it effectively alleviates the feature shift problem of non-independent identically distributed (Non-IID) data and enhances cross-scene generalization ability.

[0150] The system supports collaborative training of heterogeneous devices: its minimalist architecture design is compatible with edge nodes with different hardware capabilities (such as IoT devices and mobile terminals), enabling seamless collaboration of large-scale heterogeneous devices.

[0151] The system dynamically optimizes training stability: by dynamically adjusting the weight coefficients, it balances cross-modal tasks and distribution alignment goals, suppresses client drift, and improves convergence efficiency and robustness.

[0152] The system is deployed efficiently end-to-end: its lightweight backbone and modular design support rapid inference and real-time updates, meeting the application requirements of industrial-grade low latency and high responsiveness.

[0153] In one example of the present invention, the lightweight model building module includes:

[0154] The computational balancing unit is configured to fuse convolutional networks and self-attention mechanisms in the image encoder, thereby balancing computational efficiency and expressive power through enhanced local perception and global semantic association.

[0155] The computation simplification unit is configured to use equivalent reparameterization techniques during the training and inference phases to transform complex network structures into a compact form, thereby reducing floating-point operations and parameter size.

[0156] Lightweight units are configured to use lightweight operations instead of fully connected layers to reduce computational complexity.

[0157] In one example of the present invention, the minimalist model building module includes:

[0158] The feature weighting unit is configured to generate a feature attention mask by channel-wise scaling or non-linear mapping, and to perform weighted adjustments on the original features;

[0159] The extremely simplified unit is configured to use a linear transformation layer with fewer than a preset threshold of parameters as an extremely simplified form of the adaptation module, while retaining key discrimination information in conjunction with the information bottleneck theory.

[0160] Specific Cases

[0161] This specific implementation systematically evaluates the proposed Mobiflip and several comparative models on multiple datasets, and delves into various factors such as data distribution heterogeneity, number of clients, and number of local training rounds. It mainly includes the datasets used and experimental settings, overall classification performance, analysis of the impact of different numbers of clients and training rounds on the results, and verification of the effectiveness of key modules through ablation experiments.

[0162] Experimental setup:

[0163] This study selected multiple publicly available datasets and set various hyperparameters and comparison models to comprehensively evaluate the performance and transferability of the proposed method under different tasks and environments. Specifically, the experiments used the following datasets: First, CIFAR-10, which contains 60,000 color images (32×32 pixels) across 10 classes, with 50,000 images for training and 10,000 images for testing; second, CIFAR-100, which contains 100 classes, also with 32×32 images, and 50,000 images for training and 10,000 images for testing. In addition, to verify its feasibility in medical imaging scenarios, BT brain tumor data was also introduced into the ablation experiments. The dataset sample is as follows: Figure 4 As shown, this is to test the potential of this method in professional fields.

[0164] In terms of evaluation metrics, this work uses Balanced Accuracy (Bacc) to measure the model's overall performance across all categories. Compared to traditional accuracy, Bacc is more robust to potential class imbalances or heterogeneous distributions and better reflects the true recognition capability in multi-source data.

[0165] The comparison models include:

[0166] 1. MobileCLIP (non-federated version): directly uses the pre-trained MobileCLIP model for inference;

[0167] 2. FedAvg, the most classic federated learning framework, aggregates local updates from various clients;

[0168] 3. FedProx adds a proximal regularization term to the local loss to mitigate client drift;

[0169] 4. MOON integrates contrastive learning into the personalized federated learning process;

[0170] 5. Mobiflip, the minimalist federated adaptive MobileCLIP proposed in this paper, aims to significantly reduce communication and computation by adding a minimalist adaptation module to the back end of the image encoder and freezing the backbone.

[0171] In all experiments, the batch size was uniformly set to 32, the initial learning rate was 6 × 10^(-5), and communication (i.e., global aggregation) was performed only once to focus on the impact of local training on model performance. To simulate a multi-source heterogeneous environment, the number of clients N varied from {1, 5, 10, 20}; the degree of non-IID was controlled by the concentration parameter β ∈ {0.3, 0.6, 1.0} of the Dirichlet distribution, with a smaller β value indicating a more uneven distribution; simultaneously, the number of local training epochs varied from {2, ..., 10} to observe the impact of different numbers of local iterations on model accuracy and convergence. The above settings cover both the extreme case of fewer clients / small epochs and the case of larger number of clients / high epochs, thus evaluating the performance of each algorithm in heterogeneous and resource-constrained environments from multiple dimensions.

[0172] Overall classification performance:

[0173] This study explores the impact of heterogeneity in different data distributions (controlled by the concentration parameter β) on the overall model performance. For easier cross-dataset comparison, Table 1 combines the experimental results of CIFAR-10 and CIFAR-100 under three β ∈ {0.3, 0.6, 1.0} conditions into a single table. The number of clients N and the number of local training epochs are fixed at relatively large values ​​(N = 10, Epoch = 10) to ensure that each method can be fully trained and exhibit steady-state performance. Meanwhile, the accuracy of MobileCLIP (non-federated version) is included in the table to dispel the misconception that "the method's advantage comes solely from the capabilities of the multimodal large model itself": although MobileCLIP originally possesses excellent cross-modal priors, its performance may catastrophically degrade in highly heterogeneous scenarios when faced with non-IID data environments without federated tuning or personalized fine-tuning.

[0174] Table 1: Classification accuracy (Bacc) on CIFAR-10 and CIFAR-100

[0175]

[0176]

[0177] Several points can be observed from the table. First, regardless of whether it's CIFAR-10 or CIFAR-100, small to medium β values ​​(e.g., β = 0.3 or β = 0.6) correspond to a higher degree of non-IID. At this point, Mobiflip achieves optimal or near-optimal accuracy across all three distributions, significantly outperforming traditional federated methods like FedAvg and FedProx, and far surpassing MobileCLIP (without federation). This indicates that when data distributions are highly diverse and heterogeneous, relying solely on a single model for inference on multi-source data (such as MobileCLIP) cannot adapt to the feature distributions of different clients. Integrating multi-source information through a federated framework can significantly improve accuracy. More importantly, Mobiflip maintains a high level of accuracy even when the concentration parameter β is extremely small, further demonstrating that its minimalist adaptation module, designed under the guidance of information bottleneck theory, strongly promotes feature alignment and avoids severe drift or forgetting due to large differences in client distributions.

[0178] Secondly, comparing MOON, which also incorporates contrastive learning, we can see that MOON achieves some improvement even with a small or moderate β (e.g., CIFAR-10, β = 0.6), but still lags behind Mobiflip by about 0.35 in performance. The fundamental reason is that MOON focuses on personalized contrastive loss but still requires large-scale updates to the backbone network. If the client distribution varies significantly, it may repeatedly cover some effective representations in multiple iterations. In contrast, Mobiflip, by freezing most parameters of the backbone MobileCLIP and adding a minimalist adaptation module in the backend, not only avoids large-scale parameter oscillations but also quickly extracts differentiated features locally, minimizing noise and redundant channels. As explained in the methodology section, this approach reduces communication and computational burden while preserving key discriminative information.

[0179] Finally, on the CIFAR-100 dataset, when β = 1.0 (relatively balanced distribution), Mobiflip still maintains a leading advantage over other algorithms, which indirectly proves that its design is not only for highly heterogeneous scenarios. When the data is relatively uniform, large-scale parameter sharing also has certain advantages, but it is still not as robust as Mobiflip's combination of "minimalist adaptation + powerful backbone".

[0180] Key Factor Impact Analysis:

[0181] The experiment focused on two key dimensions: the number of clients (N) and the number of local training epochs. Figure 5In this paper, we present experimental results by combining 3D surfaces and 2D slices to comprehensively demonstrate the performance of various algorithms (FedAvg, FedProx, MooN, and Mobiflip) on CIFAR-10 and CIFAR-100 under different N and Epoch conditions. The 3D surface provides a macroscopic observation of the overall trend, while the corresponding 2D plots are several typical "slices" extracted from the surface. By combining the two for analysis, we can not only see the stability and efficiency of Mobiflip in multi-source data and multi-round training, but also theoretically understand the advantages of its adaptation module in suppressing overfitting and client drift.

[0182] First, it's immediately apparent from the 3D surface that as the number of clients N increases, traditional federated methods (such as FedAvg, FedProx, and MooN) often exhibit significant jitter or degradation when dealing with non-IID distributions: when N exceeds a certain scale, the surface shows significant sinking or uneven gradients, meaning that data differences between clients are amplified, easily leading to global aggregation getting trapped in local optima. In contrast, the surface corresponding to Mobiflip remains relatively smooth and at a higher level, exhibiting a relatively stable high Bacc region in both small-client and large-client scenarios from a 3D perspective. Figure 5 The two-dimensional slicing in the data further confirms this: On the CIFAR-10 dataset, when N increases from 1 to 5, 10, and 20, Mobiflip's accuracy remains consistently around 0.94 to 0.95, while methods such as FedAvg struggle to even break through 0.56, indicating that Mobiflip more fully converts the diversity of multi-source data into representational power; On CIFAR-100, Mobiflip also far outperforms other methods when N=1, and its performance is not severely degraded due to distribution differences as the number of clients increases.

[0183] Secondly, regarding the number of local training epochs, the 3D surface analysis shows that if the epoch is too small, traditional methods have limited ability to fit local data, and global aggregation cannot fully absorb local differences. However, if the epoch is too high, the surfaces of models such as FedAvg and FedProx often show tilting or local concavity, reflecting the continuous accumulation of "client drift" or overfitting risks. In contrast, Mobiflip's surface consistently maintains good smoothness, and the curve does not show obvious oscillations in the mid-to-high epoch region. This indicates that it can more accurately calibrate local features through a minimalist adaptation module, while preserving the cross-modal representation capabilities of the original MobileCLIP backbone to the greatest extent. Therefore, it avoids the "catastrophic forgetting" or amplification of differences caused by large-scale parameter updates during global aggregation.

[0184] Figure 5The corresponding two-dimensional slices also corroborate the above inference: In the CIFAR-10 scenario (1 client), Mobiflip's curve reaches 0.79 from Epoch=2 and steadily climbs to 0.83-0.84 in subsequent iterations. This echoes the theoretical analysis of its lightweight adaptive network in the method section—by leveraging information bottlenecks and minimal convolution, the client can quickly and efficiently align local data distributions, compressing irrelevant noise and reducing interference with the backbone model. In contrast, FedAvg, MooN, and others require more epochs to barely break through the 0.55-0.56 level. This is because these methods often perform joint updates on large-scale parameters. Once distribution conflicts or instabilities occur in multiple rounds of aggregation, the effective representations learned in the early stages may be frequently overwritten or forgotten. As for the CIFAR-100 scenario (10 clients), Mobiflip also shows its advantage at the low Epoch stage. As the Epoch increases, the curve's growth slows down, but it is still significantly higher than other methods by about 0.03-0.08. This further illustrates that Mobiflip's minimalist adaptation strategy can quickly grasp local feature differences on the one hand, and avoid repeated rewriting of the MobileCLIP backbone on the other hand, thus maintaining a high level of cross-modal recognition capability within a small number of epochs.

[0185] It is worth noting that as the number of local training epochs increases, Mobiflip's performance does not show a significant second leap, but rather stabilizes relatively early. This is also explained in the theoretical analysis in the methodology section: because the minimal adaptation module, guided by the information bottleneck (IB) theory, can quickly extract and retain the most relevant discriminative features for the task, most redundant information is effectively filtered out in the early stages, thus enabling the model to establish a relatively complete cross-modal alignment and local data adaptation capability within a few epochs. In addition, the strong representational foundation of the backbone MobileCLIP has not been significantly modified, so subsequent epochs often only bring slight gains, and may even cause slight oscillations in the curve due to small-scale noise. Precisely because it achieves a 'high starting point' Bacc in the early stages, Mobiflip can also exert a significant advantage in resource-constrained environments or environments with high real-time response requirements, achieving faster convergence and efficiently mining the potential of multi-source heterogeneous data.

[0186] Ablation experiment:

[0187] This study explores the practical role of the minimal adaptation module in the Mobiflip method, focusing on examining the performance differences of the model on various datasets when the adaptation structure and parameter scale differ. To maintain comparability, we uniformly set the number of clients N=10, β=1.0, and Epoch=10 on the CIFAR-10, CIFAR-100, and Brain Tumor (BT) datasets, changing only the adaptation structure used; the three structures compared are as follows:

[0188] 1. 1×1 convolution: Scaling each channel individually, with the fewest parameters;

[0189] 2. Two linear layers: More trainable parameters are added to the adaptation module in order to improve the flexibility of the representation;

[0190] 3. A single linear layer: The number of parameters is similar to that of convolution, but it lacks channel-level operations.

[0191] Table 2 shows the accuracy and corresponding parameter count (Parm) of these three structures on various datasets. It can be seen that the 1×1 convolution achieves the best balance between accuracy and scale: achieving an accuracy of 0.9484 on CIFAR-10, 0.3756 on CIFAR-100, and 0.9143 on the brain tumor (BT) dataset, requiring only about 0.262M parameters. In contrast, using two or one linear layer did not yield better results, and even resulted in a decline in accuracy on some datasets (such as CIFAR-10 and CIFAR-100), indicating that more complex adaptations do not necessarily equate to higher accuracy.

[0192] Table 2: Ablation experiments: Accuracy and parameter values ​​(Bacc and parameter values) for different fitting structures

[0193]

[0194] Based on the above results, we can see that, on the one hand, strong compression does not compromise discriminative ability: guided by the information bottleneck theory, filtering and scaling the channel dimension through 1×1 convolution allows the model to effectively retain task-related features and suppress interference noise. On the other hand, more complex structures are not necessarily better. Compared to "minimalist adaptation," although the parameter scale of two or one linear layer is larger, the accuracy is not significantly improved, indicating that in cross-modal or cross-domain scenarios, fine-grained operations at the channel level are often more efficient. Meanwhile, in medical scenarios, brain tumor (BT) data also confirms the feasibility of 1×1 convolutional adaptation, achieving an accuracy of 0.9143, providing feasibility support for subsequent deployment in real medical environments. In conclusion, the simple and efficient convolutional adaptation scheme has broad application potential in resource-constrained devices and heterogeneous federated learning environments.

[0195] The foregoing description, with reference to preferred embodiments, details exemplary implementations of the computational and communication efficiency cross-modal model collaborative training method and system proposed in this invention. However, those skilled in the art will understand that various modifications and alterations can be made to the above specific embodiments without departing from the concept of this invention, and various combinations can be made to the various technical features and structures proposed in this invention without exceeding the protection scope of this invention, which is determined by the appended claims.

Claims

1. A computationally and communication-efficient cross-modal model collaborative training method, characterized in that, Includes the following steps: S10: Constructing a lightweight cross-modal basic model architecture: Adopting a hybrid network structure that integrates local perception and global semantic association, and combining structural reparameterization technology to compress the model size to construct a lightweight cross-modal basic model architecture; S20: Construct a simplified feature adaptation module: Add a simplified feature adaptation module to the output of the cross-modal basic model architecture, and gradually simplify the simplified feature adaptation module from a two-layer linear layer structure to a one-layer linear layer structure, and then further simplify it to a 1×1 convolutional structure to achieve local feature differentiation optimization, while reducing the amount of communication transmission data. S30: Introducing a distributed feature alignment strategy: During local training, by comparing the feature distribution differences between the public reference dataset and the local data, the distribution distance loss function is minimized, which promotes the gradual alignment of the feature spaces of each participant. S40: Distributed parameter aggregation and update: The server maintains the parameters of the global feature adaptation module. Each participant fine-tunes the module based on local data and then uploads the update. Global model optimization is achieved through weighted averaging.

2. The computational and communication efficiency cross-modal model collaborative training method according to claim 1, characterized in that, Step S10 specifically includes the following steps: In the image encoder, convolutional networks and self-attention mechanisms are integrated to enhance computational efficiency and expressive power through local perception and global semantic association. In the training and inference phases, equivalent reparameterization techniques are used to transform complex network structures into compact forms, thereby reducing floating-point operations and parameter size. Lightweight operations are used instead of fully connected layers to reduce computational complexity.

3. The computational and communication efficiency cross-modal model collaborative training method according to claim 1, characterized in that, In step S10, compressing the model size using structural reparameterization techniques specifically includes the following steps: Transform the multi-branch network in the training phase into a single-path structure in the inference phase; Replacing some fully connected layers with 1×1 convolutions reduces parameter redundancy and computational complexity.

4. The computational and communication efficiency cross-modal model collaborative training method according to claim 1, characterized in that, In step S20, constructing the minimalist feature adaptation module specifically includes the following steps: S21: Generate feature attention masks by scaling or non-linear mapping through channels, and then perform weighted adjustments on the original features; S22: A linear transformation layer with fewer than a preset threshold of parameters is used as an extremely simplified form of the adaptation module, while retaining key discrimination information in conjunction with the information bottleneck theory.

5. The computational and communication efficiency cross-modal model collaborative training method according to claim 4, characterized in that, The training process of the minimalist feature adaptation module satisfies the following conditions: 1) By constraining the feature compression process through information bottleneck theory, the amount of mutual information related to the task is maximized. 2) An end-to-end joint optimization strategy is adopted to synchronously update the parameters of the adaptation module and the cross-modal comparative learning target.

6. The computational and communication efficiency cross-modal model collaborative training method according to claim 1, characterized in that, Step S30 specifically includes the following steps: S31: Select an unlabeled public dataset and calculate the difference in distribution between it and the local data in the feature space; S32: Simultaneously optimize cross-modal contrastive loss and distribution alignment loss during local training, constraining features from each client to align to the shared space; S33: Dynamically balance the weight coefficients of cross-modal tasks and alignment objectives to improve the model convergence stability in non-IID scenarios.

7. The computational and communication efficiency cross-modal model collaborative training method according to claim 1, characterized in that, Step S40 further includes: S41: Freeze the backbone parameters of the cross-modal base model and perform local training and global aggregation only on the feature adaptation module; S42: Combining gradient quantization, parameter truncation, or low-rank approximation techniques to further compress communication transmission volume.

8. A computationally and communication-efficient cross-modal model collaborative training system, characterized in that, include: The lightweight model building module is configured to use a hybrid network structure that integrates local perception and global semantic association, and combines structural reparameterization technology to compress the model size in order to build a lightweight cross-modal basic model architecture. The simplified model building module is configured to add a simplified feature adaptation module to the output of the cross-modal basic model architecture. The simplified feature adaptation module is gradually simplified from a two-layer linear layer structure to a one-layer linear layer structure, and then further simplified to a 1×1 convolutional structure to achieve local feature differentiation optimization while reducing the amount of communication transmission data. The feature alignment module is configured to minimize the distribution distance loss function by comparing the feature distribution differences between the public reference dataset and the local data during local training, thereby promoting the gradual alignment of the feature spaces of each participant. The parameter aggregation and update module is configured to adapt the global feature parameters maintained on the server side to the module. Each participant fine-tunes the module based on local data and then uploads the updated data. Global model optimization is achieved through weighted averaging.

9. The computational and communication efficiency cross-modal model collaborative training system according to claim 8, characterized in that, The lightweight model building module includes: The computational balancing unit is configured to fuse convolutional networks and self-attention mechanisms in the image encoder, thereby balancing computational efficiency and expressive power through enhanced local perception and global semantic association. The computation simplification unit is configured to use equivalent reparameterization techniques during the training and inference phases to transform complex network structures into a compact form, thereby reducing floating-point operations and parameter size. Lightweight units are configured to use lightweight operations instead of fully connected layers to reduce computational complexity.

10. The computational and communication efficiency cross-modal model collaborative training system according to claim 8, characterized in that, The minimalist model building module includes: The feature weighting unit is configured to generate a feature attention mask by channel-wise scaling or nonlinear mapping, and to adjust the weight of the original features. The extreme simplification unit is configured to use a linear transformation layer with fewer than a preset threshold of parameters as an extreme simplification form of the adaptation module, and retain key discrimination information in conjunction with the information bottleneck theory.

Images