A Federated Node Collaborative Computation Method for Multimodal Data

Abstract

This application relates to the fields of network security and information security technology, specifically disclosing a federated node collaborative computing method for multimodal data. First, it constructs lightweight low-rank adapter parameters based on a pre-trained backbone configuration and distributes them to edge nodes to be loaded into frozen backbone network bypasses. During the local computation phase, a two-stream mechanism is used to extract features from the multimodal data, and directional backpropagation is performed only for the bypass adapters, thus avoiding gradient calculation for all parameters. Furthermore, adaptive sparse pruning and entropy coding compression are applied to the original adapter gradients to generate a very small compressed gradient data packet, which is then uploaded to the cloud. The server performs weighted aggregation of the decompressed gradients to update the global adapter, achieving a reduction in communication load and transmission latency during the federated collaboration process while ensuring model performance.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of artificial intelligence and distributed computing, and more specifically, to a federated node collaborative computing method for multimodal data. Background Technology

[0002] With the deepening application of artificial intelligence technology in the fields of the Internet of Things and edge computing, collaborative model training using multimodal data (such as visual images, text descriptions, and sensor signals) distributed across edge nodes has become a key path for building high-performance intelligent systems such as smart cities, autonomous driving, and the Industrial Internet. Federated learning, as a privacy-preserving computing paradigm, allows nodes to collaboratively optimize a global model without sharing raw data, effectively solving the data silo problem of multimodal data while ensuring data security and privacy compliance.

[0003] However, existing federated collaborative methods for multimodal data face severe communication bandwidth bottlenecks in practical deployments. First, current mainstream multimodal deep neural networks (such as large models based on the Transformer architecture) have extremely large parameter counts, typically reaching hundreds of millions or even billions. In traditional federated learning frameworks (such as FedAvg), edge nodes need to upload all model parameters or complete gradient updates in each round of collaborative training. This not only consumes enormous network bandwidth but also leads to extremely high communication latency, severely restricting the system's real-time response capability in bandwidth-constrained or unstable edge environments (such as 4G / 5G and V2X networks). Second, existing collaborative mechanisms often employ a full-scale fine-tuning strategy, failing to fully utilize the parameter redundancy characteristics of large multimodal models in specific tasks—that is, when facing a specific downstream task, most of the general knowledge in the pre-trained backbone does not need to be changed, and the full transmission contains a large amount of invalid or low-value gradient information, resulting in a significant waste of communication resources. Furthermore, although some existing technologies attempt to use model compression, simple gradient quantization or random sparsification often fails to maintain model accuracy under extremely low bandwidth and lacks adaptive adjustment mechanisms for federated heterogeneous environments, making it difficult to achieve an ideal balance between communication efficiency and model performance.

[0004] Therefore, an optimized collaborative computing scheme for federated nodes oriented towards multimodal data is desired. Summary of the Invention

[0005] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a federated node collaborative computing method for multimodal data, comprising: S1: Based on the pre-trained backbone configuration and federated task requirements, construct global adapter parameters containing low-rank matrix weights, and distribute them to edge nodes to load into the frozen backbone network to obtain local adapter parameters. S2: Input the collected multimodal data stream into the frozen backbone network loaded with local adapter parameters to obtain the predicted logic value and intermediate layer activation value; S3: Based on the predicted logical value, intermediate layer activation value and local real label, perform loss calculation and directed backpropagation to obtain the original adapter gradient; S4: Based on a preset sparse threshold, the original adapter gradient is sparsely encoded and losslessly compressed to obtain a compressed gradient data packet containing node identifiers; S5: Perform global adapter aggregation on the compressed gradient data packets of multiple edge nodes to update the global adapter parameters, so as to obtain the updated global adapter parameters for the next round of collaborative distribution.

[0006] Compared with existing technologies, this application proposes a federated node collaborative computing method for multimodal data. It constructs global adapter parameters containing low-rank matrix weights, distributes them to edge nodes, and loads them into a frozen pre-trained backbone network bypass. During the local computation phase, a two-stream architecture of the frozen backbone and trainable adapters is used to process multimodal data, performing directional backpropagation only on the adapter branches to obtain low-rank gradients. Subsequently, adaptive sparse pruning and entropy coding techniques are combined to compress the gradients at a high rate, generating lightweight data packets that are uploaded to the server for global aggregation. This scheme achieves decoupling and on-demand transmission of model parameters, reducing communication load while ensuring the collaborative performance of multimodal models in heterogeneous edge environments through continuous iteration of the global adapter. Attached Figure Description

[0007] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0008] Figure 1 This is a flowchart of a federated node collaborative computing method for multimodal data according to an embodiment of this application; Figure 2 This is a data flow diagram illustrating a federated node collaborative computing method for multimodal data according to an embodiment of this application; Figure 3 This is a flowchart illustrating a method for collaborative computing of federated nodes for multimodal data according to an embodiment of this application. It describes how global adapter parameters, including low-rank matrix weights, are constructed based on pre-trained backbone configuration and federated task requirements, and how these parameters are distributed to edge nodes to be loaded into the frozen backbone network to obtain local adapter parameters. Figure 4 This is a flowchart illustrating a federated node collaborative computing method for multimodal data according to an embodiment of this application, which involves loss calculation and directed backpropagation based on predicted logical values, intermediate layer activation values, and local real labels to obtain the original adapter gradient. Figure 5 This is a flowchart illustrating a federated node collaborative computing method for multimodal data based on an embodiment of this application, which involves sparse encoding and lossless compression of the original adapter gradient based on a preset sparse threshold to obtain a compressed gradient data packet containing node identifiers. Figure 6 This is a flowchart illustrating the adaptive sparse pruning of the original adapter gradient to obtain a sparse gradient matrix in a federated node collaborative computing method for multimodal data according to an embodiment of this application. Detailed Implementation

[0009] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0010] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0011] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0012] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0013] Existing federated learning methods for multimodal data face severe challenges in edge collaboration scenarios, primarily due to communication bandwidth bottlenecks and limited computing resources. Since multimodal deep neural networks typically contain hundreds of millions of parameters, traditional full-parameter transmission methods lead to severe network congestion and high latency. Furthermore, existing compression techniques struggle to effectively balance model accuracy and convergence efficiency under extremely low bandwidth conditions. Therefore, this application proposes a federated node collaborative computing method for multimodal data. This method achieves a paradigm shift from full-parameter fine-tuning to lightweight instruction-based collaboration by constructing a decoupled architecture of a frozen backbone and low-rank adaptation. Specifically, firstly, based on the pre-trained backbone configuration and federated task requirements, global adapter parameters containing low-rank matrix weights are constructed and distributed to edge nodes to be loaded into the frozen backbone network bypass. During the local computation phase, the collected multimodal data streams are input into a dual-stream network loaded with local adapter parameters. While extracting general features using the backbone network, semantic biases specific to the task are captured through the bypass adapter. Based on this, directional backpropagation is performed, that is, under the premise of strictly locking the backbone network weights, only the original adapter gradient of the low-rank matrix is ​​derived. Subsequently, based on a preset sparsity threshold, the gradient is adaptively pruned and entropy encoded to compress, generating compressed gradient data packets containing high-value information. Finally, the server performs multi-parallel decompression and gradient reconstruction on the compressed packets uploaded by multiple edge nodes, and performs weighted aggregation to update the global adapter parameters. This reduces the communication load while ensuring the collaborative performance and robustness of the multimodal model in a heterogeneous environment.

[0014] Figure 1 This is a flowchart of a federated node collaborative computing method for multimodal data according to an embodiment of this application. Figure 2 This is a data flow diagram illustrating a federated node collaborative computing method for multimodal data according to an embodiment of this application. Figure 1 and Figure 2As shown, a federated node collaborative computing method for multimodal data according to an embodiment of this application includes: S1, constructing global adapter parameters containing low-rank matrix weights based on pre-trained backbone configuration and federated task requirements, and distributing them to edge nodes to load into the frozen backbone network to obtain local adapter parameters; S2, inputting the collected multimodal data stream into the frozen backbone network loaded with local adapter parameters to obtain predicted logical values ​​and intermediate layer activation values; S3, performing loss calculation and directed backpropagation based on predicted logical values, intermediate layer activation values, and local real labels to obtain the original adapter gradient; S4, performing sparse encoding and lossless compression processing on the original adapter gradient based on a preset sparse threshold to obtain a compressed gradient data packet containing node identifiers; S5, performing global adapter aggregation on the compressed gradient data packets of multiple edge nodes to update the global adapter parameters to obtain updated global adapter parameters for the next round of collaborative distribution.

[0015] Specifically, in step S1, based on the pre-trained backbone configuration and the requirements of the federated task, global adapter parameters containing low-rank matrix weights are constructed and distributed to edge nodes to be loaded into the frozen backbone network to obtain local adapter parameters. It should be noted that, given that in typical multimodal data application scenarios such as smart cities or vehicle-to-everything (V2X) collaboration, edge nodes typically need to run deep neural networks with a large number of parameters to process complex visual and textual information, directly transmitting all model parameters during the federated collaboration process would not only cause severe communication congestion and high latency, but also make it difficult for the limited computing resources of edge devices to support the high energy consumption requirements of full parameter fine-tuning. Therefore, the technical solution of this application first constructs global adapter parameters containing low-rank matrix weights based on the pre-trained backbone configuration and the requirements of the federated task, and then distributes these parameters to edge nodes to be loaded into the frozen backbone network to obtain local adapter parameters. Through the above processing, the general feature extraction capability of large models and the adaptability to specific tasks can be effectively decoupled. By freezing the massive backbone network as a static asset, parameter construction and distribution are only performed on lightweight low-rank adapters. This reduces federated communication load and edge computing overhead while ensuring that heterogeneous nodes can quickly build computing models adapted to local multimodal data distributions based on unified lightweight instructions.

[0016] Figure 3 This document describes a flowchart illustrating a method for collaborative computing among federated nodes for multimodal data, based on pre-trained backbone configuration and federated task requirements. It outlines the process of constructing global adapter parameters containing low-rank matrix weights and distributing them to edge nodes to load into a frozen backbone network, thus obtaining local adapter parameters. (See attached flowchart.) Figure 3As shown, step S1 includes: S11, parsing the pre-trained backbone configuration to determine the hidden layer dimension of the model, and constructing the initial parameters of the global adapter consisting of a dimension reduction matrix and a dimension increase matrix in combination with the requirements of the federated task; S12, traversing the candidate node list to evaluate the online status and computing resources, and removing nodes that do not meet the conditions based on the evaluation results to obtain the selected node list; S13, distributing the initial parameters of the global adapter to each node in the selected node list through the downlink control channel, so that they are instantiated and loaded into the locally pre-stored frozen backbone network bypass to obtain the local adapter parameters.

[0017] In step S11, the pre-trained backbone configuration is parsed to determine the hidden layer dimension of the model, and the initial parameters of the global adapter, consisting of a dimensionality reduction matrix and an increased dimensionality matrix, are constructed in accordance with the requirements of the federated task. It should be noted that since the pre-trained multimodal backbone network has a high-dimensional hidden layer structure, directly fine-tuning or transmitting all parameters during the federated collaboration process would generate a huge computational load and communication overhead. Furthermore, the data heterogeneity between different edge nodes requires the adapter to have flexible adjustment capabilities for specific tasks. Based on this, the technical solution of this application first parses the pre-trained backbone configuration to determine the hidden layer dimension of the model, and then constructs the initial parameters of the global adapter, consisting of a dimensionality reduction matrix and an increased dimensionality matrix, in accordance with the requirements of the federated task. The dimensionality reduction matrix is ​​initialized using a Gaussian distribution, and the increased dimensionality matrix is ​​initialized to all zeros. Through the above processing, the structural decoupling of the model's general feature extraction capability and its specific task adaptation capability can be effectively achieved, ensuring that the constructed low-rank adapter accurately matches the frozen backbone network in terms of dimensionality, and ensuring the numerical stability of the model in the early stages of training through a specific initialization strategy. This reduces the parameter size while laying a convergence foundation for subsequent federated collaborative optimization.

[0018] More specifically, in a concrete example of this application, the hidden layer dimension parameter, representing the feature width of the model, is first extracted by reading a pre-pre-trained model configuration file. The rank of the adapter is then set according to the category and complexity of the current federated learning task; this rank is set to an integer value much smaller than the hidden layer dimension parameter. Based on this, a dimensionality reduction matrix and an increasing dimensionality matrix are constructed to form a low-rank adapter. The dimensionality reduction matrix is ​​configured to map high-dimensional input features to a low-dimensional subspace, while the increasing dimensionality matrix is ​​configured to restore low-dimensional features back to the original hidden layer space. During the initialization phase, targeted parameter assignment operations are performed. The dimensionality reduction matrix is ​​randomly sampled and initialized using a Gaussian distribution with zero mean and a specific variance to introduce initial randomness; simultaneously, all elements of the increasing dimensionality matrix are forcibly initialized to zero. This initialization configuration follows mathematical relationships. ,in B This represents an up-dimensional matrix initialized with all zeros. AThe dimensionality reduction matrix representing Gaussian initialization ensures that the output of the bypass adapter branch is strictly zero at the initial moment of training. This operation guarantees that the inference behavior of the hybrid model after loading the adapter remains completely consistent with the original pre-trained backbone network before the first round of training, preventing random initialization noise from damaging the pre-trained feature representation, thereby ensuring that the model can start from a stable baseline state for federated collaborative optimization of heterogeneous multimodal data.

[0019] In step S12, the candidate node list is traversed to evaluate online status and computing resources, and nodes that do not meet the conditions are eliminated based on the evaluation results to obtain the selected node list. It should be noted that due to the objective heterogeneity and dynamic fluctuations in the physical attributes of each terminal node in the edge computing environment, some nodes may be unable to support the forward inference of the multimodal backbone network and the gradient calculation of the adapter due to limited memory capacity, insufficient remaining power, or unstable network connection. Forcing such nodes to participate in collaboration would lead to training task interruption or lag effects. Therefore, the technical solution of this application further traverses the candidate node list to evaluate online status and computing resources, and eliminates nodes that do not meet the conditions based on the evaluation results to obtain the selected node list. Through the above processing, an admission screening mechanism based on computing capabilities can be effectively established, ensuring that nodes participating in collaboration in each round have the hardware foundation to support the operation of the frozen backbone network and low-rank adapter, thereby eliminating the shortcomings of federated training caused by insufficient node capabilities at the source.

[0020] More specifically, in a concrete example of this application, a communication handshake connection is first established with each node in the candidate node set. Real-time operational data of the nodes is obtained by issuing status probe commands. This operational data includes the node's online heartbeat signal, currently allocable GPU memory size, CPU load rate, and network communication link bandwidth. Next, based on the static memory usage required to load and freeze the backbone network and the dynamic computational overhead required to train the low-rank adapter, minimum admission thresholds for each resource dimension are set. A parallel traversal method is used to perform threshold matching verification on the real-time operational data of each node. During this process, nodes that fail to provide timely heartbeat signals are judged as offline and directly eliminated. Nodes that are online but have available GPU memory below the static memory usage or network bandwidth below the transmission requirements are marked and filtered. Finally, nodes that pass all verification conditions are confirmed as qualified collaborative objects, generating a selected node list for subsequent parameter distribution and task deployment to ensure the hardware-level feasibility of the federated task.

[0021] In step S13, the initial parameters of the global adapter are distributed to each node in the selected node list via the downlink control channel, so that they are instantiated and loaded into the locally pre-stored frozen backbone network bypass to obtain local adapter parameters. It should be noted that since the backbone network parameters of multimodal large models typically reach hundreds of millions, if the full model file is distributed every time a federated task starts, it will consume unacceptable communication bandwidth and transmission time. Furthermore, edge nodes usually have pre-configured common basic model files, and repeated transmission results in a double waste of network and storage resources. In addition, to implement a management mechanism that separates data ownership from usage rights, an efficient interaction method is needed that transmits computational instructions without affecting the underlying static assets. Based on this, the technical solution of this application further distributes the initial parameters of the global adapter to each node in the selected node list via the downlink control channel, so that they are instantiated and loaded into the locally pre-stored frozen backbone network bypass to obtain local adapter parameters. Through the above processing, lightweight distribution of computing instructions can be effectively achieved. By reusing the frozen backbone network stored locally on edge nodes as a static asset, only the adapter parameters with extremely low dimensionality are dynamically updated and distributed as usage instructions. This allows the deployment of specific federated tasks and model assembly to be completed with extremely low latency, avoiding communication congestion caused by the transmission of all parameters.

[0022] More specifically, in a concrete example of this application, the initial parameters of the constructed global adapter are first serialized into a compact binary control message. This message contains only the weight data of the reduced-dimensional and increased-dimensional matrices and the corresponding topological location indices, without any backbone network information. Subsequently, this message is pushed to each edge terminal in the selected node list using a highly reliable downlink control channel. Upon receiving the message, the edge terminal performs a deserialization operation to parse out the weight values ​​of the low-rank matrix and reconstructs them into tensor objects in its local memory. Next, based on a predefined model configuration file, the interfaces of each layer of the backbone network are locked and frozen. The reconstructed tensor objects are then mounted to specific linear layer positions of the backbone network in the form of residual bypasses, thereby constructing the... A parallel computing structure. In this structure, W 0 This represents the frozen backbone weight. B and A Represents the newly loaded dimensionality-upgrading and dimensionality-reducing adapter weights, where α is the scaling factor. r For rank, x This serves as the input data. Finally, after confirming that all bypass branches are loaded correctly, the combined model state is marked as the local adapter parameter to prepare for receiving multimodal data streams for subsequent training calculations.

[0023] Specifically, in step S2, the collected multimodal data stream is input into a frozen backbone network loaded with local adapter parameters to obtain predicted logical values ​​and intermediate layer activation values. It should be noted that, due to the objective heterogeneity of the multimodal data collected by edge nodes in terms of physical attributes and semantic dimensions, and because the pre-trained backbone network is frozen to reduce computational overhead, directly inputting the original data or failing to effectively fuse the features of the backbone and adapter will result in the model being unable to correctly understand the input content or fine-tune it for specific tasks. Based on this, the technical solution of this application further performs modal alignment preprocessing and tensor quantization transformation on the multimodal data stream to obtain a multimodal input tensor; the multimodal input tensor is simultaneously injected into the main path of the frozen backbone network and the bypass path loaded with local adapter parameters to obtain a fused feature vector, while the multimodal input tensor is cached as intermediate layer activation values; the fused feature vector is input into a specific task header for nonlinear mapping and probability distribution estimation to obtain the predicted logical value. Through the above processing, the format differences of heterogeneous data can be effectively eliminated, a unified tensor input standard can be established, and the dynamic fusion of general features and specific task features can be achieved through a two-stream computing mechanism. At the same time, the necessary input state data is reserved for subsequent directed backpropagation only for the adapter, so as to complete the inference and training preparation for local data without updating the backbone parameters.

[0024] More specifically, in a concrete example of this application, standardization processing is first performed on the acquired multimodal raw data, including visual images and text descriptions. For visual data, size normalization based on interpolation and pixel normalization based on channel statistics are performed; for text data, tokenization encoding based on a vocabulary and sequence completion are performed. The processed data is then converted into a multimodal input tensor conforming to the model's input dimensions. Subsequently, a parallel computing architecture is used to feed this input tensor into two computational paths. In the main path, matrix multiplication is performed on the input tensor using a locked pre-trained weight matrix to extract general semantic features. In the side path, a dimensionality reduction matrix is ​​used to map the input tensor to a lower-dimensional space, and then an up-dimensional matrix is ​​used to restore it to the original dimensions to capture task-specific residual features. During this process, at each network layer where the adapter intervenes, the current input tensor is completely preserved and a copy is created in GPU memory as the activation value for intermediate layers. Finally, the general features of the main path and the residual features of the side path are added element-wise to generate a fused feature vector, which is then sent to the classification head or regression head of the specific task. The vector is then processed by a fully connected layer and a Softmax or Sigmoid nonlinear activation function to calculate the predicted logistic values ​​that represent the confidence of each category.

[0025] Specifically, in one example embodiment, the pre-trained backbone network can be a dual-tower structure of a visual encoder and a text encoder: the visual encoder uses ViT-B / 16, and the text encoder uses a 12-layer Transformer. Low-rank adapters are inserted at the linear mapping layers (including Q / K / V projections and output projections) in each Transformer block; the adapter rank... r For example, the scaling factor α can be 4 to 16, and the scaling factor α can be 8 to 32.

[0026] Specifically, in step S3, loss calculation and directed backpropagation are performed based on the predicted logical values, intermediate layer activation values, and local real labels to obtain the original adapter gradient. It should be noted that, given that full parameter fine-tuning of multimodal large models typically involves extremely high memory usage and computational consumption, and that the limited hardware resources of edge nodes make it difficult to support backpropagation updates of the backbone network with hundreds of millions of parameters, and in order to implement the mechanism of separating static asset protection from dynamic instruction updates in federal control, this application's technical solution further performs loss calculation and directed backpropagation based on the predicted logical values, intermediate layer activation values, and local real labels to obtain the original adapter gradient. During this process, the backpropagation engine is activated and the weight attributes of the backbone network are strictly locked and frozen, preventing them from participating in gradient calculation. Only the partial derivatives with respect to the layer outputs are calculated for the bypass adapter branches, and the gradient tensor of the low-rank matrix is ​​derived using the chain rule in conjunction with the cached intermediate layer activation values. Through the above processing, the computational load of model training can be effectively focused on a very small adapter parameter subspace, avoiding the resource squeeze on edge nodes caused by full gradient calculation. At the same time, it ensures that the generated gradient objects only contain incremental update information for specific federated tasks, thus providing a concise and high-value data foundation for subsequent low-bandwidth communication transmission.

[0027] Figure 4 This is a flowchart illustrating a federated node collaborative computation method for multimodal data according to an embodiment of this application, which involves loss calculation and directed backpropagation based on predicted logical values, intermediate layer activation values, and local real labels to obtain the original adapter gradient. Figure 4 As shown, step S3 includes: S31, calculating the difference between the predicted logical value and the local true label to obtain the calculated loss value; S32, starting the backpropagation engine and locking the weight attributes of the frozen backbone network, calculating the partial derivative with respect to the layer output only for the bypass adapter branch to obtain the upstream error gradient; S33, based on the upstream error gradient and the activation value of the intermediate layer, deriving the gradient tensors of the dimension reduction matrix and the dimension increase matrix respectively using the chain rule to obtain the original adapter gradient.

[0028] In step S31, the difference between the predicted logical value and the local true label is calculated to obtain the calculated loss value. It should be noted that since the core of supervised machine learning lies in driving parameter optimization by minimizing the deviation between the model's prediction and the actual data distribution, in a federated edge collaboration scenario, this deviation must be quantified at the local node to provide a differentiable numerical basis for subsequent directional gradient derivation for the low-rank adapter. Based on this, the technical solution of this application further calculates the difference between the predicted logical value and the local true label to obtain the calculated loss value. Through the above processing, the mapping deviation of multimodal features in the semantic space can be effectively transformed into a single scalar metric, thereby establishing the error starting point of the backpropagation algorithm and ensuring that the subsequent gradient descent direction can accurately guide the local adapter parameters to converge in the direction of reducing prediction error.

[0029] More specifically, in a concrete example of this application, the loss function operator supported locally by the node is first invoked, using the predicted logistic value output by the specific task header and the local true label read from local storage as input. For the multimodal classification task, a cross-entropy loss calculation process is executed. This process first normalizes the predicted logistic value using the Softmax activation function to generate a probability distribution, and then calculates the information entropy difference between this probability distribution and the one-hot encoded vector of the true label. This calculation process follows a mathematical expression. ,in The calculated loss value is defined as the final output. C This represents the total number of categories covered by the classification task. y c Represents the corresponding category in the local real-world tags. c The truth state, p c Represents the corresponding category in the predicted logistic value vector. c The original value, This represents the Softmax function used for probability mapping. This operation generates a scalar loss value characterizing the current model's prediction accuracy, which is then used as the input signal to start the subsequent directed backpropagation engine.

[0030] In step S32, the backpropagation engine is started and the weight attributes of the backbone network are locked and frozen. Partial derivatives with respect to layer outputs are calculated only for the bypass adapter branches to obtain the upstream error gradient. It should be noted that since multimodal pre-trained backbone networks typically contain a large number of parameters, performing backpropagation calculations involving all parameters at edge nodes would consume excessive GPU memory and computational power. Furthermore, updating the backbone network weights violates the design principles of federated collaboration regarding reducing communication load and maintaining the stability of static assets. Therefore, the technical solution of this application further starts the backpropagation engine and locks and freezes the weight attributes of the backbone network, calculating partial derivatives with respect to layer outputs only for the bypass adapter branches to obtain the upstream error gradient. Through this processing, the gradient update requirements of the backbone network parameters in the computation graph can be effectively shielded, concentrating limited computational resources to the bypass branches where the low-rank adapters reside. This allows for lightweight error propagation and parameter fine-tuning for specific federated tasks while maintaining the general feature extraction capability.

[0031] More specifically, in a concrete example of this application, the automatic differentiation engine of the deep learning framework is first invoked, using the computational loss value obtained from the preceding steps as the starting node for backpropagation. During the construction of the backpropagation computation graph, all weight tensors of the pre-trained backbone network are traversed, and their gradient requirement properties are explicitly set to an untrainable state, thereby logically blocking the flow of error gradients to the backbone parameters and their preceding nodes. Subsequently, for each layer of the network structure with the adapter loaded, only the gradient computation paths of the bypass branches are activated, and the partial derivative of the loss function with respect to the fused output h of that layer is calculated according to the chain rule. This partial derivative is defined as the upstream error gradient. It characterizes the degree to which small changes in the fusion output value of this layer affect the final prediction error, and serves as the direct input basis for the subsequent derivation of the gradient of the dimension reduction matrix and the dimension increase matrix, ensuring that the error signal only propagates and accumulates in the parameter space of the low-rank adapter.

[0032] In step S33, based on the upstream error gradient and the intermediate layer activation values, the gradient tensors of the dimensionality reduction matrix and the dimensionality increase matrix are derived using the chain rule to obtain the original adapter gradient. It should be noted that, given that in federated edge collaboration scenarios, in order to achieve efficient model fine-tuning at computationally limited edge nodes, the weight parameters of the backbone network are pre-locked to a non-updateable state. Therefore, it is impossible to directly obtain the optimization gradient for a specific task through conventional full-parameter backpropagation. Furthermore, the low-rank adapter, as an independent component carrying task-specific knowledge, requires precise gradient information to guide the iteration direction of its parameters in the low-dimensional subspace. Based on this, the technical solution of this application further derives the gradient tensors of the dimensionality reduction matrix and the dimensionality increase matrix using the chain rule based on the upstream error gradient and the intermediate layer activation values ​​to obtain the original adapter gradient. Through the above processing, the computational path of backpropagation can be effectively limited to the low-rank matrix range of the bypass adapter. By using the mathematically rigorous chain rule, the gradients of the dimension reduction matrix and the dimension increase matrix that can minimize the local prediction error can be analyzed, thus providing accurate numerical basis for subsequent compression and aggregation updates of these lightweight parameters, and realizing directional parameter optimization without affecting the backbone network.

[0033] More specifically, in a concrete example of this application, for each network layer loaded with a low-rank adapter, a gradient resolution process based on a computational graph is performed. First, the upstream error gradient passed to that layer by the backpropagation engine is obtained; this gradient characterizes the impact of changes in the layer's output on the final loss function. Simultaneously, the intermediate layer activation values ​​cached during the forward propagation phase are retrieved, serving as the raw input data for the adapter branch. Subsequently, the forward propagation formula is applied... The determined functional relationships are respectively for the increased-dimensional matrix. B With dimensionality reduction matrix A Apply the chain rule for finding the derivative of a composite function. For increasing-dimensional matrices... B , its gradient tensor G B The calculation is the product of the upstream error gradient and the transpose of the input vector after dimensionality reduction projection; for the dimensionality reduction matrix... A , its gradient tensor G A The calculation is the product of the transpose of the upgraded matrix, the upstream error gradient, and the transpose of the original input vector. This calculation follows the mathematical expression below:

[0034] in, G B Represents the gradient of an up-dimensional matrix. G A Represents the gradient of the dimension reduction matrix. Represents the upstream error gradient. xRepresents the activation value of the intermediate layer. A Represents the current weights of the dimensionality-reduced matrix. B Represents the current weights of the upgraded matrix, α represents the scaling factor. r Representing rank, This represents the matrix transpose operation. The above calculations generate the original adapter gradient, which contains only the partial derivatives of the low-rank matrix, and this gradient is used as the input for subsequent sparse compression processing.

[0035] Specifically, in step S4, the original adapter gradient is sparsely encoded and losslessly compressed based on a preset sparse threshold to obtain a compressed gradient data packet containing node identifiers. It should be noted that, given that uplink communication bandwidth is typically very limited in edge network environments, and the original adapter gradient tensor often contains a large number of near-zero redundant values ​​or unreliable fluctuations caused by local data noise, indiscriminate full transmission would not only cause severe communication congestion but may also introduce invalid or harmful gradient noise into the global model. Therefore, the technical solution of this application further performs sparse encoding and lossless compression on the original adapter gradient based on a preset sparse threshold to obtain a compressed gradient data packet containing node identifiers. An adaptive pruning strategy is used to remove insignificant elements with amplitudes below the threshold, retaining only gradient components carrying high-value update information. Coordinate mapping and entropy coding techniques are then used to perform high-density encapsulation of the sparsed non-zero values ​​and their position indices. Through the above processing, spatial and statistical redundancy in the gradient tensor can be effectively removed, reducing the transmission load from the dense matrix level to the sparse bitstream level. This alleviates the bandwidth pressure of federated communication while ensuring that the global model can converge efficiently based on high-confidence feature updates.

[0036] Figure 5 This is a flowchart illustrating a federated node collaborative computing method for multimodal data according to an embodiment of this application. It describes a process where the original adapter gradient is sparsely encoded and losslessly compressed based on a preset sparse threshold to obtain a compressed gradient data packet containing node identifiers. Figure 5 As shown, step S4 includes: S41, performing adaptive sparse pruning on the original adapter gradient to obtain a sparse gradient matrix; S42, performing coordinate mapping and entropy encoding compression on the sparse gradient matrix to obtain an encoded bitstream; S43, encapsulating the encoded bitstream into a payload based on a federated management protocol, and attaching node identity identifiers and data integrity verification codes to obtain compressed gradient data packets.

[0037] In step S41, adaptive sparse pruning is performed on the original adapter gradient to obtain a sparse gradient matrix. It should be noted that, given the extremely scarce communication bandwidth resources in edge federated collaboration scenarios, and the fact that multimodal data collected by edge nodes often involves severe heterogeneous noise and labeling errors, mechanical truncation based solely on gradient magnitude can easily lead to high-amplitude noise being misjudged as important features and uploaded, while ignoring effective convergence signals with smaller amplitudes but high consistency. This results in the global model being contaminated by outliers or experiencing slow convergence. Therefore, the technical solution of this application further performs adaptive sparse pruning on the original adapter gradient to obtain a sparse gradient matrix. By introducing a gradient uncertainty and signal-to-noise ratio (SNR) evaluation mechanism, the variance and signal strength of each gradient element in the time dimension are quantified, and a saliency mask is dynamically generated to filter out unreliable gradients with low SNR. Through the above processing, deep cleaning and simplification of gradient information can be effectively achieved. While reducing the communication transmission load, the propagation of local noise to the global model is blocked at the source, ensuring that only high-value gradients that have undergone rigorous reliability verification participate in subsequent federated aggregation updates.

[0038] Figure 6 This is a flowchart illustrating the adaptive sparse pruning of the original adapter gradient to obtain a sparse gradient matrix in a federated node collaborative computing method for multimodal data according to an embodiment of this application. Figure 6 As shown, step S41 includes: S411, performing uncertainty update and signal-to-noise ratio evaluation on the original adapter gradient and the historical gradient variance cached in the previous round to obtain the gradient signal-to-noise ratio map and the updated historical gradient variance; S412, determining the signal-to-noise ratio threshold based on the gradient signal-to-noise ratio map; S413, performing saliency mask mapping and weighted pruning on the original adapter gradient based on the signal-to-noise ratio threshold to obtain the sparse gradient matrix.

[0039] In step S411, uncertainty updates and signal-to-noise ratio (SNR) evaluations are performed on the original adapter gradient and the historical gradient variance cached in the previous round to obtain a gradient SNR map and the updated historical gradient variance. It should be noted that, given the practical application of federated edge collaboration, simply relying on gradient magnitude cannot accurately determine the true contribution of data, and edge node data is often affected by environmental noise, causing gradients to fluctuate drastically between different rounds. That is, the larger the variance, the more drastic the gradient fluctuations between different batches or rounds, and the less reliable the information it contains. Therefore, it is necessary to quantify the reliability of the gradient, rather than simply focusing on its magnitude. Based on this, the technical solution of this application further performs uncertainty updates and SNR evaluations on the original adapter gradient and the historical gradient variance cached in the previous round to obtain a gradient SNR map and the updated historical gradient variance. Through the above processing, the local computing resources of the nodes can be effectively utilized to continuously track the second moment of the gradient based on the exponential moving average algorithm, which is used as a measure of gradient uncertainty. An evaluation model containing nonlinear adaptive gating terms is constructed, thereby realizing fine-grained control of federated data, accurately distinguishing strong signals from high-amplitude noise, eliminating unreliable gradient information from the source, and ensuring that the data transmitted subsequently has a very high degree of confidence.

[0040] More specifically, in a concrete example of this application, the gradient variance estimate and gradient first moment estimate cached from the previous round are first retrieved. Combined with the original adapter gradient calculated in the current round, an uncertainty update calculation is performed to obtain the current variance, which reflects the severity of gradient fluctuations. Subsequently, based on the updated variance and the current gradient magnitude, a preset gradient signal-to-noise ratio evaluation model is substituted into the data for element-wise computation. This computation process follows the mathematical expression below:

[0041] in, v t This represents the gradient variance estimate (i.e., uncertainty) for the current round t. v t-1 The variance of the previous round, G t For the current gradient, m t-1 This is the first moment estimate of the gradient from the previous round, where β is the smoothing coefficient. S SNR This represents the calculated signal-to-noise ratio score. To prevent the smoothing term from being divided by zero, λ is a hyperparameter that adjusts the noise suppression strength. It should be noted that the exponential term in the formula... It serves as an adaptive gating mechanism: when the gradient magnitude... G t Much smaller than the noise level When the signal approaches 0, it strongly suppresses noise; conversely, it approaches 1, preserving the original signal. Through the above calculations, a gradient signal-to-noise ratio (SNR) map with the same dimension as the gradient tensor is generated, and the historical gradient variance is updated simultaneously for use in the next iteration. Taking a specific edge vision perception scenario as an example, suppose that due to illumination flicker, the collected image data of a certain edge node contains pixel noise in a specific area, causing the gradient of the model parameters at the corresponding location to exhibit large positive and negative oscillations in consecutive training rounds. At this time, although the gradient magnitude at that location... The variance may be large, but the variance calculated using the EMA algorithm described above is... v t It will also accumulate and increase. When evaluating the signal-to-noise ratio, due to the increase in the variance term in the denominator and the suppressive effect of the adaptive gating term, the final calculated value at this position will be higher. S SNR The score will be significantly reduced. Based on this, the system can identify that the high-amplitude gradient is actually an unreliable noise perturbation, and remove it from the upload queue in subsequent steps based on the low signal-to-noise ratio score, thereby avoiding dirty data from misleading the convergence direction of the global model.

[0042] In steps S412 and S413, a signal-to-noise ratio (SNR) threshold is determined based on the gradient SNR map. Then, based on the SNR threshold, the original adapter gradient is subjected to saliency masking and weighted pruning to obtain a sparse gradient matrix. It should be noted that, given the inconsistent data quality of edge nodes in heterogeneous multimodal federated learning scenarios, existing mechanisms cannot distinguish between high-amplitude noise and high-value signals. If only gradient values ​​are retained, high-amplitude noise caused by sensor malfunctions or labeling errors is easily misjudged as important features and uploaded, thus polluting the global model. Therefore, the technical solution of this application further determines the SNR threshold based on the gradient SNR map, and then performs saliency masking and weighted pruning on the original adapter gradient based on the SNR threshold to obtain a sparse gradient matrix. Through the above processing, gradient elements that truly contribute to the global model can be selected, abandoning the crude approach of sorting by magnitude. This ensures that each gradient value retained is the essence that has undergone rigorous reliability verification. While achieving deep sparsity of the data, it effectively resists the interference of high-noise data from a single node, reflecting implicit control over the quality of node data.

[0043] More specifically, in a concrete example of this application, a saliency mask generation step based on signal-to-noise ratio (SNR) sorting is first performed. A sorting operator is invoked to perform a Top-K sort on the elements in the generated gradient SNR map, arranging all gradient elements from high to low according to their SNR scores. Then, scores at specific positions are truncated according to a preset sparsity rate to determine a retention threshold. Subsequently, the score of each element in the graph is compared point-by-point with the threshold to generate a sparse mask based on the signal-to-noise ratio (SNR): if the score is higher than or equal to the threshold, the corresponding position in the mask is set to 1; otherwise, it is set to 0. Next, this mask is applied to the original gradient, and a Hadamard product is performed, retaining only the original gradient values ​​at the positions where the mask is 1, and forcing the remaining positions to zero, thus obtaining a high-confidence sparse gradient containing only high SNR elements. This calculation logic follows the following mathematical expression:

[0044] in, M ij For the generated sparse mask, This is an indicator function used to determine whether a condition is true or false. For position ( i,j The signal-to-noise ratio score at point (). Preserve a threshold for the signal-to-noise ratio; The final generated high-confidence sparse gradient, G t For the original gradient, This represents the Hadamard product. Taking the collaborative operation of roadside sensing nodes in a smart transportation scenario as an example, suppose that at a certain moment, the camera lens of a specific roadside unit generates continuous high-frequency noise due to rain or sensor malfunction. During backpropagation, this noise causes high-amplitude noise with violent oscillations in the underlying parameters of the model. If traditional methods are used, these invalid updates with large amplitudes will be considered as important features and uploaded. However, under the mechanism of this scheme, although the absolute value of the gradient at this location is large, its signal-to-noise ratio score is lower than the retention threshold determined by Top-K ranking. During the saliency masking stage, the mask value corresponding to this region will be set to 0, so that the sparse gradient matrix generated in the end will automatically eliminate the influence of this noisy data and retain only the high-value signals of other normal perceived road features.

[0045] In step S42, the sparse gradient matrix is ​​subjected to coordinate mapping and entropy encoding compression to obtain an encoded bitstream. It should be noted that although the sparse gradient matrix obtained after pre-pruning removes a large number of redundant noise parameters, it still retains a high-dimensional framework consistent with the original tensor in its physical storage structure. The massive number of zero-valued elements contained within it are not substantially removed, and the non-zero gradient elements exhibit statistical regularity in their numerical distribution. Directly transmitting a matrix structure containing a large amount of zero-padding would still consume valuable communication bandwidth resources. Based on this, the technical solution of this application further performs coordinate mapping and entropy encoding compression on the sparse gradient matrix to obtain an encoded bitstream. Through the above processing, the spatial sparsity of the gradient data can be effectively transformed into a compact binary storage form. By eliminating spatial redundancy in the matrix structure and statistical redundancy in the numerical representation, the transmission load is compressed to near the theoretical limit of information entropy, thereby minimizing the network throughput pressure when edge nodes send updated data to the server at the hardware level.

[0046] More specifically, in a concrete example of this application, a non-zero element extraction operation is first performed on the sparse gradient matrix. The tensor space is traversed to locate all retained gradient values ​​and their corresponding multidimensional spatial coordinate indices, and these discretely distributed non-zero data are mapped into a compact linear value-index pair sequence. Subsequently, for the gradient value sequence, floating-point numbers are directly input into an entropy encoder for statistical compression according to the IEEE 754 byte sequence to achieve lossless compression. Simultaneously, run-length encoding is used to differentially compress the coordinate index sequence to record the positional spacing between adjacent non-zero elements, rather than absolute coordinates. Based on this, the entropy encoding algorithm operator is invoked to construct a probability model based on the frequency of occurrence of quantized values ​​and positional differences in the current data packet. Shorter binary codewords are assigned to data patterns with higher frequency occurrences, and longer codewords are assigned to low-frequency patterns. Finally, a continuous, high-density encoded bitstream is generated, which serves as the payload for subsequent network transmission, achieving lossless compression and encapsulation of model update information.

[0047] In step S43, the encoded bitstream is encapsulated into a payload based on the federated control protocol, and node identification and data integrity check codes are appended to obtain compressed gradient data packets. It should be noted that, since federated collaborative networks involve the concurrent access of massive heterogeneous edge nodes, and the network transmission environment often carries the risk of packet loss or bit flipping, if the receiving end cannot accurately identify the source node of the gradient data or detects data integrity corruption during transmission, it will lead to incorrect aggregation weight allocation or the introduction of corrupted gradient noise, thereby compromising the convergence stability of the global model. Based on this, the technical solution of this application further encapsulates the encoded bitstream into a payload based on the federated control protocol, and appends node identification and data integrity check codes to obtain compressed gradient data packets. Through the above processing, a communication data format with traceability and fault tolerance mechanisms can be effectively established, ensuring that the server can execute differentiated aggregation strategies based on node identification and automatically eliminate invalid data packets with transmission errors using check codes, thereby ensuring the security and reliability of the federated collaborative process.

[0048] More specifically, in a concrete example of this application, the encapsulation interface of the network communication protocol stack is first invoked to divide the entropy-encoded bitstream generated in the preceding steps into standard-length data segments and map them to the data payload area of ​​the transport layer protocol. Subsequently, the unique identifier registered by the edge device in the federated network and the global round index of the current collaborative task are read from the local protected storage space and written into the header field of the data packet as the basis for the server to identify the data source and time synchronization. Next, cyclic redundancy check or hash digest calculation is performed on the encapsulated data payload to generate a fixed-length data integrity check code, which is then appended to the tail field of the data packet. Finally, the binary sequence containing header control information, middle gradient payload, and tail check information is assembled into a compressed gradient data packet conforming to the federated control protocol specification, ready to be sent to the parameter server via the uplink communication link.

[0049] Specifically, in step S5, the compressed gradient data packets of multiple edge nodes are aggregated using a global adapter to update the global adapter parameters, resulting in updated global adapter parameters for the next round of collaborative distribution. It should be noted that, given that in federated collaborative scenarios with distributed data storage, a single edge node only holds locally heterogeneous multimodal data, its calculated gradient updates often have significant local biases or cannot represent the global data distribution. Furthermore, the uploaded data is a compressed bitstream encapsulated in a high-entropy encoded form, which cannot be directly applied to mathematical operations on model parameters. Based on this, the technical solution of this application further aggregates the compressed gradient data packets of multiple edge nodes using a global adapter to update the global adapter parameters, resulting in updated global adapter parameters for the next round of collaborative distribution. Through the above processing, the sparse bitstream from different nodes can be effectively restored into a standard-dimensional gradient tensor by utilizing the multi-path parallel decompression and gradient reconstruction mechanism. Then, a weighted average calculation is performed according to the preset aggregation weights to obtain the global average gradient. This global gradient is then used to update the low-rank matrix in the global adapter of the current round only by gradient descent. In this way, while integrating the local knowledge of each node to improve the generalization ability of the global model, the frozen state of the backbone network is maintained, completing the closed-loop iteration from local compressed information to global lightweight instructions.

[0050] More specifically, in a specific example of this application, step S5 includes: performing multi-parallel decompression and gradient reconstruction on the received compressed gradient data packet to obtain a reconstructed gradient set; performing weighted average calculation on the reconstructed gradient set according to a preset aggregation weight to obtain a global average gradient; and using the global average gradient to update the low-rank matrix in the global adapter parameters of the current round with gradient descent to obtain the updated global adapter parameters.

[0051] Accordingly, the received compressed gradient data packets are decompressed and reconstructed in multiple parallel paths to obtain a reconstructed gradient set. It should be noted that since the data received by the server via the federated management protocol is a compressed payload encapsulated in binary bitstream form, this payload cannot directly participate in the tensor algebra-based model parameter aggregation calculation. Furthermore, facing massive concurrent upload requests from numerous edge nodes, serial processing would result in excessive aggregation latency, thus slowing down the overall federated training pace. Therefore, the technical solution of this application further decompresses and reconstructs the received compressed gradient data packets in multiple parallel paths to obtain a reconstructed gradient set. Through the above processing, the high-performance parallel computing capabilities of the server can be effectively utilized to restore the high-compression ratio data streams from different heterogeneous nodes, after entropy encoding and sparse pruning, into gradient tensor objects that are strictly consistent with the dimensions of the global adapter parameters, thereby providing a standardized mathematical input basis for subsequent weighted average aggregation operations.

[0052] More specifically, in a concrete example of this application, a high-performance parallel processing pipeline is first built on the server side. A multi-threaded or distributed computing framework listens to the uplink communication port. Once compressed gradient data packets from the edge nodes are captured, they are immediately distributed to independent decoding computing units. Next, a reverse entropy decoding operator matching the edge node's encoding end is invoked to parse the payload in the data packets into a bitstream, recovering the numerical sequence of non-zero gradients and their corresponding sparse coordinate indices. Subsequently, based on the pre-configured low-rank adapter dimension information, a zero-based gradient tensor template is initialized, and a sparse reconstruction operation is performed. The corresponding gradient values ​​are filled into the specified positions of the zero-based tensor according to the parsed coordinate indices, thereby generating a complete reconstructed gradient tensor. This reconstruction process follows a mathematical expression. ,in Representing the i The full-size gradient tensor after each node is recovered P i The received compressed data packet represents the Decode function, the Reconstruct function represents the sparse padding distribution function, and the Shape function represents the reverse decoding function. A with Shape B These represent the geometric shape parameters of the reduced-dimensional and increased-dimensional matrices. Finally, all successfully reconstructed gradient tensors are aggregated into a memory buffer pool to form a set of reconstructed gradients for use by the aggregation algorithm.

[0053] Accordingly, a weighted average calculation is performed on the reconstructed gradient set according to the preset aggregation weights to obtain the global average gradient. This global average gradient is then used to update the low-rank matrix in the global adapter parameters of the current round using gradient descent to obtain the updated global adapter parameters. It should be noted that, due to the imbalanced sample size and heterogeneous feature distribution of the multimodal data held by a single edge node, and because the server only receives gradient increments for the low-rank adapter rather than complete model parameters, the lack of a unified global aggregation and update mechanism will cause the model to shift towards the local optimum of a single node, failing to form a global model with generalization capabilities. Based on this, the technical solution of this application further performs a weighted average calculation on the reconstructed gradient set according to the preset aggregation weights to obtain the global average gradient, and uses this global average gradient to update the low-rank matrix in the global adapter parameters of the current round using gradient descent to obtain the updated global adapter parameters. Through the above processing, the specific task knowledge learned by different edge nodes on local data can be effectively integrated, the data contribution of different nodes can be balanced through a weighting mechanism, and the low-rank adapter parameters can be driven to iterate in the direction of minimizing global loss by using the global average gradient, thereby achieving global collaborative convergence of the federated model while keeping the backbone network frozen.

[0054] More specifically, in a concrete example of this application, the aggregation weights for each node are first determined based on the proportion of data samples used for training locally by each participating node to the total number of samples. Then, the aggregation operator of the parameter server is invoked to iterate through each gradient tensor in the reconstructed gradient set, performing a scalar multiplication operation with its corresponding aggregation weight. The weighted results are then element-wise summed to generate the global average gradient representing the global update direction. This calculation process follows the mathematical expression:

[0055] in, Represents the global average gradient, and N represents the total number of nodes participating in this round of aggregation. w i Representing the i The aggregate weight of each node, Representing the i The reconstructed gradient tensor of each node. Next, the optimizer operator is invoked, using this global average gradient to update the parameters of the reduced and increased dimension matrices stored in the global adapter for the current round. This update process follows the mathematical expression:

[0056] Where, θ t+1 Represents the updated global adapter parameters, θ t The parameters represent the current round, and η represents the global learning rate. This represents the global average gradient. Finally, the updated low-rank matrix parameters are solidified into the new global model state for use in the next round of distribution and deployment.

[0057] Specifically, in one example embodiment, edge node local training employs the AdamW optimizer, with 1–5 local epochs, a batch size of 8–64, and an adapter learning rate of 1×10⁻⁶. -5 ~5×10 -4 Aggregate weights w i The sparse mask can be set according to the proportion of samples in each node. The sparse mask can be generated using the Top-K method, with a retention ratio of, for example, 0.1% to 5%.

[0058] In summary, a federated node collaborative computing method for multimodal data according to embodiments of this application is illustrated. It employs a parameter decoupling strategy, pre-constructing global adapter parameters containing low-rank matrix weights and distributing them to edge nodes, which are then loaded into the frozen pre-trained backbone network bypass. During local training, the multimodal data stream drives parallel computation of the frozen backbone and adapter branches, performing gradient derivation only for low-rank parameters, thus avoiding the transmission of all parameter updates. Furthermore, adaptive sparse pruning based on amplitude or signal-to-noise ratio is applied to the original adapter gradients, and lossless compression and packaging of the gradient subspace is performed using coordinate mapping and entropy coding techniques. The server only needs to decompress and weighted aggregate the received compressed gradient packets to complete the iterative update of the global adapter. This mechanism realizes the transformation from full model transmission to lightweight instruction interaction, reducing the communication bandwidth consumption during the federated collaboration process.

[0059] As described above, the federated node collaborative computing method for multimodal data according to embodiments of this application can be implemented in various distributed intelligent sensing and collaborative computing platforms, such as smart city edge computing gateways, vehicle-road cooperative roadside sensing units, or industrial internet smart terminals. In one possible implementation, this method can be integrated into an edge intelligent management system or IoT data analysis platform as a lightweight model fine-tuning engine or a low-bandwidth collaborative training component. For example, the method can be an independent multimodal federated learning application running on an edge server, a communication optimization plugin and low-rank adaptation module for an existing distributed AI framework, or a management middleware service deployed in the cloud and distributing lightweight instructions through a downlink channel. Of course, the frozen backbone network loading, low-rank residual gradient calculation, and sparse compression modules based on signal-to-noise ratio evaluation in this method can also run in embedded devices, smart cameras, or vehicle computing units equipped with limited computing power, serving as the underlying collaborative inference foundation of this heterogeneous federated system.

[0060] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A federated node collaborative computing method for multimodal data, characterized in that, include: S1: Based on the pre-trained backbone configuration and federated task requirements, construct global adapter parameters containing low-rank matrix weights, and distribute them to edge nodes to load into the frozen backbone network to obtain local adapter parameters. S2: Input the collected multimodal data stream into the frozen backbone network loaded with local adapter parameters to obtain the predicted logic value and intermediate layer activation value; S3: Based on the predicted logistic values, intermediate layer activation values, and local real labels, perform loss calculation and directed backpropagation to obtain the original adapter gradient; S4: Based on a preset sparse threshold, the original adapter gradient is sparsely encoded and losslessly compressed to obtain a compressed gradient data packet containing node identifiers; S5: Perform global adapter aggregation on the compressed gradient data packets of multiple edge nodes to update the global adapter parameters, so as to obtain the updated global adapter parameters for the next round of collaborative distribution.

2. The federated node collaborative computing method for multimodal data according to claim 1, characterized in that, Step S1 includes: The pre-trained backbone configuration is analyzed to determine the hidden layer dimension of the model, and the initial parameters of the global adapter, consisting of a dimension reduction matrix and a dimension increase matrix, are constructed in combination with the requirements of the federated task. The candidate node list is traversed to evaluate online status and computing resources, and nodes that do not meet the criteria are removed based on the evaluation results to obtain the selected node list. The initial parameters of the global adapter are distributed to each node in the selected node list via the downlink control channel, so that they are instantiated and loaded into the locally pre-stored frozen backbone network bypass to obtain the local adapter parameters.

3. The federated node collaborative computing method for multimodal data according to claim 2, characterized in that, The dimension reduction matrix is ​​initialized using a Gaussian distribution, while the dimension increase matrix is ​​initialized with all zeros.

4. The federated node collaborative computing method for multimodal data according to claim 1, characterized in that, Step S2 includes: Modality alignment preprocessing and tensor quantization transformation are performed on the multimodal data stream to obtain the multimodal input tensor; The multimodal input tensor is simultaneously injected into the main path of the frozen backbone network and the bypass path loaded with local adapter parameters to obtain the fused feature vector, while the multimodal input tensor is cached as the activation value of the intermediate layer. The fused feature vector is input into a specific task head for nonlinear mapping and probability distribution estimation to obtain the predicted logical value.

5. The federated node collaborative computing method for multimodal data according to claim 1, characterized in that, Step S3 includes: The difference between the predicted logical value and the local true label is calculated to obtain the calculated loss value; Start the backpropagation engine and lock the weight properties of the frozen backbone network, and calculate the partial derivatives with respect to the layer output only for the bypass adapter branches to obtain the upstream error gradient; Based on the upstream error gradient and the intermediate layer activation values, the gradient tensors of the dimension reduction matrix and the dimension increase matrix are derived using the chain rule to obtain the original adapter gradient.

6. The federated node collaborative computing method for multimodal data according to claim 1, characterized in that, Step S4 includes: Adaptive sparse pruning is performed on the original adapter gradient to obtain a sparse gradient matrix; Coordinate mapping and entropy coding compression are performed on the sparse gradient matrix to obtain the encoded bitstream; Based on the federal control protocol, the encoded bit stream is encapsulated into a payload, and node identity identifiers and data integrity check codes are attached to obtain compressed gradient data packets.

7. The federated node collaborative computing method for multimodal data according to claim 1, characterized in that, Step S5 includes: The received compressed gradient data packets are decompressed and reconstructed in multiple parallel streams to obtain a reconstructed gradient set; The reconstructed gradient set is weighted and averaged according to the preset aggregation weights to obtain the global average gradient. The global average gradient is then used to update the low-rank matrix in the global adapter parameters of the current round using gradient descent to obtain the updated global adapter parameters.

8. The federated node collaborative computing method for multimodal data according to claim 1, characterized in that, Adaptive sparse pruning is applied to the original adapter gradient to obtain a sparse gradient matrix, including: Uncertainty updates and signal-to-noise ratio assessments are performed on the original adapter gradient and the historical gradient variance cached in the previous round to obtain the gradient signal-to-noise ratio map and the updated historical gradient variance. The signal-to-noise ratio threshold is determined based on the gradient signal-to-noise ratio map. Based on the signal-to-noise ratio threshold, the original adapter gradient is subjected to saliency masking and weighted pruning to obtain a sparse gradient matrix.

Images