Federal learning method for vehicle-mounted multi-mode perception

Abstract

The invention discloses a federated learning method for vehicle-mounted multi-modal perception, which introduces an interpretable tool to observe that the contribution degree of each modal data of different nodes on different rounds of multi-modal nodes to local output is dynamically changed, and the contribution degree is determined by different modal data characteristics in different node physical environments. According to the method, the tradeoff of communication efficiency and accuracy is realized on the premise of not losing precision, the influence of modal coupling and distribution heterogeneity is reduced through a local decision-making level multi-modal fusion network, and meanwhile, an interpretable tool is introduced to evaluate the contribution degree of each modal model in the federation aggregation process; and a selection model uploading strategy is designed by using the difference between modes. In combination with an interpretability technology, the method can adapt to a limited communication bandwidth condition on the premise of ensuring precision in typical multi-modal federated learning scenes such as intelligent network connection vehicles and the like, and realizes a multi-modal sensing task.

Description

Technical Field

[0001] The present invention relates to the field of multimodal artificial intelligence technology, and in particular to a federated learning method for vehicle-mounted multimodal perception. Background Art

[0002] With the rapid development of the Internet of Things and artificial intelligence technologies, multimodal perception systems have shown great application potential in the fields of autonomous driving, behavior recognition, medical testing, etc. Its core concept is to achieve more complex and dynamic perception tasks by integrating multiple complementary modal data. However, existing multimodal perception systems often require direct access to raw sensor data. This centralized processing method seriously threatens user privacy and limits its application in real scenarios. Federated learning, as a distributed machine learning paradigm, allows multiple devices to collaborate and train only by sharing model parameters, thereby improving model performance without uploading data to a central server. In the traditional federated learning process, the client trains the model on local data, then uploads it to a central server for aggregation, and finally the client downloads and deploys the aggregated global model.

[0003] Therefore, multimodal federated learning has attracted much attention, especially in the field of intelligent connected vehicles. By deploying multiple sensors such as cameras, lidar, millimeter-wave radar, and GPS, vehicles can make optimal control decisions in various driving scenarios, different weather conditions, and limited fields of view, while also relying on federated learning for collaborative learning between vehicles.

[0004] Multimodal federated learning allows training on distributed multimodal data, but communication limitations are particularly significant in edge devices with limited wireless bandwidth. Traditional federated learning solutions are mostly applicable to single-modal tasks and cannot solve the unique problems of multimodal federated learning. For example, nodes in multimodal federated learning usually adopt complex multimodal fusion networks, which leads to high communication costs and difficulty in deployment in bandwidth-constrained edge scenarios. In addition, traditional multimodal federated learning frameworks usually require nodes with the same data modality to conduct collaborative learning, which limits the aggregation of models between nodes of different types. Directly aggregating models of different types of nodes will lead to a significant decrease in accuracy because the coupled modes and distribution heterogeneity between heterogeneous node data are not taken into account. In multimodal federated learning scenarios, in addition to the heterogeneity of data distribution caused by factors such as users and environments, modal heterogeneity is also common, that is, different sensor configurations between nodes or dynamic failures during operation, which greatly increases the complexity of the system.

[0005] Existing multimodal federated learning solutions usually require uploading all modal models to the server for aggregation, which brings about a large communication overhead. Therefore, it is critical to design a suitable model upload strategy to minimize the communication cost while ensuring accuracy. Summary of the invention

[0006] The purpose of the present invention is to make up for the shortcomings of the existing technology and provide a federated learning method for in-vehicle multimodal perception, aiming to formulate a suitable model upload strategy in a resource-constrained and highly heterogeneous in-vehicle environment, effectively balance the accuracy and communication efficiency of federated learning, and promote the widespread application and implementation of multimodal federated learning in bandwidth-constrained edge scenarios.

[0007] The present invention is achieved through the following technical solutions:

[0008] A federated learning method for vehicle-mounted multimodal perception includes the following steps:

[0009] Step 1: Collect multimodal data and transmit it to the on-board computer for data processing to build a local data set;

[0010] Step 2: Each node trains the corresponding modal model in parallel for each modality in its local data set. The node uses the prediction results of each modal model to fuse through the integrated model, and uses the Shapley value to calculate the contribution of each modal model to the integrated model prediction to obtain the corresponding prediction result label set;

[0011] Step 3: Model the node bandwidth limit and the size and contribution of each modal model, and formulate a dynamic strategy for uploading the selected model;

[0012] Step 4: The node packages the model parameters, modal information, and sample quantity information data packets and uploads them to the server for aggregation according to the formulated model upload strategy;

[0013] Step 5: The server receives the data packets from the nodes and performs weighted aggregation based on the number of samples of each data modality to obtain a global model for each modality.

[0014] In step 6, the node downloads the global model of the corresponding mode, and updates the local single-modal models and the integrated model in combination with the local data, and continuously iterates the above process until the model converges.

[0015] The multimodal data collected in step 1 is multimodal data formed by collecting the surrounding environment information of the connected vehicle through the camera, lidar and GPS installed in the connected vehicle.

[0016] After each node in step 2 trains the corresponding modal model in parallel for each modality in its local data set, it calculates the parameter quantity of each modal model and calculates the corresponding upload time in combination with the node bandwidth information; then the contribution degree and upload time of each modal model obtained are respectively Min-Max normalized, and a weighted calculation is used to obtain a comprehensive score. The initial upload probability of each modal model is determined by the comprehensive score, thereby formulating a dynamic model selection upload strategy.

[0017] The dynamic selection model upload strategy is as follows:

[0018] Set label set to Y k , the node communication bandwidth is B k , the multimodal dataset on the kth multimodal node Input into the corresponding unimodal model respectively Instead of training a single multimodal network, parallel training is performed. represents the mth modal dataset of the kth node (M k for a specific mode). The node then converts the output of each modal model After the integrated model ω k Perform decision-level multimodal fusion to obtain the final prediction Right now:

[0019]

[0020] When the federated training model is updated, the node only uploads the single-modal model. Instead of uploading the local integrated model k .

[0021] The interpretability tool Shapley value is introduced to calculate the modal contribution:

[0022]

[0023] The absolute value of the modal model contribution and the corresponding single communication time The two benchmarks are normalized by Min-Max to obtain and

[0024] The weighted calculation of the two results in a comprehensive score:

[0025]

[0026] S m is the comprehensive score of the node internal mode m

[0027] where α s , α t It is an adjustable empirical parameter used to control the relationship between modal contribution and single communication time, so that modal models with high contribution and small model size can obtain higher comprehensive scores;

[0028] The upload probability of each modality model is determined based on the comprehensive score:

[0029]

[0030] S m is the comprehensive score of the node internal mode m, Sum the comprehensive scores of all modes within the node.

[0031] Calculate entropy based on upload probability:

[0032]

[0033] Introducing entropy adjustment factor λ eaf Adjust upload probability:

[0034]

[0035] Among them, λ eaf An adjustable parameter that controls the effect of entropy. When entropy is low, it means that the probability distribution is more concentrated, which may lead to over-selection of the mode and neglect of other modes. By introducing the entropy adjustment factor, the diversity of the probability distribution can be increased while considering the comprehensive score, preventing over-selection and making the upload probability more reasonable.

[0036] The adjusted upload probability is normalized to obtain the final probability distribution to ensure that the sum of the upload probabilities of the three modes is 1:

[0037]

[0038] The advantages of the present invention are: the present invention introduces an interpretability tool, and observes that the contribution of each modal data to the local output of different nodes in different rounds of multimodal nodes is dynamically changing, which is determined by the characteristics of different modal data in the physical environment of different nodes. The present invention achieves a trade-off between communication efficiency and accuracy without losing accuracy, and reduces the influence of modal coupling and distribution heterogeneity through a multimodal fusion network at the local decision level. At the same time, an interpretability tool is introduced to evaluate the contribution of each modal model in the federated aggregation process, and a selection model upload strategy is designed using differences between modalities. The present invention combines interpretability technology in typical multimodal federated learning scenarios such as intelligent connected vehicles to adapt to limited communication bandwidth conditions while ensuring accuracy, and realize multimodal perception tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the system module diagram;

[0040] Figure 2 It is the calculation principle diagram;

[0041] Figure 3 It is an operation diagram. DETAILED DESCRIPTION

[0042] A federated learning method for vehicle-mounted multimodal perception is based on a scenario of an intelligent connected vehicle equipped with multiple sensors, where the data acquisition equipment includes a camera, a lidar, and a GPS. The specific steps in the preparation phase are as follows:

[0043] Start the connected vehicle and drive along the route. Meanwhile, the data acquisition module operates normally to collect information about the surrounding environment.

[0044] The on-board computer performs preliminary processing on the multimodal perception data and constructs a data set for subsequent use by the system.

[0045] The federated learning system proposed in this invention includes three main modules: local training, model uploading and model aggregation. Figure 1 As shown in the figure, each module works together to complete the multimodal perception task. The specific steps are as follows:

[0046] Step 1: Collect multimodal data and transmit it to the on-board computer for data processing to build a local data set;

[0047] Step 2: Each node trains the corresponding modality model in parallel for each modality in its local data set to obtain the corresponding prediction result label set;

[0048] Step 3: The node uses the prediction results of each modal model to fuse through the integrated model, and uses the Shapley value to calculate the contribution of each modal model to the prediction of the integrated model;

[0049] Step 4: Model the node bandwidth limit and the size and contribution of each modal model, and formulate a dynamic strategy for uploading the selected model;

[0050] Step 5: The node packages the model parameters, modal information, and sample quantity information data packets and uploads them to the server for aggregation according to the formulated model upload strategy;

[0051] Step 6: The server receives the data packets from the nodes and performs weighted aggregation based on the number of samples of each data modality to obtain a global model for each modality.

[0052] Step 7: The node downloads the global model of the corresponding mode, and updates the local single-mode models and the integrated model in combination with the local data, and continuously iterates the above process until the model converges.

[0053] The dynamic selection model upload strategy is as follows:

[0054] Set label set to Y k , the node communication bandwidth is B k , the multimodal dataset on the kth multimodal node Input into the corresponding unimodal model respectively Parallel training is performed, where represents the mth modal dataset of the kth node, M k For a specific mode, the node then converts the output of each modal model After the integrated model ω k Perform decision-level multimodal fusion to obtain the final prediction Right now:

[0055]

[0056] When the federated training model is updated, the node only uploads the single-modal model. Instead of uploading the local integrated model k ;

[0057] The interpretability tool Shapley value is introduced to calculate the modal contribution, that is, to evaluate the integrated model ω k Input and output The relationship between:

[0058]

[0059] in, is the Shapley value of the input mode m, i.e. the modal contribution, is the output does not include a subset of mode m, Is the ensemble model using only the collection The corresponding predicted value is when the mode in is

[0060] The absolute value of the modal model contribution and the corresponding single communication time The two benchmarks are normalized by Min-Max to obtain and

[0061] is the unimodal model of the kth node mode m, B k is the communication bandwidth of the kth node;

[0062] The weighted calculation of the two results in a comprehensive score:

[0063]

[0064] in is the comprehensive score of the kth node mode m, α s , α t It is an adjustable empirical parameter used to control the relationship between modal contribution and single communication time, so that modal models with high contribution and small model size can obtain higher comprehensive scores;

[0065] The upload probability of each modality model is determined based on the comprehensive score:

[0066]

[0067] S m is the comprehensive score of the node internal mode m, Sum the comprehensive scores of all modes within the node;

[0068] Calculate entropy based on upload probability:

[0069]

[0070] Introducing entropy adjustment factor λ eaf Adjust upload probability:

[0071]

[0072] Among them, λ eaf An adjustable parameter that controls the effect of entropy. When entropy is low, it means that the probability distribution is more concentrated.

[0073] The adjusted upload probability is normalized to obtain the final probability distribution to ensure that the sum of the upload probabilities of the three modes is 1:

[0074]

[0075] In a multimodal federated learning environment, some modalities may be over-selected, resulting in other modal information being ignored, thus affecting model performance. Therefore, entropy is introduced in step 3 above to balance the upload probability of different modal models, ensuring that all modalities have a chance to be uploaded. Nodes can find a more balanced probability distribution while maintaining contribution differences by setting different entropy adjustment factors, thereby improving system robustness and generalization capabilities.

[0076] Due to the complex modal heterogeneity between different multimodal nodes, for example, even cars from the same manufacturer may be equipped with different types and numbers of sensors; even nodes of the same type may suffer from modal loss during operation due to dynamic sensor failure. For this reason, in step 1, different integrated models are allowed to be set for each node, which can be configured by the node according to its own unique deployment scenario. In step 2, the integrated model only serves as an intermediate state to assist in calculating the contribution of each modal model, and does not participate in model upload; in step 6, after the node receives the updated global single-modal model sub-model, it can use the integrated model in combination with local data to complete personalization and improve node recognition accuracy.

[0077] The operation of the proposed multimodal federated learning framework is further explained below.

[0078] The calculation principle diagram is as follows Figure 2 As shown, Figure 3 For the sake of convenience, the multi-modal sensing task of using the present invention to solve the fast sector selection of the vehicle-mounted millimeter wave antenna using multi-sensor data in a vehicle-mounted scenario is taken as an example.

[0079] Step 1-1: The connected car is equipped with sensors such as cameras, lidar, and GPS to collect multimodal data, and transmit it to the on-board computer for data processing to build a data set;

[0080] Step 1-2, each node trains the corresponding modality model in parallel for each modality in the local data set;

[0081] Step 1-3, the prediction results of each modal model are integrated through the integrated model, and the contribution of each mode is calculated using the Shapley value;

[0082] Step 1-4, calculate the parameter quantity of each modal model and calculate the corresponding upload time in combination with the node bandwidth information;

[0083] Step 1-5, the contribution and transmission time of each mode obtained in step 1-3 and step 1-4 are normalized by Min-Max respectively. In order to comprehensively consider the impact of the two on the upload strategy, a weighted calculation is used to obtain a comprehensive score. The ratio of the two can be controlled by the node through parameters, so that the modal model with high contribution and fast transmission can obtain a higher comprehensive score.

[0084] Steps 1-6, determine the initial upload probability of each modality model based on the comprehensive score;

[0085] Steps 1-7 introduce an entropy adjustment factor to rebalance the probability distribution, prevent a certain mode from being over-selected, and make the upload probability more reasonable.

[0086] Steps 1-8: Normalize to obtain the probability distribution of the final selection for uploading, and the node packages the model parameters, sample quantity, modality type and other information and sends them to the server;

[0087] Step 1-9: The server performs weighted aggregation on the uploaded modal models according to the number of samples to obtain a global model for each type;

[0088] Step 1-10: The node downloads the global modal model and updates the local single modal models and the integrated model based on the data;

[0089] Steps 1-11, iterate the above process until the model converges.

[0090] This invention combines explainability technology in typical multimodal federated learning scenarios such as intelligent connected vehicles to adapt to limited communication bandwidth conditions while ensuring accuracy and realize multimodal perception tasks. In order to facilitate deployment, a local multimodal fusion network is designed in combination with ensemble learning to adapt to our system. In summary, there are several advantages:

[0091] (1) It can balance accuracy and communication overhead and adapt to application scenarios with different resource constraints.

[0092] (2) The contribution of each modality is effectively quantified, enhancing the interpretability of the system.

[0093] (3) It can adapt to highly heterogeneous nodes in vehicle scenarios, and its modular design facilitates deployment.

[0094] The present invention demonstrates a federated learning framework for vehicle-mounted multimodal perception. The system is mainly composed of three modules: local training, model uploading, and model aggregation. The main innovations are decision-level multimodal fusion based on ensemble learning, dynamic selection of modal model uploading based on the interpretability tool Shapley value, and a multimodal federated learning method based on the combination of the two. The present invention can adapt to communication bandwidth limitations and modal heterogeneity of nodes, and realize vehicle-mounted multimodal perception tasks based on federated learning. The scope of protection is local decision-level multimodal fusion based on ensemble learning, dynamic selection of modal model uploading based on the interpretability tool Shapley value, and the overall system operation method. There are many studies on the multimodal federated learning framework, and the methods of transformation and improvement based on the present invention also fall within the scope of protection of the present invention.

Claims

1. A federated learning method for vehicle-mounted multimodal perception, characterized by: The specific steps include: Step 1: Collect multimodal data and transmit it to the on-board computer for data processing to build a local data set; Step 2: Each node trains the corresponding modal model in parallel for each modality in its local data set. The node uses the prediction results of each modal model to fuse through the integrated model, and uses the Shapley value to calculate the contribution of each modal model to the integrated model prediction to obtain the corresponding prediction result label set; Step 3: Model the node bandwidth limit and the size and contribution of each modal model, and formulate a dynamic strategy for uploading the selected model; Step 4: The node packages the model parameters, modal information, and sample quantity information data packets and uploads them to the server for aggregation according to the formulated model upload strategy; Step 5: The server receives the data packets from the nodes and performs weighted aggregation based on the number of samples of each data modality to obtain a global model for each modality. In step 6, the node downloads the global model of the corresponding mode, and updates the local single-modal models and the integrated model in combination with the local data, and continuously iterates the above process until the model converges.

2. The method for federated learning of vehicle-mounted multimodal perception according to claim 1, characterized in that: The multimodal data collected in step 1 is multimodal data formed by collecting the surrounding environment information of the connected vehicle through the camera, lidar and GPS installed in the connected vehicle.

3. The federated learning method for vehicle-mounted multimodal perception according to claim 1, characterized in that: After each node in step 2 trains the corresponding modal model in parallel for each modality in its local data set, it calculates the parameter quantity of each modal model and calculates the corresponding upload time in combination with the node bandwidth information; then the contribution degree and upload time of each modal model obtained are respectively Min-Max normalized, and a weighted calculation is used to obtain a comprehensive score. The initial upload probability of each modal model is determined by the comprehensive score, thereby formulating a dynamic model selection upload strategy.

4. The method for federated learning of vehicle-mounted multimodal perception according to claim 3, characterized in that: The dynamic selection model upload strategy is as follows: Set label set to Y k , the node communication bandwidth is B k , the multimodal dataset on the kth multimodal node Input into the corresponding unimodal model respectively Parallel training is performed, where represents the mth modal dataset of the kth node, M k For a specific mode, the node then converts the output of each modal model After the integrated model ω k Perform decision-level multimodal fusion to obtain the final prediction Right now: When the federated training model is updated, the node only uploads the single-modal model. Instead of uploading the local integrated model k ; The interpretability tool Shapley value is introduced to calculate the modal contribution, that is, to evaluate the integrated model ω k Input and output The relationship between: in, is the Shapley value of the input mode m, i.e. the modal contribution, is the output The subset of mode m is not included in k (y) is the ensemble model using only the set The corresponding predicted value is when the mode in is The absolute value of the modal model contribution and the corresponding single communication time The two benchmarks are normalized by Min-Max to obtain and is the unimodal model of the kth node mode m, B k is the communication bandwidth of the kth node; The weighted calculation of the two results in a comprehensive score: in is the comprehensive score of the kth node mode m, α s , α t It is an adjustable empirical parameter used to control the relationship between modal contribution and single communication time, so that modal models with high contribution and small model size can obtain higher comprehensive scores; The upload probability of each modality model is determined based on the comprehensive score: S m is the comprehensive score of the node internal mode m, Sum the comprehensive scores of all modes within the node; Calculate entropy based on upload probability: Introducing entropy adjustment factor λ eaf Adjust upload probability: P i (2) =P i (1) ·(1+λ eaf ·(1-H(P (1) ))) Among them, λ eaf An adjustable parameter that controls the effect of entropy. When entropy is low, it means that the probability distribution is more concentrated. The adjusted upload probability is normalized to obtain the final probability distribution to ensure that the sum of the upload probabilities of the three modes is 1:

Images