Method, system and equipment for balanced training of multimodal Internet of Vehicles models for federated learning

Abstract

The present application provides a method for balanced training of a multimodal Internet of Vehicles model for federated learning, which mainly includes: building an Internet of Vehicles model on the server side and distributing it to N clients, the server side determining the clients participating in the model training based on the evaluation score and the dynamic client selection algorithm, the server side calculating the learning rate of each modal type of data in the local training set according to the learning rate adjustment formula, and the server side determining the modal optimization strategy for the next round of training based on the learning rate. Each client of the present application can select a suitable modality for training based on local conditions, which greatly improves the adaptability of the client. In the Internet of Vehicles environment, multimodal data can include vehicle position, speed, camera images, radar data, etc., which can comprehensively consider various information and improve the accuracy and adaptability of the model.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle network model training, and in particular to a method, system, and equipment for balanced training of multimodal vehicle network models for federated learning. Background Art

[0002] Multimodal learning is an important research direction in machine learning and artificial intelligence. It involves processing and integrating data and information from multiple modalities. The goal of multimodal learning is to improve the system's ability to understand the world and behave intelligently, because real-world tasks and decisions often require integrating multiple types of sensory input.

[0003] For example, real-world data carriers include text, voice, images, and video in multiple modalities. Deep learning models understand these data carriers differently than humans do. Some deep learning models cannot accept multimodal data. For example, OPEN AI's CHAT GPT3.5 can only accept and analyze text data, while KIMI can accept and analyze both text and image data. Clearly, KIMI is more convenient for communicating with users than CHAT GPT3.5.

[0004] Multimodal learning is a hot research topic. Some models that support multimodal input and output first parse non-text data into text and then convert it back into non-text data. This inconsistency and even contradiction in the parsing of multimodal data makes it difficult to optimize model training.

[0005] Therefore, the development of multimodal models still faces many technical challenges, especially the problem of training multimodal models for medical diagnosis faced by this application.

[0006] Therefore, the Internet of Vehicles model that performs multimodal learning and federated learning at the same time will face imbalance problems in multimodal data and federated learning, such as imbalance in training data and imbalance in client computing power.

[0007] The phenomenon of imbalanced multimodal training data includes: imbalanced data modalities uploaded by different clients, and different data modalities have different inherent characteristics and information volumes.

[0008] The phenomena that cause multimodal computing power imbalance include: the computing power of each client device is different. Due to the characteristics of federated learning, clients with weak computing power will slow down the learning process of the entire model.

[0009] Therefore, in order to train a multimodal Internet of Vehicles model and solve the above-mentioned imbalance problem at the same time, the present application provides a balanced training method, system and device for a multimodal Internet of Vehicles model for federated learning. Summary of the Invention

[0010] To overcome the problems existing in the related art, the present application provides: 1. A balanced training method for a multimodal Internet of Vehicles model for federated learning, characterized by comprising:

[0011] S1. Build an Internet of Vehicles model on the server and distribute it to N clients, where N is an integer greater than or equal to 1.

[0012] S2. The server determines an evaluation score of the client according to a client evaluation formula;

[0013] S3. The server determines the client to participate in model training based on the evaluation score of the nth client and the dynamic client selection algorithm, ensuring efficient resource utilization and modal clients required for modal balancing;

[0014] S4. The server obtains intermediate features and performs model training based on the intermediate features; the intermediate features are extracted and uploaded by each client participating in the model training based on a local training set; the intermediate features include local data features collected by N clients; the training parameters of the global model are adjusted based on the Shapley value of modality i to ensure that the contribution of each modality is fairly reflected in the training process;

[0015] S5. The server calculates the learning rate of each modality type data in the total training set based on the learning rate to ensure that the contributions of different modalities in the training process are treated fairly and to prevent the data of a certain modality from having too great an impact on the model;

[0016] S6. The server determines a modality optimization strategy for the next round of training based on the learning rate;

[0017] S7. The server distributes the updated modality optimization strategy to the clients participating in the training;

[0018] S8. The client adjusts local feature extraction parameters according to the modal optimization strategy and extracts intermediate features from the respective local data sets;

[0019] S9. Determine whether the global model has completed training. If so, output the trained global model. If not, execute step S3.

[0020] In one embodiment, the client evaluation formula is:

[0021] Score n =α1×DII n +α2×MCI n +β×CPI n

[0022] Among them, DII n Data completeness index of the local training set provided to the nth client, MCI n The modal coverage of the local training set provided to the nth client, CPI n is the computing power of the nth client, α1, α2 and β are the coefficients for adjusting the weights respectively, Score n is the evaluation score of the nth client.

[0023] In one embodiment, the dynamic customer selection algorithm is:

[0024] C n =Score n +γ×MF n

[0025] Among them, γ is the coefficient that adjusts the importance of modal factors, MF n Indicates the diversity of modal types provided by the nth client and the score of the current server's demand for the modality.

[0026] In one embodiment, the learning rate is calculated as follows:

[0027]

[0028] Among them, LR m is the learning rate of mode i, ERR i is the current error rate of modality i, TGT is the target error rate, γ is the learning rate adjustment factor, LR base is the initial learning rate.

[0029] In one embodiment, after the current round of training is completed, the server will use the model performance evaluation algorithm to determine the model performance;

[0030] The model performance evaluation algorithm is:

[0031] P=ω1×ACC+ω2×F1

[0032] Among them, ACC is the accuracy, F1 is the F1 score, ω1 and ω2 are coefficients to adjust the importance of the two, and P is the model performance indicator after the current round of update;

[0033] Adjust the dynamic customer selection algorithm and modality optimization strategy based on the model performance indicators to optimize the results of subsequent training batches.

[0034] In one embodiment, the modal weight parameter is determined based on the Shapley value of the modality i in the training set, and the calculation formula of the Shapley value is:

[0035]

[0036] Among them, ψ i (f) is the Shapley value of mode i, M is the set of modes, m is the total number of modes, T is any mode set that does not contain mode i, f(T) is the model performance value of mode set T, and f(T∪{i}) is the model performance value of mode set T combined with mode i;

[0037] The calculation formula of the modal weight parameter is:

[0038]

[0039] Among them, W i,s represents the weight of mode i in sample S, ψ i,s represents the Shapley value of mode i.

[0040] In one embodiment, the modal optimization strategy specifically includes:

[0041] The client performs feature extraction according to the loss function of intermediate feature extraction;

[0042] The intermediate feature extraction damage function is:

[0043]

[0044] Among them, Loss i is the loss function for mode i, K is the number of samples, Y k is the true label of the kth sample, X k is the input data of the kth sample, f(X k θ i ) is the predicted value of the feature extraction function for mode i, θ i is the weight parameter of the feature extraction function. Specifically, the θ of the feature extraction loss function i Adjust optimization based on learning rate.

[0045] A second aspect of the present application provides a distributed model training system, comprising: a client and a server;

[0046] The server is used to execute the aggregation step of the global model in the multimodal Internet of Vehicles model balancing training method;

[0047] The client is used to execute the local training step in the multimodal Internet of Vehicles model balancing training method.

[0048] The third aspect of the present application provides a distributed model training device, which is based on the multimodal Internet of Vehicles model balanced training method and is used for training a multimodal Internet of Vehicles model.

[0049] The technical solution provided by this application may have the following beneficial effects:

[0050] (1) This application uses a dynamically adjusted learning strategy to adjust the learning process based on the characteristics of different modalities, thereby allowing for a certain degree of error in the selection phase. This ensures that even modalities with relatively weak data volume or quality receive appropriate attention and reinforcement, ensuring a more balanced and efficient learning process for the entire model.

[0051] (2) This application uses a dynamic client selection algorithm to select qualified clients to participate in federated training. This application allows clients that provide the most valuable data to participate in the current training batch, thereby avoiding the need for all clients to participate in every training session, significantly reducing the computational burden. In particular, when the performance of participating devices varies, the computational requirements for low-performance devices can be significantly reduced.

[0052] (3) The present application can be easily integrated into various existing multimodal federated learning frameworks without changing the original model architecture. Instead, the present invention can be seamlessly integrated into various existing technologies by introducing dynamic client selection and modality optimization mechanisms during the model training process.

[0053] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0055] Figure 1 This is a flow chart of a balanced training method for a multimodal Internet of Vehicles model for federated learning as shown in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of a dynamic customer selection algorithm shown in an embodiment of the present application;

[0057] Figure 3 This is a schematic diagram of the modal optimization strategy shown in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0059] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0060] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various data, these data should not be limited to these terms. These terms are only used to distinguish data of the same type from each other. For example, without departing from the scope of this application, the first data may also be referred to as the second data, and similarly, the second data may also be referred to as the first data. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0061] Example 1

[0062] In the multimodal training of IoV models, IoV data is a typical example of multimodal data, including vehicle location data, speed data, camera images, radar data, and in-vehicle sensor data. This data is crucial for the intelligent management and optimization of IoV systems.

[0063] In everyday connected car applications, such as autonomous driving systems, vehicles use cameras to capture road images, radar to detect the surrounding environment, and in combination with data from onboard sensors to make driving decisions. While driving, a vehicle needs to comprehensively analyze information such as road images, radar data, and vehicle speed to determine whether to slow down or stop.

[0064] In the pre-trained Internet of Vehicles model, the vehicle's multimodal data (such as camera images and radar data) is input, and the corresponding driving decision results can be obtained based on the characteristics of each modal data and the vehicle's operating status characteristics.

[0065] Therefore, in the field of deep learning for IoV applications, IoV data of visual modalities and other perceptual modalities need to be used as training datasets.

[0066] In reality, different vehicles may be equipped with different sensing devices, such as cameras, lidar, or millimeter-wave radar. If a larger proportion of camera image modalities is input during IoV model training, the IoV model will lose its ability to accurately judge data from other modalities.

[0067] To improve the adaptability of the IoV model in multimodal data analysis, it is necessary to adjust the weights of the training datasets of each client. That is, the IoV model provided by the embodiments of the present application can be applied to different models of vehicles to avoid performance differences due to different sensor devices installed in the vehicles.

[0068] Specifically, if excessive millimeter-wave radar modal data is used in the training of the Internet of Vehicles model, the Internet of Vehicles model's ability to reason about data of other modal types will decrease, and the accuracy of the Internet of Vehicles model will decrease in vehicles without millimeter-wave radar.

[0069] Therefore, the embodiment of the present application provides a method for balancing training of a multimodal Internet of Vehicles model for federated learning, such as Figure 1 As shown, the following steps are included:

[0070] S1. Build an Internet of Vehicles model on the server and distribute it to N clients, where N is an integer greater than or equal to 1.

[0071] In this embodiment of the present application, the client is a vehicle equipped with an autonomous driving system, and the client's local dataset is data collected by the vehicle during autonomous driving. For example, a vehicle equipped with a lidar generates a real-time 3D point cloud image during autonomous or assisted driving. The vehicle extracts features from the 3D point cloud image based on a local vehicle network model and performs autonomous driving operations based on the extracted features.

[0072] The server of the embodiment of the present application collects various data constructed by each vehicle participating in the training for the training of the vehicle network model, and then distributes the vehicle network model optimized in each round to the client (i.e., the vehicle).

[0073] S2. The server determines an evaluation score of the client according to a client evaluation formula;

[0074] Specifically, the client evaluation formula is:

[0075] Score n =α1×DII n +α2×MCI n +β×CPI n

[0076] Among them, DII nData completeness index of the local training set provided to the nth client, MCI n The modal coverage of the local training set provided to the nth client, CPI n is the computing power of the nth client, α1, α2 and β are the coefficients for adjusting the weights respectively, Score n is the evaluation score of the nth client.

[0077] In this embodiment, the DII, or Data Integrity Index, indicates the integrity and usability of client data. For example, if a client's dataset contains no missing values ​​and every data field is verified and accurate, its DII can be set to 1. Conversely, if the dataset contains 30% missing or erroneous data, the DII can drop to 0.7.

[0078] For example, before sending a local data set to the server, the nth client first calculates the hash function value of the local data set to be sent and sends the hash function value to the server. After receiving the local data set, the server recalculates and verifies whether the hash function value of the received local data set is correct.

[0079] Specifically, a hash function can transform data of arbitrary length into an output of fixed length.

[0080] Optionally, MCI n Indicates how often data of different modalities are accessed and used during training. For example, if a client uses image data 80 times out of 100 times during training and text data 20 times, the coverage of the image modality is 0.8 and the coverage of the text modality is 0.2.

[0081] Optionally, in the local training set of the nth client, the modality coverage reflects the number of modality types in the local dataset. For example, if the modality types of a local training set include image modality, text modality, and 3D point cloud modality, then the modality coverage of the local training set is the ratio of the number of client modality types to the total number of server modality types.

[0082] In this embodiment of the application, the so-called CPI is a weighted score of the number of cores and frequency of the client's CPU (i.e., processor). In this embodiment of the application, each client must first undergo a multimodal data quality and performance assessment. A specific quality assessment algorithm is used to evaluate the integrity of the client's data, the diversity of the modalities, and the client's computing power. This assessment is related to the weight in the dynamic selection.

[0083] S3. The server determines the client to participate in model training based on the evaluation score of the nth client and the dynamic client selection algorithm;

[0084] In the embodiment of the present application, the server ensures efficient resource utilization and modality clients required for modality balancing based on the evaluation score and dynamic client selection algorithm.

[0085] See Figure 2 , Figure 2 This is a schematic diagram of the dynamic customer selection algorithm shown in an embodiment of the present application.

[0086] Specifically, the dynamic customer selection algorithm is:

[0087] C n =Score n +γ×MF n

[0088] Among them, γ is the coefficient that adjusts the importance of modal factors, MF n Indicates the diversity of modal types provided by the nth client and the score of the current server's demand for the modality.

[0089] Specifically, MF n The server scores the local training set sent by the nth client based on the data demand for modality i in the current training round. The so-called data demand for modality i is expressed by the demand coefficient δ i Demand coefficient δ i It can be dynamically adjusted according to the data volume requirements of each modality in the current round of training.

[0090] For example, MF n The calculation formula is

[0091]

[0092] In the examples of this application, δ i is the data requirement coefficient of mode i, and Pro i is the proportion of modality i data in the local dataset of the nth client.

[0093] Before the server performs the next round of training, the server calculates the C value of the nth client. n Execute dynamic customer selection algorithm. For example, C n Clients with scores above the preset threshold undergo model training.

[0094] The dynamic client selection algorithm specifically considers the modality specificity provided by each client to ensure the data quality of the total training set.

[0095] During the training process on the server side, in order to adjust the weight of each modality data and ensure the fairness and effectiveness of each modality during the training process, the server side further executes step S4.

[0096] S4. The server obtains intermediate features and performs model training based on the intermediate features; the intermediate features are extracted and uploaded by each of the clients participating in the model training based on the local training set; the intermediate features include local data features collected by N clients; the training parameters of the global model are adjusted according to the Shapley value of modality i to ensure that the contribution of each modality is fairly reflected in the training process.

[0097] Specifically, the calculation formula of Shapley value is:

[0098]

[0099] Among them, ψ i (f) is the Shapley value of mode i, M is the set of modes, m is the total number of modes, T is any mode set that does not contain mode i, f(T) is the model performance value of mode set T, and f(T∪{i}) is the model performance value of mode set T combined with mode i;

[0100] Furthermore, the calculation formula of the modal weight parameter is:

[0101]

[0102] Among them, W i,s represents the weight of mode i in the current training data, ψ i,s Represents the Shapley value of mode i in the current training data.

[0103] The vehicle network model training method of the embodiment of the present application uses Shapley's algorithm to calculate the value quantification index of each mode in the total training set, and adjusts the weight of each mode in the local training set according to the value quantification index.

[0104] After completing the current round of training, for the clients selected to participate in the training, the server will define a specific learning rate and iteration adjustment scheme for each modality based on the performance feedback of the global model to address the performance imbalance between modalities.

[0105] Therefore, the server further performs step S5 to calculate and issue adjustment parameters to ensure that each client can make corresponding adjustments based on its own modal characteristics.

[0106] See Figure 3 , Figure 3 This is a schematic diagram of the modal optimization strategy shown in an embodiment of the present application.

[0107] S5. The server iterates according to the learning rate of each modality type data;

[0108] In this embodiment of the present application, to ensure that the contributions of different modalities are treated fairly during training and to prevent data from a particular modality from having an excessive impact on the model, the server will define a specific learning rate and iteration adjustment scheme for each modality based on the performance feedback of the global model to address the performance imbalance between modalities.

[0109] Specifically, the calculation formula of the learning rate is:

[0110]

[0111] Among them, LR m is the learning rate of mode i, ERR i is the current error rate of modality i, TGT is the target error rate, γ is the learning rate adjustment factor, LR base is the initial learning rate.

[0112] In an embodiment of the present application, the server calculates the learning rate of each modality type during training. If the current error rate of modality i is higher than the target error rate, the learning rate of modality i is reduced.

[0113] Furthermore, the server formulates a modality optimization strategy based on the learning rates of different modality types.

[0114] S6. The server determines a modality optimization strategy for the next round of training based on the learning rate;

[0115] Specifically, if the learning rate of the training data of modality i varies greatly, it is necessary to reduce the feature extraction weight of modality i in each client.

[0116] S7. The server distributes the updated modality optimization strategy to the clients participating in the training;

[0117] S8. The client adjusts local feature extraction parameters according to the modal optimization strategy and extracts intermediate features from the respective local data sets;

[0118] Specifically, the modal optimization strategy includes:

[0119] The client performs feature extraction according to the loss function of intermediate feature extraction;

[0120] Among them, the intermediate feature extraction loss function is:

[0121]

[0122] Among them, Loss i is the loss function for mode i, K is the number of samples, Y k is the true label of the kth sample, X kis the input data of the kth sample, f(X k θ i ) is the predicted value of the feature extraction function for mode i, θ i is the weight parameter of the feature extraction function. Specifically, the θ of the feature extraction loss function i Adjust optimization based on learning rate.

[0123] S9: Determine whether the global model has completed training. If not, re-execute step S3 until the model training on the server is completed.

[0124] The model performance evaluation algorithm is:

[0125] P=ω1×ACC+ω2×F1

[0126] Among them, ACC is the accuracy, F1 is the F1 score, ω1 and ω2 are coefficients to adjust the importance of the two, and P is the model performance indicator after the current round of update;

[0127] Therefore, in the embodiment of the present application, each client can select a modality suitable for itself for feature extraction according to local conditions, which greatly improves the adaptability of the client.

[0128] In an embodiment of the present application, after the client receives the new global model and modality optimization strategy, it can personalize the data weights of different modalities and send the extracted intermediate features to the server for global training.

[0129] Example 2

[0130] A distributed model training system, comprising: a client and a server;

[0131] The server is used to perform the global model aggregation step in the multimodal Internet of Vehicles model balancing training method described in Example 1;

[0132] The client is used to execute the local training step in the multimodal Internet of Vehicles model balancing training method described in Example 1.

[0133] Example 3

[0134] A distributed model training device is provided, which is based on the multimodal Internet of Vehicles model balancing training method described in Example 1 and is used for training a multimodal Internet of Vehicles model.

[0135] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0136] The embodiments of the present application have been described above. The above description is illustrative and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A balanced training method for a multimodal Internet of Vehicles model for federated learning, characterized by: include: S1. Build the Internet of Vehicles model on the server and distribute it to N clients; Said N is an integer greater than or equal to 1; S2. The server determines an evaluation score of the client according to a client evaluation formula; S3. The server determines the client to participate in model training based on the evaluation score of the nth client and the dynamic client selection algorithm; S4. The server obtains intermediate features and performs model training based on the intermediate features; the intermediate features are extracted and uploaded by each client participating in the model training based on a local training set; the intermediate features include local data features collected by N clients; Adjust the training parameters of the global model according to the Shapley value of mode i; S5. The server iterates according to the learning rate of each modality type data; S6. The server determines a modality optimization strategy for the next round of training based on the learning rate; S7. The server distributes the updated modality optimization strategy to the clients participating in the training; S8. The client adjusts local feature extraction parameters according to the modal optimization strategy and extracts intermediate features from the respective local data sets; S9, determine whether the global model has completed training, if so, output the trained global model; if not, execute step S3; In step S4, the modal weight parameter is determined based on the Shapley value of the mode i in the training set. The calculation formula of the Shapley value is: in, is modal The Shapley value of is a collection of modes, is the total number of modes, Is any one that does not contain a modal The modal set of is a modal set The model performance value, is a modal set With modal The performance value of the combined model; The calculation formula of the modal weight parameter is: in, Represents a sample Medium mode The weight of Indicates modality The Shapley value of 2. The multimodal vehicle network model balancing training method according to claim 1 is characterized in that: The client evaluation formula is: in The data integrity index of the local training set provided by the nth client, The modal coverage of the local training set provided to the nth client, is the computing power of the nth client, and are the coefficients for adjusting the weights, For the The evaluation score of each client.

3. The method for balancing a multimodal Internet of Vehicles model according to claim 2, characterized in that: The dynamic customer selection algorithm is: in, is the coefficient that adjusts the importance of the modal factor, Indicates the diversity of modal types provided by the nth client and the score of the current server's demand for the modality.

4. The method for balancing a multimodal Internet of Vehicles model according to claim 1, wherein: The learning rate is calculated as follows: in, is the learning rate of mode i, is the current error rate of modality i, is the target error rate, is the learning rate adjustment factor, is the initial learning rate.

5. The multimodal Internet of Vehicles model balancing training method according to claim 1, characterized in that: After the current round of training is completed, the server will determine the model performance based on the model performance evaluation algorithm; The model performance evaluation algorithm is: in, is the accuracy, is the F1 score, and is the coefficient that adjusts the importance of the two, The updated model performance index for the current round; Adjust the dynamic customer selection algorithm and modality optimization strategy based on the model performance indicators to optimize the results of subsequent training batches.

6. The method for balancing a multimodal Internet of Vehicles model according to claim 1, characterized in that: The modal optimization strategy specifically includes: The client performs feature extraction according to the loss function of intermediate feature extraction; The intermediate feature extraction damage function is: in, For modal The loss function is, K is the number of samples, is the true label of the i-th sample, is the input data of the i-th sample, is the predicted value of the feature extraction function for mode i, is the weight parameter of the feature extraction function.

7. A distributed model training system, characterized in that: include: Client and server; The server is used to perform the global model aggregation step in the multimodal vehicle network model balancing training method according to any one of claims 1 to 6; The client is used to execute the local training step in the multimodal vehicle network model balancing training method described in any one of claims 1 to 6.

8. A distributed model training device, characterized in that: The multimodal Internet of Vehicles model balancing training method according to any one of claims 1 to 6 is used for training a multimodal Internet of Vehicles model.

Images