A dynamic sparse training method for Internet of Vehicles federation based on cluster expansion

Abstract

The present invention belongs to the field of vehicle-to-vehicle (VoI) network communications and discloses a cluster-expanding method for dynamic sparse training of VoI federated learning. The method comprises constructing a hierarchical federated learning training architecture for VoI scenarios: the architecture comprises a two-tier client-server structure, including edge servers and a road system. Based on the hierarchical federated learning training architecture, dynamic sparse training is performed: under the coordination and management of edge servers, training nodes train a sparse network structure based on local data and periodically update the sparse network structure according to predefined reconstruction and update criteria. Finally, a global network structure that meets the target sparsity is aggregated and reconstructed at the edge servers.

Description

Technical Field

[0001] The present invention relates to the field of vehicle network communication technology, and in particular to a dynamic sparse training method for vehicle network federated learning based on cluster expansion. Background Art

[0002] The Internet of Vehicles (IoV) is a complex intelligent network integrating multiple elements, including vehicles, roadside units (RSUs), base stations, and personnel. Frequent inter-node communication during federated learning training consumes significant bandwidth. Furthermore, the increasingly complex neural network models commonly used in IoV scenarios for tasks such as object detection and recognition further exacerbate the pressure on communication and computing resources. Several technologies have attempted to reduce the communication overhead during federated learning training. For example, some studies have employed model quantization techniques to reduce the precision of transmitted parameters, thereby reducing communication overhead. However, this approach has limited compression effectiveness and may introduce significant loss of model accuracy. Furthermore, some methods utilize knowledge distillation mechanisms to obtain a student model with good performance and low computational complexity, reducing communication resource consumption. However, this approach also has significant drawbacks: it requires additional computing resources and a large amount of high-quality data samples to assist in the training process, thus presenting significant challenges in practical application.

[0003] Therefore, the communication optimization mechanism based on dynamic sparse training has gradually become a research hotspot. By sparsifying the network during the training process, the communication and computing overhead in federated learning training can be significantly reduced. However, the existing dynamic sparse training methods still face challenges in the Internet of Vehicles scenario, such as the mobility of the client and the unreliability of the communication link. At the same time, when the existing methods achieve a higher communication overhead optimization effect, they will bring about a large loss of model accuracy. Therefore, how to implement an efficient federated dynamic sparse training mechanism in the Internet of Vehicles scenario is still a problem that needs to be solved urgently. The present invention designs a method for federated dynamic sparse training of the Internet of Vehicles based on cluster expansion. By quantifying the computing power and data distribution of the vehicle client, dynamically dividing the clusters, and performing expansion training on the basic global sparse model through the cluster, the accuracy of the trained model is improved, making it more adaptable to the unique network environment of the Internet of Vehicles. Summary of the Invention

[0004] Based on the deficiencies of the existing technology, the main purpose of the present invention is to provide a cluster expansion-based dynamic sparse training method for the Internet of Vehicles federation, which is suitable for Internet of Vehicles scenarios. It is a dynamic cluster partitioning algorithm based on the degree of system and data heterogeneity.

[0005] Its main technical solutions are as follows: a cluster-expanding federated dynamic sparse training method for the Internet of Vehicles (IoV) system, which constructs a hierarchical federated learning training architecture for IoV scenarios: the hierarchical federated learning training architecture for IoV scenarios is a two-layer client-server structure, including edge servers and road systems; federated dynamic sparse training is performed based on the hierarchical federated learning training architecture for IoV scenarios: under the coordination and management of edge servers, training nodes train sparse network structures based on local data, and periodically update the sparse network structures according to predefined reconstruction and update criteria; finally, a global network structure that meets the target sparsity is aggregated and reconstructed at the edge servers.

[0006] The edge server serves as a central node and is used to:

[0007] a) Pre-training of neural network models: Pre-training is performed on public datasets using local computing resources to obtain an initialized sparse network structure. This structure is then sparsified based on weight standards and sparsity requirements for subsequent formal training.

[0008] b) Vehicle client clustering: The kinematic information and vehicle terminal resource information of vehicles entering the area covered by the edge server are collected and quantified. The binary division is continued until the result satisfies the actual exploration rate between vehicle clusters reaching 100%, and the client cluster results are obtained.

[0009] c) Federated training process management: responsible for issuing sparse network structures and aggregating client sparse network structures to form a global sparse network structure, and reconstructing the aggregated model according to the predefined target sparsity.

[0010] The process of obtaining the initialized sparse network model is as follows: the edge server pre-trains the neural network using a public dataset, and obtains a sparse network structure that meets the target sparsity through a pre-defined weight amplitude standard as the initial sparse network structure; in the pre-training stage, the layer importance factor γ is introduced to dynamically determine the sparsity of different network layers in the sparse network structure, and the layer importance factor γ of the lth layer is l The calculation formula is as follows:

[0011]

[0012] Among them, m l (1) is the number of non-zero parameters in the lth layer, m l (0) is the number of parameters with zero weights in layer l.

[0013] The vehicle client cluster division is based on resource and model similarity; the initial target number of clusters is set to 2, and each cluster is recursively divided. After each division is completed, the actual exploration rate of each cluster of the current division result is calculated. When the total actual exploration rate of the cluster is 100%, the cluster division is terminated, otherwise the cluster division is continued;

[0014] 1) After receiving the content uploaded by the vehicle node, the edge server normalizes the vehicle terminal resource information to obtain the vehicle resource vector The calculation formula is as follows:

[0015]

[0016] where f i is the CPU main frequency of vehicle client i terminal, r i is the data transmission rate of vehicle client i terminal, a i The available memory size of the vehicle client i terminal;

[0017] 2) The server quantifies the model similarity of the currently trained model w of each vehicle node received, so as to facilitate the subsequent similarity measurement of all features. The calculation formula is as follows:

[0018]

[0019] where w i is the current local sparse network model of vehicle i and w j is the current local sparse network model of vehicle j;

[0020] 3) Comprehensively consider the vehicle's kinematic information, vehicle terminal resource information, and the similarity of the current training model to improve the stability of the clustering results. The similarity function calculation formula is as follows:

[0021]

[0022] Among them, α, β, and γ are hyperparameters used to adjust the impact of speed, resources, and model similarity on quantifying the similarity between vehicles;

[0023] 4) Expansion Exploration Rate: The sparse network model received by the vehicle cluster from the central node is defined as the base model. Each vehicle cluster first calculates the total number of resources of the current vehicle cluster and calculates the target expansion exploration rate of the current vehicle cluster based on this. The average resource information of the current cluster is calculated, and the size of the trainable cluster local sparse network model is calculated based on this average. The expansion exploration rate that can be achieved compared with the base model is then calculated and the actual expansion exploration rate is compared with the target expansion exploration rate. When the actual exploration rate is not less than the target exploration rate, the cluster stops dividing. The cluster expansion exploration rate calculation formula is as follows:

[0024]

[0025] The operator ∪ represents merging the exploration results of all clusters, M z represents the local sparse network model trained by cluster z, |W| counts the total number of parameters of the original dense network, and ∥·∥0 represents the l0 norm, which is used to count the total number of all explored parameters;

[0026] 5) Target Expansion Exploration Rate: When performing dynamic clustering, before each clustering using the Kmeans algorithm, the target exploration rate of the current cluster must be calculated. This is calculated by measuring the ratio of the total computing resources owned by all members of the current vehicle cluster to the total computing resources of all vehicle clusters, and multiplying this ratio by 100% to obtain the target expansion exploration rate. The calculation formula is as follows:

[0027]

[0028] Among them S i is the computing resources of the current cluster, is the total computing resources of all clusters.

[0029] 6) Actual expansion exploration rate: When performing dynamic cluster division, in addition to calculating the target expansion exploration rate, it is also necessary to calculate the actual expansion exploration rate of the cluster. The normalized resource vector interval in the cluster members calculated by formula (2) is recorded as The specific calculation formula is as follows:

[0030]

[0031] where R * It is a predefined parameter used to define the exploration rate that can be achieved per unit resource vector;

[0032] After the cluster division is completed, the average model structure of the cluster is compared with the global model structure based on formula (3) for similarity, and sorted in descending order. First, the basic model θ is calculated. base In the dense network model θ dense The complement of the above can be used to obtain the explorable model θ ava Then, based on the actual expansion exploration rate of each cluster and the average sparse neural network model of the current vehicle cluster, additional parameter connections are randomly generated for each vehicle cluster in sequence to obtain the expansion part θ extend The model structure θ used in the next round of training i , then update the explorable model θ ava The calculation formulas are as follows:

[0033] θ ava =θdense \θ extend (8)

[0034] θ i =θ base ∪θ extend (9)

[0035] θ ava =θ ava \θ extend (10)

[0036] where θ dense is the complete dense network model, given before training; θ base is the edge server initialization sparse network model; θ i is the sparse network model structure of cluster i.

[0037] The road system is composed of vehicle clusters, and vehicle nodes are divided into cluster head nodes and cluster member nodes according to their different roles in the vehicle cluster structure;

[0038] a) Cluster member nodes: responsible for performing federated training based on locally collected data and uploading the trained sparse network structure to the cluster head node;

[0039] b) Cluster head node: It is not only responsible for local training, but also for aggregating models from cluster member nodes and uploading the aggregated models to the edge server to assist in updating and reconstructing the global model.

[0040] The Federated Dynamic Sparse Training (FedCET) mechanism is based on a parallel scalable federated dynamic sparse training mechanism. This mechanism divides a large number of vehicle clients into several clusters, allowing different clusters to explore different neural network connections in parallel beyond the basic sparse network structure. Through collaboration between clusters, it completes a complete exploration of all connections in the dense network structure from a spatial perspective, covers a sufficient number of connections during the training process, and dynamically optimizes and updates the structure of the global sparse network on this basis, thereby significantly improving the generalization ability of the global sparse model in high-sparse scenarios and achieving higher model accuracy.

[0041] When performing federated dynamic sparse training, the trained sparse network structure is represented by transmitting a sparse network mask. The method for generating a sparse network mask based on the local sparse network structure trained by the node is as follows: let W be the sparse neural network structure parameter matrix, the mask is a binary matrix M with the same shape as it, and each element m in M ​​is a,b The corresponding weight w in W a,b Decision, the specific rules are: if w a,b =0, then m a,b=0; if w a,b ≠0, then m a,b =1;

[0042] After generating the sparse network masks, the vehicle nodes in each cluster use these masks for local sparse training. The specific method is as follows: the vehicle node i of cluster z uses the corresponding sparse network mask M z Perform local sparse training; in the tth communication round, vehicle node i uses local data X i , optimized with learning rate η, the calculation formula is as follows:

[0043]

[0044] In order to further improve the efficiency of parameter exploration through training and optimize the sparse network structure, the parameter expansion exploration phase is introduced. The specific method is: every ΔT rounds, the sparse network structure is gradually fine-tuned through parallel parameter exploration between clusters. The goal is to obtain the optimal global sparse network structure M through iterative updates while meeting the sparsity requirements. * ; The edge server randomly explores new parameters of proportion f for each cluster to form a new sparse network structure; For the zth cluster, its sparse network structure is recorded as M z , by merging the sparse network structure of all clusters at the edge server; gradually fixing the structure of the sparse network through the cosine decay strategy, the value of the new exploration parameter ratio f is initialized to μ, and end The communication round decays to 0; in the tth communication round, the calculation formula of f is as follows:

[0045]

[0046] When f = 0, the parameter exploration stops, and the global sparse network structure obtained at this time is the optimal sparse network structure M * ,In the subsequent training, the global sparse network structure remains fixed until the training stops after convergence;

[0047] During aggregation, the sparse network structure trained for each vehicle cluster is personalized while ensuring low differences between clusters. Each sparse network structure is logically considered to be two parts:

[0048] 1) Public infrastructure base , these parameters are shared by all clusters;

[0049] 2) Private part w self , corresponding to the newly explored parameters of each cluster, which are private to the cluster;

[0050] Suppose there are n z clients, the model of cluster z after aggregation is:

[0051]

[0052] Among them, |D i | is the amount of data owned by vehicle node i, and They represent the basic public and private parameters of the i-th vehicle node, respectively. All clients participate in the aggregation of the public basic part, while the private part parameters are only aggregated and averaged among the clients that retain the current parameters. After the aggregation is completed, all vehicle nodes will receive the same public basic part parameters, while the private part parameters are independent between vehicle clusters.

[0053] After the central node completes receiving and aggregating the sparse network structures uploaded by all vehicle nodes, it performs sparse network reconstruction to meet the target sparsity requirement. The specific method is as follows: every fixed round ΔT, the parameter sensitivity score g defined by formula (14) is selected from the currently explored sparse network structure. j Select the most promising sparse network weights, comprehensively consider the influence of gradient information and weight size, and reconstruct a new global sparse network structure M with a sparsity of S+(1-S)f G , and serves as the starting point for the next round of parallel expansion sparse training; until the preset T end When f = 0 in the round, parameter exploration stops, and the sparse network structure of all vehicle terminals is exactly the same, with no private parts, marking the end of sparse network parameter exploration. After that, the network structure is used for training until convergence stops.

[0054]

[0055] Where L is the loss function;

[0056] Communication cost calculation: The communication cost of training a sparse network is positively correlated with the model size. When the client and server communicate with each other, only the retained parameters are transmitted. The sparse mask representing the network structure is only transmitted when it is updated. Assuming that the original size of the model is L and the target sparsity is S, the total communication cost of each communication round for a single client is

[0057]

[0058] in is the additional overhead of transmitting the updated mask structure.

[0059] Beneficial effects of the present invention: The present invention proposes a dynamic sparse training method for the Internet of Vehicles federation based on cluster expansion, which aims to improve the training efficiency and accuracy of the model in the complex network environment of the Internet of Vehicles by optimizing the use of communication and computing resources. Through the dynamic sparse training mechanism, the present invention significantly reduces the data transmission requirements between networks and reduces bandwidth consumption. Especially in the Internet of Vehicles environment where the data transmission cost is high, this reduction can effectively save resources. At the same time, the use of a parallel expansion training strategy enables different vehicle clusters to explore different connections of the neural network in parallel. This method not only optimizes the generalization ability of the global model, but also improves the model accuracy, especially when dealing with highly sparse network structures.

[0060] Furthermore, this invention uses edge servers to intelligently analyze the computing power and data distribution of vehicle clients and dynamically divide them into clusters, ensuring optimal resource allocation and utilization. This maximizes the actual exploration rate of each cluster, thereby improving the efficiency and effectiveness of overall training. Through this centralized management and intelligent allocation strategy, vehicle client resources are used more efficiently, while reducing overall training time and resource consumption.

[0061] After comparative experiments with the baseline algorithm, the algorithm proposed in this invention shows significant advantages in communication overhead, computational overhead, global model accuracy and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of a real-world Internet of Vehicles scenario;

[0063] Figure 2 Schematic diagram of the hierarchical IoV federated learning training architecture;

[0064] Figure 3 This is a schematic diagram of the dynamic sparse training process;

[0065] Figure 4 This is a schematic diagram of the parallel expansion-based dynamic sparse training process for the Internet of Vehicles federation;

[0066] Figure 5 Schematic diagram of model aggregation strategy;

[0067] Figure 6 To verify the relationship between parameter exploration rate and model performance

[0068] Figure 7 This is an ablation experiment for the pre-training stage;

[0069] Figure 8 This is a performance comparison chart of the federated dynamic sparse training algorithm;

[0070] Figure 9 This is a comparison chart of algorithm performance under different target sparsity;

[0071] Figure 10 Schematic diagram of the communication overhead and computational overhead of the FedCET algorithm under different sparsity. DETAILED DESCRIPTION

[0072] Build a hierarchical federated learning training architecture for the Internet of Vehicles scenario;

[0073] (1) Client-Server Two-Layer Structure: Considering the actual situation of the Internet of Vehicles, the infrastructure such as vehicle nodes and edge servers deployed by roadside units (RSUs) or base stations are divided into different layers. The overall training framework consists of two parts: edge servers and road systems.

[0074] 1) Edge server (usually deployed at a base station): As the central node, it is responsible for managing the entire federated learning training process. Its main functions include:

[0075] a) Pre-training of neural network models: Pre-training is performed on public datasets based on local computing resources to obtain an initialized sparse network structure with good performance. This structure is then sparsified based on weight standards and sparsity requirements for subsequent formal training.

[0076] b) Vehicle client clustering: Kinematic information (position, direction, speed) and vehicle terminal resource information (CPU frequency, available memory, data transfer rate) of vehicles entering the area covered by the edge server are collected and quantified. The Kmeans algorithm is used to continuously perform binary partitioning. When the partitioning result satisfies the structural exploration rate between vehicle clusters reaching 100% (the trained sparse network structure is the same as the original dense network structure), the clustering is stopped and the client clustering result is obtained.

[0077] c) Federated training process management: responsible for issuing sparse network structures and aggregating client sparse network models, and reconstructing the aggregated models according to the predefined target sparsity.

[0078] 2) Road system: It consists of vehicle clusters, where vehicle nodes are divided into cluster head nodes and cluster member nodes according to their different roles in the cluster structure.

[0079] a) Cluster member node: responsible for performing federated training based on locally collected data and uploading the trained sparse network structure to the cluster head node.

[0080] b) Cluster head node: It is not only responsible for local training, but also for aggregating models from cluster member nodes and uploading the aggregated models to the edge server to assist in updating and reconstructing the global model.

[0081] (2) Federated dynamic sparse training mechanism: Under the coordination and management of the central node, the training nodes train the sparse network structure based on local data, and periodically update the network structure according to the predefined reconstruction update criteria. Finally, a global network structure that meets the target sparsity is aggregated and reconstructed at the central node.

[0082] Initialize the sparse network structure;

[0083] (1) Pre-training stage: Different from the traditional research method that uses random methods to generate sparse network structures, the present invention relies on the powerful computing power of the edge server, uses public data sets to pre-train the neural network, and obtains a sparse network structure that meets the target sparsity through the weight amplitude standard as the initial sparse network structure.

[0084] (2) Calculating layer sparsity: Different from the traditional mechanism that fixes the sparsity of each layer in the neural network, the present invention introduces the layer importance factor γ in the pre-training stage to dynamically determine the sparsity of each layer. The layer importance factor γ of the lth layer is l The calculation formula is as follows:

[0085]

[0086] where m l (1) is the number of non-zero parameters in the lth layer, m l (0) is the number of parameters with zero weights in layer l.

[0087] Vehicle client cluster division mechanism based on resource and model similarity;

[0088] (1) Vehicle information processing: When a vehicle enters the coverage area of ​​the edge server, the vehicle's speed s, direction and other driving information, as well as resource information such as CPU main frequency f, data transmission rate r, available memory size a and the current training model w are uploaded to the central node.

[0089] 1) After receiving the content uploaded by the vehicle, the server normalizes the resource information. The calculation formula is as follows:

[0090]

[0091] Get the vehicle's resource vector

[0092] 2) The server quantifies the similarity of the currently trained model w of the received vehicle using the following formula:

[0093]

[0094] where w i and w jare the current local sparse neural network model structures of vehicle i and vehicle j respectively.

[0095] (2) Dynamic clustering algorithm based on Kmeans method: The initial target number of clusters is set to 2, and each cluster is recursively divided. After each division, the actual exploration rate of each cluster in the current division result is calculated. When the total actual exploration rate of the cluster is 100%, the cluster division is terminated, otherwise the cluster division is continued.

[0096] 1) Similarity function: This function comprehensively considers the vehicle client's kinematic information, computing resource conditions, and model similarity to improve the stability of clustering results and reduce the impact of resource and data heterogeneity on federated learning training. The calculation formula is as follows:

[0097]

[0098] Among them, α, β, and γ are hyperparameters used to adjust the impact of speed, resources, and model similarity on quantifying the similarity between vehicles.

[0099] 2) Expansion Exploration Rate: The sparse network model received by the cluster from the central server is defined as the base model. Each cluster first calculates the total number of resources of the current cluster, and then calculates the target expansion exploration rate of the current cluster based on this. Next, the average resource information of the current cluster is calculated, and the maximum trainable model size is calculated based on this average value. The expansion exploration rate that can be achieved compared with the base model is then calculated, and the actual expansion exploration rate is compared with the target expansion exploration rate. When the actual expansion exploration rate is not less than the target expansion exploration rate, the cluster stops dividing. The cluster expansion exploration rate calculation formula is as follows:

[0100]

[0101] The operator ∪ merges the exploration results of all clusters, |W| counts the total number of parameters in the original dense network, and ∥·∥0 represents the l0 norm, which is used to count the total number of explored parameters. The merge operation only represents a logical merger, which is used to represent the set of parameters explored by all clusters.

[0102] according to Figure 6 Experimental results show that the value of R is positively correlated with the performance of the sparse network. A larger parameter exploration space and higher parameter exploration efficiency can more quickly and reliably find a sparse network structure with better performance, thereby significantly improving the performance of dynamic sparse training.

[0103] 3) Target Exploration Rate: When performing dynamic clustering, before each clustering using the Kmeans algorithm, the target exploration rate of the current cluster must be calculated. This calculation is achieved by measuring the ratio between the total computing resources owned by all members in the current cluster and the total computing resources in all clusters. This ratio is then multiplied by 100% to obtain the target exploration rate. The calculation formula is as follows:

[0104]

[0105] Among them S i is the computing resources of the current cluster, is the total computing resources of all clusters.

[0106] 4) Actual exploration rate: When performing dynamic cluster division, in addition to calculating the target exploration rate, it is also necessary to calculate the actual exploration rate of the cluster. The normalized resource vector interval of the cluster members calculated by formula (2) is recorded as The specific calculation formula is as follows:

[0107]

[0108] where R * It is a predefined parameter used to define the exploration rate that can be achieved per unit resource vector.

[0109] (3) Neural network model expansion mechanism: After the cluster division is completed, the average model structure of the cluster is compared with the global model structure based on formula (3) for similarity, and sorted in descending order. First, the explorable model θ is obtained. ava ,Then based on the expansion exploration rate of each cluster and the current cluster model structure, additional parameter connections are randomly generated for each cluster in order to obtain the expansion part θ extend And the model structure θ used in the next round of training i , then update the explorable model θ ava The calculation formulas are as follows:

[0110] θ ava =θ dense \θ extend

[0111] θ i =θ base ∪θ extend

[0112] θ ava =θ ava \θ extend

[0113] where θ dense is the complete dense network model structure, θ baseis the initial sparse network structure on the server side, θ i is the sparse model structure of cluster i.

[0114] Based on parallel expansion of the federated dynamic sparse training mechanism;

[0115] Based on the parallel expansion of sparse training methods, a new fully sparse federated learning training framework, FedCET, is proposed. While reducing client computing and communication overheads, the performance of the global sparse model is not reduced. Specifically, FedCET divides a large number of vehicle clients into several clusters, allowing different clusters to explore different neural network connections in parallel beyond the basic sparse network structure. Through collaboration between clusters, it completes a complete exploration of all connections in the dense network structure from a spatial perspective, covers enough connections during the training process, and dynamically optimizes and updates the structure of the global sparse network on this basis, thereby significantly improving the generalization ability of the global sparse model in high-sparse scenarios and achieving higher model accuracy.

[0116] (1) Client-side local sparse training algorithm: Vehicle client i of cluster j uses the corresponding network structure mask M j Perform local sparse training. In the tth communication round, client i uses local data X i , optimized with learning rate η, the calculation formula is as follows:

[0117]

[0118] (2) Parameter expansion exploration: Every ΔT rounds, the sparse network structure is gradually fine-tuned through parallel parameter exploration between clusters. The goal is to obtain the optimal global sparse network structure M through iterative updates while meeting the sparsity requirements. * The server randomly explores new parameters of proportion f for each cluster to form a new sparse network structure. For the jth cluster, its sparse network structure is recorded as M j By merging the network structures of all clusters on the server side, we can logically achieve full coverage of network structure parameters. FedCET uses the cosine decay strategy to gradually fix the structure of the sparse network, and initialize the value of the new exploration parameter ratio f to μ. end The communication round decays to 0. In the tth communication round, the calculation formula of f is as follows:

[0119]

[0120] When f = 0, the parameter exploration stops, and the global sparse network structure obtained at this time is the optimal sparse network structure M * , the global sparse network structure remains fixed in subsequent training until the training stops after convergence.

[0121] (3) Model aggregation: Since the sparse network structures trained by different clusters are different, the aggregation brings the challenge of model heterogeneity. In the FedCET process, a new aggregation method is adopted to ensure the personalization of the network structure trained by each cluster during aggregation while ensuring that the differences between clusters are low. Each network structure is logically regarded as two parts:

[0122] 1) Public infrastructure base , these parameters are shared by all clusters.

[0123] 2) Private part w self , corresponding to the newly explored parameters of each cluster, are private to the cluster.

[0124] The reason for dividing the model structure into two parts is that, on the one hand, the public part provides the possibility of aggregating data from all clusters and training a globally representative model, and on the other hand, the private part is used to achieve parameter expansion exploration and thus optimize the structure of the global sparse network. Assume that there are n j clients, the model of cluster j after aggregation is:

[0125]

[0126] where |D i | is the amount of data owned by client i, and They represent the basic public and private parameters of the i-th client respectively. All clients participate in the aggregation of the public basic part, while the private part parameters are only aggregated and averaged among the clients that retain the current parameters. After the aggregation is completed, all clients will receive the same public basic part parameters, while the private part parameters are independent among clusters.

[0127] In addition, the proportion f of the newly explored parameters in point (2) also determines the degree of model heterogeneity between clusters. An excessively large f will indirectly affect the stability of sparse training and aggregation. Therefore, the common basic part occupies the majority of the total training parameters and is shared among clusters as a common structure, which can be fully generalized to all clusters for training.

[0128] (4) Sparse network reconstruction: Every fixed round ΔT, the parameter sensitivity score g defined by formula (14) is selected from the currently explored network structure. j Select the most promising parameter connections, comprehensively consider the influence of gradient information and weight size, and reconstruct a new global sparse network structure M with a sparsity of S+(1-S)f G , and serves as the starting point for the next round of parallel expansion sparse training. Until the preset T endWhen f=0 in the round, parameter exploration stops, the sparse network structure of all terminals is exactly the same, and there is no private part anymore, which marks the end of the sparse network parameter exploration. After that, the network structure is used for training until convergence stops.

[0129]

[0130] Where L is the loss function.

[0131] (5) Communication overhead calculation: The communication overhead of training a sparse network is positively correlated with the model size. When the client and server communicate with each other, only the retained parameters are transmitted. The sparse mask representing the network structure is only transmitted when it is updated. Assuming that the original size of the model is L and the target sparsity is S, the total communication overhead of each communication round for a single client is

[0132]

[0133] in is the additional overhead of transmitting the updated mask structure.

[0134] Algorithm 1 below is the pseudo code of the federated dynamic sparse training algorithm based on cluster expansion.

[0135]

[0136]

[0137] Algorithm 2 below is the pseudo code of the vehicle client clustering algorithm based on resource and model similarity.

[0138]

[0139] Output: Client cluster division results And the corresponding sparse model

[0140]

[0141]

[0142] The hardware and software environment of this implementation plan is shown in Table 1.

[0143] Table 1 Software and hardware configuration environment

[0144] name Configuration operating system Ubuntu 18.04 CPU i5-12400F Memory 16G

[0145] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings.

[0146] Step 1: Identify the application scenario: Figure 1As shown in the figure, for the universal Internet of Vehicles scenario, the vehicle collects a large amount of data through its own sensors and has a certain computing capability. Due to the serious mismatch between the collection speed and the body storage rate or the data transmission rate uploaded to the cloud, under the coordinated management of the edge server deployed near the base station, the vehicle performs federated learning training on the data source side, which can not only save the process of transmitting to the remote cloud, but also protect the privacy of the vehicle-collected data and adapt to the scenario requirements.

[0147] Step 2: System Architecture: Figure 2 As shown in the figure, this system is mainly divided into two levels. The edge server deployed at the base station serves as the central node, coordinating and managing the vehicles within its coverage area for federated learning training, and is also responsible for receiving and aggregating models. The second level is the road system, which clusters the vehicle clients and has two specific roles: 1) Cluster head node: responsible for optimizing training based on local data and aggregating models within the cluster; 2) Cluster member nodes: receiving models from the cluster head node, optimizing training based on their own data, and completing model upload. This system is mainly divided into two functional modules: a dynamic clustering module based on vehicle computing power and data similarity, and a federated dynamic sparse training module based on this module. The clustering module can be divided into three main functional modules: information collection, parameter exploration rate calculation, and Kmeans classification. The federated dynamic sparse training module can be divided into functional modules such as sparse network initialization, local optimization training, network structure reconstruction and update, and model aggregation.

[0148] Step 3: Federated dynamic sparse training experiment setup:

[0149] (1) Constructing traffic flow information: Two types of running vehicle information are set according to system requirements, namely kinematic information (speed, direction) and resource information (available memory, CPU information and data transmission rate). Specific values ​​that conform to the normal distribution are generated based on the random generation method.

[0150] (2) Model: In order to verify the effectiveness of the proposed mechanism, a typical deep convolutional neural network VGG16 was selected to perform the CIFAR10 classification task.

[0151] (3) Training settings: 200 communication rounds, reconstruction ratio μ = 0.3, ΔT = 25, T end =150, the SGD optimizer is used in local optimization, the learning rate η = 0.01, and the batch size of each data sample is 32.

[0152] (4) Comparison method:

[0153] 1) Dense: The original dense model, serving as the benchmark algorithm with the highest inference accuracy;

[0154] 2) Static: A randomly generated sparse network that meets the target sparsity is trained from scratch. The network structure is not updated during training. This serves as the baseline algorithm with the lowest inference accuracy.

[0155] 3) RigL: A dynamic sparse training method that currently works well in centralized scenarios, which guides the update of sparse network structures based on gradient information.

[0156] 4) FedDST: This is a currently effective federated learning dynamic sparse training algorithm that allows each client to periodically update the sparse network structure based on its own local data through RigL. The server then aggregates all local sparse network structures and reconstructs a new global sparse network based on the weight amplitude.

[0157] Step 4: Result analysis:

[0158] Generally speaking, higher sparsity reduces the computational and communication overhead of federated learning training, but also increases the difficulty of training. The goal of this system is to minimize the loss of model accuracy while maintaining the highest possible sparsity, thereby minimizing communication and computational resource overhead.

[0159] pass Figure 7 The comparison of experimental results in shows that the pre-training stage on the public dataset and the optimized initialization sparse network method provide a better starting point for the FedCET algorithm proposed in this invention, effectively reducing the sparse network exploration time and significantly improving the final performance. Figure 8 The algorithm comparison in Figure 1 shows a comprehensive comparison between the algorithm FedCET proposed in this system and other baseline algorithms. Thanks to the improvement in parameter exploration rate, the model accuracy obtained by FedCET training is significantly better than other baseline methods. Figure 9 The final accuracy comparison of various methods at different sparsity levels is presented. When the sparsity S = 0.9, FedCET's performance advantage is particularly prominent in the extremely sparse case, with its performance only 5% lower than the baseline algorithm, while the performance gap between other algorithms and the baseline algorithm reaches up to 16%. This result further demonstrates that increasing the parameter exploration rate can effectively enhance the representativeness and fitting ability of sparse networks.

[0160] In terms of overhead reduction at different sparsity levels, Figure 10The communication cost (Commu_cost) and FLOPs computation cost (FLOPs_cost) of FedCET at different sparsities are shown. Other sparse training methods are basically consistent with this method because the number of parameters retained by different methods is basically the same under the same target sparsity. The total FLOPs computation cost is the sum of all multiplication and addition operations. Thanks to the advantage of dynamic sparse training that trains a sparse network structure throughout the entire process, the resource cost of the method proposed in this system is significantly reduced as the sparsity of the model increases, such as Figure 10 As shown, compared with training an original dense model, when S = 0.95, FedCET only requires 0.192× the FLOPs computation overhead and 0.095× the communication overhead on the VGG16 model.

[0161] In summary, the federated dynamic sparse training system based on parallel expansion in the Internet of Vehicles scenario involved in the present invention has been designed and implemented, achieving the goal of maintaining low model performance loss on the basis of significantly reducing communication and computing overheads. Its effectiveness in improving the model accuracy of federated learning training in resource-constrained Internet of Vehicles scenarios has been verified.

Claims

1. A dynamic sparse training method for Internet of Vehicles federation based on cluster expansion, characterized by: Constructing a hierarchical federated learning training architecture for the Internet of Vehicles scenario: The hierarchical federated learning training architecture for the Internet of Vehicles scenario is a two-layer client-server structure, including edge servers and road systems; performing federated dynamic sparse training based on the hierarchical federated learning training architecture for the Internet of Vehicles scenario: Under the coordination and management of the edge server, the training node trains a sparse network structure based on local data, and periodically updates the sparse network structure according to predefined reconstruction update criteria; finally, a global network structure that meets the target sparsity is aggregated and reconstructed at the edge server.

2. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 1 is characterized in that: The edge server serves as a central node and is used to: a) Pre-training of neural network models: Pre-training is performed on public datasets using local computing resources to obtain an initialized sparse network structure. This structure is then sparsified based on weight standards and sparsity requirements for subsequent formal training. b) Vehicle client clustering: The kinematic information and vehicle terminal resource information of vehicles entering the area covered by the edge server are collected and quantified. The binary division is continued until the result satisfies the actual exploration rate between vehicle clusters reaching 100%, and the client cluster results are obtained. c) Federated training process management: responsible for issuing sparse network structures and aggregating client sparse network structures to form a global sparse network structure, and reconstructing the aggregated model according to the predefined target sparsity.

3. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 2 is characterized in that: The process of obtaining the initialized sparse network model is as follows: the edge server pre-trains the neural network using a public data set, and obtains a sparse network structure that meets the target sparsity as the initialized sparse network structure through a predefined weight amplitude standard; The layer importance factor γ is introduced in the pre-training stage to dynamically determine the sparsity of different network layers in the sparse network structure. The layer importance factor γ of the lth layer is l The calculation formula is as follows: Among them, m l (1) is the number of non-zero parameters in the lth layer, m l (0) is the number of parameters with zero weights in layer l.

4. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 3 is characterized in that: The vehicle client cluster division is based on resource and model similarity; Set the initial target number of clusters to 2, and recursively divide each cluster. After each division, calculate the actual exploration rate of each cluster in the current division result. When the total actual exploration rate of the cluster is 100%, the cluster division is terminated, otherwise continue cluster division. 1) After receiving the content uploaded by the vehicle node, the edge server normalizes the vehicle terminal resource information to obtain the vehicle resource vector The calculation formula is as follows: where f i is the CPU main frequency of vehicle client i terminal, r i is the data transmission rate of vehicle client i terminal, a i The available memory size of the vehicle client i terminal; 2) The server quantifies the model similarity of the currently trained model w of each vehicle node received, so as to facilitate the subsequent similarity measurement of all features. The calculation formula is as follows: where w i is the current local sparse network model of vehicle i, w j is the current local sparse network model of vehicle j; 3) Comprehensively consider the vehicle's kinematic information, vehicle terminal resource information, and the similarity of the current training model to improve the stability of the clustering results and calculate the similarity function. The calculation formula is as follows: Among them, α, β, and γ are hyperparameters used to adjust the impact of speed, resources, and model similarity on quantifying the similarity between vehicles; 4) Expansion Exploration Rate: The sparse network model received by the vehicle cluster from the central node is defined as the base model. Each vehicle cluster first calculates the total number of resources of the current vehicle cluster and calculates the target expansion exploration rate of the current vehicle cluster based on this. The average resource information of the current cluster is calculated, and the size of the trainable cluster local sparse network model is calculated based on this average. The expansion exploration rate that can be achieved compared with the base model is then calculated and the actual expansion exploration rate is compared with the target expansion exploration rate. When the actual exploration rate is not less than the target exploration rate, the cluster stops dividing. The cluster expansion exploration rate calculation formula is as follows: The operator ∪ represents merging the exploration results of all clusters, M z represents the local sparse network model trained by cluster z, |W| counts the total number of parameters of the original dense network, and ∥·∥0 represents the l0 norm, which is used to count the total number of all explored parameters; After the cluster division is completed, the average model structure of the cluster is compared with the global model structure based on formula (3) for similarity, and sorted in descending order. First, the basic model θ is calculated. base In the dense network model θ dense The complement of the above can be used to obtain the explorable model θ ava Then, based on the actual expansion exploration rate of each cluster and the average sparse neural network model of the current vehicle cluster, additional parameter connections are randomly generated for each vehicle cluster in sequence to obtain the expansion part θ extend The model structure θ used in the next round of training i , then update the explorable model θ ava The calculation formulas are as follows: i ava =θ dense \i extend (6) i i =θ base ∪θ extend (7) i ava =θ ava \i extend (8) where θ dense is the complete dense network model, given before training; θ base The edge server initializes the sparse network model; θ i is the sparse network model structure of cluster i.

5. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 4 is characterized in that: The target expansion exploration rate is calculated as follows: When performing dynamic cluster division, before each cluster division using the Kmeans algorithm, the target exploration rate of the current cluster must be calculated. The target expansion exploration rate is calculated by measuring the ratio of the total computing resources owned by all members of the current vehicle cluster to the total computing resources in all vehicle clusters, and multiplying this ratio by 100%. The calculation formula is as follows: Among them S i is the computing resources of the current cluster, is the total computing resources of all clusters.

6. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 4 is characterized in that: The actual expansion exploration rate is calculated as follows: When performing dynamic cluster division, in addition to calculating the target expansion exploration rate, the actual cluster expansion exploration rate also needs to be calculated; the normalized resource vector interval in the cluster member is calculated by formula (2) and recorded as The specific calculation formula is as follows: where R * It is a predefined parameter used to define the exploration rate that can be achieved per unit resource vector.

7. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 1, characterized in that: The road system is composed of vehicle clusters, and vehicle nodes are divided into cluster head nodes and cluster member nodes according to their different roles in the vehicle cluster structure; a) Cluster member nodes: responsible for performing federated training based on locally collected data and uploading the trained sparse network structure to the cluster head node; b) Cluster head node: It is not only responsible for local training, but also for aggregating models from cluster member nodes and uploading the aggregated models to the edge server to assist in updating and reconstructing the global model.

8. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 1, characterized in that: The federated dynamic sparse training is a parallel and scalable federated dynamic sparse training mechanism FedCET. The parallel and scalable federated dynamic sparse training mechanism FedCET divides a large number of vehicle clients into several clusters, allowing different clusters to explore different neural network connections in parallel outside the basic sparse network structure. Through collaboration between clusters, it completes the complete exploration of all connections of the dense network structure from a spatial perspective, covers a sufficient number of connections during the training process, and dynamically optimizes and updates the structure of the global sparse network on this basis, thereby significantly improving the generalization ability of the global sparse model in high-sparse scenarios and achieving higher model accuracy.

9. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 8, characterized in that: When performing federated dynamic sparse training, the trained sparse network structure is represented by transmitting the sparse network mask; The method of generating a sparse network mask based on the local sparse network structure trained by the node is as follows: let W be the sparse neural network structure parameter matrix, the mask is a binary matrix M with the same shape as it, and each element m in M ​​is a,b The corresponding weight w in W a,b The specific rules are: if w a,b =0, then m a,b =0; if w a,b ≠0, then m a,b =1; After generating the sparse network masks, the vehicle nodes in each cluster use these masks for local sparse training. The specific method is as follows: the vehicle node i of cluster z uses the corresponding sparse network mask M z Perform local sparse training; In the tth communication round, vehicle node i uses local data X i , optimized with learning rate η, the calculation formula is as follows: In order to further improve the efficiency of parameter exploration through training and optimize the sparse network structure, the parameter expansion exploration phase is introduced. The specific method is: every ΔT rounds, the sparse network structure is gradually fine-tuned through parallel parameter exploration between clusters. The goal is to obtain the optimal global sparse network structure M through iterative updates while meeting the sparsity requirements. * ; The edge server randomly explores new parameters of proportion f for each cluster to form a new sparse network structure; for the zth cluster, its sparse network structure is recorded as M z ,by merging the sparse network structure of all clusters at the edge server; The structure of the sparse network is gradually fixed by the cosine decay strategy, and the value of the new exploration parameter ratio f is initialized to μ. end The communication round decays to 0; in the tth communication round, the calculation formula of f is as follows: When f = 0, the parameter exploration stops, and the global sparse network structure obtained at this time is the optimal sparse network structure M * ,In the subsequent training, the global sparse network structure remains fixed until the training stops after convergence; During aggregation, the sparse network structure trained for each vehicle cluster is personalized while ensuring low differences between clusters. Each sparse network structure is logically considered to be two parts: 1) Public infrastructure base , these parameters are shared by all clusters; 2) Private part w self , corresponding to the newly explored parameters of each cluster, which are private to the cluster; Suppose there are n z clients, the model of cluster z after aggregation is: Among them, |D i | is the amount of data owned by vehicle node i, and They represent the basic public and private parameters of the i-th vehicle node, respectively. All clients participate in the aggregation of the public basic part, while the private part parameters are only aggregated and averaged among the clients that retain the current parameters. After the aggregation is completed, all vehicle nodes will receive the same public basic part parameters, while the private part parameters are independent between vehicle clusters. After the central node completes receiving and aggregating the sparse network structures uploaded by all vehicle nodes, it performs sparse network reconstruction to meet the target sparsity requirement. The specific method is as follows: every fixed round ΔT, the parameter sensitivity score g defined by formula (14) is selected from the currently explored sparse network structure. j Select the most promising sparse network weights, comprehensively consider the influence of gradient information and weight size, and reconstruct a new global sparse network structure M with a sparsity of S+(1-S)f G , and serves as the starting point for the next round of parallel expansion sparse training; until the preset T end When f = 0 in the round, parameter exploration stops, and the sparse network structure of all vehicle terminals is exactly the same, with no private parts, marking the end of sparse network parameter exploration. After that, the network structure is used for training until convergence stops. Where L is the loss function; Communication cost calculation: The communication cost of training a sparse network is positively correlated with the model size. When the client and server communicate with each other, only the retained parameters are transmitted. The sparse mask representing the network structure is only transmitted when it is updated. Assuming that the original size of the model is L and the target sparsity is S, the total communication cost of each communication round for a single client is in is the additional overhead of transmitting the updated mask structure.

Images