Internet of vehicles federal dynamic sparse training method based on cluster expansion

Abstract

The invention belongs to the field of Internet of Vehicles network communication, and discloses an Internet of Vehicles federated learning dynamic sparse training method based on cluster expansion. Constructing a hierarchical federated learning training architecture under the scene of the Internet of Vehicles, wherein the hierarchical federated learning training architecture under the scene of the Internet of Vehicles is of a client-server two-layer structure and comprises an edge server and a road system; performing federated dynamic sparse training based on a hierarchical federated learning training architecture in an Internet of Vehicles scene: training a sparse network structure based on local data by a training node under coordination and management of an edge server, and periodically updating the sparse network structure according to a predefined reconstruction updating standard; and finally, aggregating and reconstructing a global network structure meeting the target sparseness at the edge server.

Description

Technical Field

[0001] The present invention relates to the field of vehicle networking network communication technology, and in particular to a vehicle networking federated learning dynamic sparse training method based on cluster expansion. Background Art

[0002] As a complex intelligent network that integrates multiple elements such as vehicles, road side units (RSU), base stations and personnel, the frequent communication process between nodes in the Internet of Vehicles brings huge bandwidth consumption during federated learning training. In the Internet of Vehicles scenario, the neural network models commonly used in tasks such as target detection and recognition are becoming more and more complex, further exacerbating the pressure on communication and computing resources. At present, some technologies have attempted to reduce the communication overhead in the process of federated learning training. For example, some studies have used model quantization technology to reduce the accuracy of transmission parameters, thereby reducing communication overhead. However, this method has limited compression effect and may introduce a large loss of model accuracy. In addition, some methods use the knowledge distillation mechanism to obtain a student model with good performance and low computational complexity, which reduces the consumption of communication resources. However, this method also has disadvantages that cannot be ignored: additional computing resources and a large number of high-quality data samples are required to assist the training process, so there are great challenges in practical applications.

[0003] Therefore, the communication optimization mechanism based on dynamic sparse training has gradually become a research hotspot. By sparsely configuring the network during the training process, the communication and computing overhead in federated learning training can be significantly reduced. However, the existing dynamic sparse training methods still face challenges in the Internet of Vehicles scenario, such as the mobility of the client and the unreliability of the communication link. At the same time, when the existing methods achieve a higher communication overhead optimization effect, they will bring about a large loss of model accuracy. Therefore, how to implement an efficient federated dynamic sparse training mechanism in the Internet of Vehicles scenario is still a problem that needs to be solved urgently. The present invention designs a method for federated dynamic sparse training of the Internet of Vehicles based on cluster expansion, which quantifies the computing power and data distribution of the vehicle client, dynamically divides the clusters, and performs expansion training on the basic global sparse model through the cluster, thereby improving the accuracy of the trained model and making it more adaptable to the unique network environment of the Internet of Vehicles. Summary of the invention

[0004] In view of the deficiencies in the prior art, the main purpose of the present invention is to provide a cluster-expanded federated dynamic sparse training method for Internet of Vehicles scenarios, which is a dynamic cluster partitioning algorithm based on the degree of system and data heterogeneity.

[0005] Its main technical solutions are as follows: a method for dynamic sparse training of the Internet of Vehicles federation based on cluster expansion, constructing a hierarchical federated learning training architecture in the Internet of Vehicles scenario: the hierarchical federated learning training architecture in the Internet of Vehicles scenario is a two-layer structure of client and server, including edge servers and road systems; federal dynamic sparse training is performed based on the hierarchical federated learning training architecture in the Internet of Vehicles scenario: under the coordination and management of the edge server, the training node trains the sparse network structure based on local data, and periodically updates the sparse network structure according to predefined reconstruction and update standards; finally, a global network structure that meets the target sparsity is aggregated and reconstructed at the edge server.

[0006] The edge server serves as a central node and is used to:

[0007] a) Pre-training of neural network models: Pre-training is performed on public datasets based on local computing resources to obtain an initialized sparse network structure, which is then sparsified based on weight standards and sparsity requirements and applied to subsequent formal training;

[0008] b) Vehicle client cluster division: collect and quantify the kinematic information and vehicle terminal resource information of vehicles entering the area covered by the edge server, and continue to divide the results into two parts until the actual exploration rate between vehicle clusters reaches 100% to obtain the client cluster results;

[0009] c) Federated training process management: responsible for issuing sparse network structures and aggregating client sparse network structures to form a global sparse network structure, and reconstructing the aggregated model according to the predefined target sparsity.

[0010] The process of obtaining the initialized sparse network model is as follows: the edge server pre-trains the neural network using a public data set, and obtains a sparse network structure that meets the target sparsity through a pre-defined weight amplitude standard as the initialized sparse network structure; in the pre-training stage, the layer importance factor γ is introduced to dynamically determine the sparsity of different network layers in the sparse network structure, and the layer importance factor γ of the lth layer is l The calculation formula is as follows:

[0011]

[0012] Among them, m l (1) is the number of non-zero parameters in the lth layer, m l (0) is the number of parameters with zero weights in layer l.

[0013] The vehicle client cluster division is based on resource and model similarity; the initial target number of clusters is set to 2, and each cluster is recursively divided. After each division, the actual exploration rate of each cluster of the current division result is calculated. When the total actual exploration rate of the cluster is 100%, the cluster division is terminated, otherwise the cluster division is continued;

[0014] 1) After receiving the content uploaded by the vehicle node, the edge server normalizes the vehicle terminal resource information to obtain the vehicle resource vector The calculation formula is as follows:

[0015]

[0016] where f i is the CPU main frequency of the vehicle client i terminal, r i is the data transmission rate of vehicle client i terminal, a i is the available memory size of the vehicle client i terminal;

[0017] 2) The server quantifies the model similarity of the current training model w of each vehicle node received, so as to measure the similarity of all features in the future. The calculation formula is as follows:

[0018]

[0019] where w i is the current local sparse network model of vehicle i and w j is the current local sparse network model of vehicle j;

[0020] 3) Comprehensively consider the vehicle's kinematic information, vehicle terminal resource information and the similarity of the current training model to improve the stability of the clustering results. The similarity function calculation formula is as follows:

[0021]

[0022] Among them, α, β, and γ are hyperparameters used to adjust the impact of speed, resources, and model similarity on quantifying the similarity between vehicles;

[0023] 4) Expansion exploration rate: The sparse network model received by the vehicle cluster from the central node is defined as the basic model. Each vehicle cluster first calculates the total number of resources of the current vehicle cluster, and based on this, calculates the target expansion exploration rate of the current vehicle cluster; calculates the average resource information of the current cluster, and uses the average value to calculate the size of the trainable cluster local sparse network model, and then calculates the achievable expansion exploration rate compared with the basic model, and compares the actual expansion exploration rate with the target expansion exploration rate. When the actual exploration rate is not less than the target exploration rate, the cluster stops dividing; the cluster expansion exploration rate calculation formula is as follows:

[0024]

[0025] The operator ∪ represents the merging of the exploration results of all clusters, M z represents the local sparse network model trained by cluster z, |W| counts the total number of parameters of the original dense network, and ∥·∥0 represents the l0 norm, which is used to count the total number of all explored parameters;

[0026] 5) Target expansion exploration rate: When performing dynamic cluster division, before each cluster division using the Kmeans algorithm, the target exploration rate of the current cluster needs to be calculated first; the target expansion exploration rate is obtained by measuring the ratio between the sum of computing resources owned by all members in the current vehicle cluster and the sum of computing resources in all vehicle clusters, and multiplying the ratio by 100%. The calculation formula is as follows:

[0027]

[0028] Where S i is the computing resources of the current cluster, is the total computing resources of all clusters.

[0029] 6) Actual expansion exploration rate: When performing dynamic cluster division, in addition to calculating the target expansion exploration rate, it is also necessary to calculate the actual expansion exploration rate of the cluster. The normalized resource vector interval in the cluster member is calculated by formula (2) and recorded as The specific calculation formula is as follows:

[0030]

[0031] Where R * It is a predefined parameter used to define the exploration rate that can be achieved by a unit resource vector;

[0032] After the cluster division is completed, the average model structure of the cluster is compared with the global model structure based on formula (3) for similarity, and sorted in descending order. First, the basic model θ is calculated. base In the dense network model θ dense The complement of the above can be used to obtain the explorable model θ ava Then, based on the actual expansion exploration rate of each cluster and the average sparse neural network model of the current vehicle cluster, some additional parameter connections are randomly generated for each vehicle cluster in sequence to obtain the expansion part θ extend The model structure θ used in the next round of training i , and then update the explorable model θ ava ; The calculation formulas are as follows:

[0033] θ ava =θdense \θ extend (8)

[0034] θ i =θ base ∪θ extend (9)

[0035] θ ava =θ ava \θ extend (10)

[0036] where θ dense is the complete dense network model, given before training; θ base is the edge server initialization sparse network model; θ i is the sparse network model structure of cluster i.

[0037] The road system is composed of vehicle clusters, and vehicle nodes are divided into cluster head nodes and cluster member nodes according to different roles in the vehicle cluster structure;

[0038] a) Cluster member node: responsible for performing federated training based on locally collected data and uploading the trained sparse network structure to the cluster head node;

[0039] b) Cluster head node: It is not only responsible for local training, but also for aggregating models from cluster member nodes and uploading the aggregated models to the edge server to assist in updating and reconstructing the global model.

[0040] The federated dynamic sparse training is a parallel-expandable federated dynamic sparse training mechanism FedCET; the parallel-expandable federated dynamic sparse training mechanism FedCET divides a large number of vehicle clients into several clusters, allowing different clusters to explore different neural network connections in parallel outside the basic sparse network structure, and completes the complete exploration of all connections of the dense network structure from a spatial perspective through the collaborative cooperation between clusters. In the process of training, enough connections are covered, and on this basis, the structure of the global sparse network is dynamically optimized and updated, thereby greatly improving the generalization ability of the global sparse model in high-sparse scenarios and achieving higher model accuracy;

[0041] When performing federated dynamic sparse training, the trained sparse network structure is represented by transmitting a sparse network mask. The method for generating a sparse network mask based on the local sparse network structure trained by the node is as follows: Let W be the sparse neural network structure parameter matrix, the mask is a binary matrix M with the same shape, and each element m in M ​​is a,b By the corresponding weight w in W a,b The specific rules are: if w a,b =0, then m a,b=0; if w a,b ≠0, then m a,b =1;

[0042] After generating the sparse network masks, the vehicle nodes in each cluster use these masks for local sparse training. The specific method is as follows: the vehicle node i of cluster z uses the corresponding sparse network mask M z Perform local sparse training; in the tth communication round, vehicle node i uses local data X i , optimized with learning rate η, the calculation formula is as follows:

[0043]

[0044] In order to further improve the efficiency of parameter exploration through training and optimize the sparse network structure, the parameter expansion exploration phase is introduced. The specific method is: every ΔT rounds, the sparse network structure is gradually fine-tuned through parallel parameter exploration between clusters. The goal is to obtain the optimal global sparse network structure M through iterative updates while meeting the sparsity requirements. * ; The edge server randomly explores new parameters of proportion f for each cluster to form a new sparse network structure; for the zth cluster, its sparse network structure is recorded as M z , by merging the sparse network structure of all clusters at the edge server; gradually fixing the structure of the sparse network through the cosine decay strategy, the value of the new exploration parameter ratio f is initialized to μ, and at T end The communication round decays to 0; in the tth communication round, the calculation formula of f is as follows:

[0045]

[0046] When f = 0, parameter exploration stops, and the global sparse network structure obtained at this time is the optimal sparse network structure M * ,In the subsequent training, the global sparse network structure remains fixed until the training stops after convergence;

[0047] During aggregation, the sparse network structure trained for each vehicle cluster is personalized while the differences between clusters are kept low. Each sparse network structure is logically considered as two parts:

[0048] 1) Public basic part w base , these parameters are shared by all clusters;

[0049] 2) Private part w self , corresponding to the newly explored parameters of each cluster, which are private to the cluster;

[0050] Suppose there are n z clients, the model of cluster z after aggregation is:

[0051]

[0052] Among them, |D i | is the amount of data owned by vehicle node i, and They represent the parameters of the basic public part and private part of the i-th vehicle node respectively. All clients participate in the aggregation of the public basic part, while the private part parameters are aggregated and averaged only among the clients that retain the current parameters. After the aggregation is completed, all vehicle nodes will receive the same public basic part parameters, while the private part parameters are independent between vehicle clusters.

[0053] After the central node receives and aggregates the sparse network structures uploaded by all vehicle nodes, it reconstructs the sparse network to meet the target sparsity requirement. The specific method is as follows: every fixed round ΔT, the parameter sensitivity score g defined by formula (14) is selected from the currently explored sparse network structure. j Select the most promising sparse network weights, comprehensively consider the influence of gradient information and weight size, and reconstruct a new global sparse network structure M with a sparsity of S+(1-S)f G , and serves as the starting point for the next round of parallel expansion sparse training; until the preset T end When f=0 in the round, parameter exploration stops, and the sparse network structure of all vehicle terminals is exactly the same, and there is no private part anymore, which marks the end of the sparse network parameter exploration. After that, the network structure is used for training until convergence stops;

[0054]

[0055] Where L is the loss function;

[0056] Communication cost calculation: The communication cost of training a sparse network is positively correlated with the model size. When the client and server communicate with each other, only the retained parameters are transmitted. The sparse mask representing the network structure is only transmitted when it is updated. Assuming that the original size of the model is L and the target sparsity is S, the total communication cost of each communication round for a single client is

[0057]

[0058] in is the additional overhead of transmitting the updated mask structure.

[0059] Beneficial effects of the present invention: The present invention proposes a method for dynamic sparse training of the Internet of Vehicles federation based on cluster expansion, which aims to improve the training efficiency and accuracy of the model by optimizing the use of communication and computing resources in the complex network environment of the Internet of Vehicles. Through the dynamic sparse training mechanism, the present invention significantly reduces the data transmission requirements between networks and reduces bandwidth consumption. Especially in the Internet of Vehicles environment where the cost of data transmission is high, this reduction can effectively save resources. At the same time, the use of a parallel expansion training strategy enables different vehicle clusters to explore different connections of the neural network in parallel. This method not only optimizes the generalization ability of the global model, but also improves the model accuracy, especially when dealing with highly sparse network structures.

[0060] In addition, the present invention uses the edge server to intelligently analyze the computing power and data distribution of the vehicle client and dynamically divide the clusters, ensuring the optimal allocation and utilization of resources, maximizing the actual exploration rate of each cluster, and thus improving the efficiency and effect of the overall training. Through this centralized management and intelligent allocation strategy, the resources of the vehicle client are used more efficiently, while reducing the overall training time and resource consumption.

[0061] After comparative experiments with the baseline algorithm, the algorithm proposed in this invention shows significant advantages in communication overhead, computational overhead, accuracy of the global model and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of a real-world Internet of Vehicles scenario;

[0063] Figure 2 This is a schematic diagram of the hierarchical IoV federated learning training architecture;

[0064] Figure 3 This is a schematic diagram of the dynamic sparse training process;

[0065] Figure 4 This is a schematic diagram of the process of dynamic sparse training of the Internet of Vehicles federation based on parallel expansion;

[0066] Figure 5 This is a schematic diagram of the model aggregation strategy;

[0067] Figure 6 To verify the relationship between parameter exploration rate and model performance

[0068] Figure 7 Ablation experiment for the pre-training stage;

[0069] Figure 8 This is a performance comparison chart of the federated dynamic sparse training algorithm;

[0070] Fig. 9 This is a comparison chart of algorithm performance under different target sparsity;

[0071] Fig.10 Schematic diagram of the communication overhead and computational overhead of the FedCET algorithm under different sparsity. DETAILED DESCRIPTION

[0072] Construct a hierarchical federated learning training architecture for the Internet of Vehicles scenario;

[0073] (1) Client-server two-layer structure: Considering the actual situation of the Internet of Vehicles, the infrastructure such as vehicle nodes and edge servers deployed by roadside units (RSUs) or base stations are divided into different layers. The overall training framework consists of two parts: edge servers and road systems.

[0074] 1) Edge server (usually deployed at the base station): As the central node, it is responsible for the management of the entire federated learning training process. Its main functions include:

[0075] a) Pre-training of neural network models: Pre-training is performed on public datasets based on local computing resources to obtain an initialized sparse network structure with good performance, which is then sparsely processed based on weight standards and sparsity requirements for subsequent formal training.

[0076] b) Vehicle client clustering: The kinematic information (position, direction, speed) and vehicle terminal resource information (CPU frequency, available memory, data transmission rate) of vehicles entering the area covered by the edge server are collected and then quantified. The Kmeans algorithm is used to continuously perform binary division. When the division result satisfies the structural exploration rate between vehicle clusters reaching 100% (the trained sparse network structure is the same as the original dense network structure), the clustering is stopped to obtain the client clustering result.

[0077] c) Federated training process management: responsible for issuing sparse network structures and aggregating client sparse network models, and reconstructing the aggregated models according to the predefined target sparsity.

[0078] 2) Road system: It consists of vehicle clusters, where vehicle nodes are divided into cluster head nodes and cluster member nodes according to their different roles in the cluster structure.

[0079] a) Cluster member node: responsible for performing federated training based on locally collected data and uploading the trained sparse network structure to the cluster head node.

[0080] b) Cluster head node: It is not only responsible for local training, but also for aggregating models from cluster member nodes and uploading the aggregated models to the edge server to assist in updating and reconstructing the global model.

[0081] (2) Federated dynamic sparse training mechanism: Under the coordination and management of the central node, the training nodes train the sparse network structure based on local data and periodically update the network structure according to the predefined reconstruction update criteria. Finally, a global network structure that meets the target sparsity is aggregated and reconstructed at the central node.

[0082] Initialize the sparse network structure;

[0083] (1) Pre-training stage: Different from the traditional research method that uses random methods to generate sparse network structures, the present invention relies on the powerful computing power of the edge server, uses public data sets to pre-train the neural network, and obtains a sparse network structure that meets the target sparsity through the weight amplitude standard as the initial sparse network structure.

[0084] (2) Calculating layer sparsity: Different from the traditional mechanism that fixes the sparsity of each layer in the neural network, the present invention introduces a layer importance factor γ in the pre-training stage to dynamically determine the sparsity of each layer. The layer importance factor γ of the lth layer is l The calculation formula is as follows:

[0085]

[0086] Where m l (1) is the number of non-zero parameters in the lth layer, m l (0) is the number of parameters with zero weights in layer l.

[0087] A vehicle client clustering mechanism based on resource and model similarity;

[0088] (1) Vehicle information processing: When a vehicle enters the coverage area of ​​the edge server, the vehicle’s driving information such as speed s and direction, resource information such as CPU main frequency f, data transmission rate r, available memory size a, and the current training model w are uploaded to the central node.

[0089] 1) After receiving the content uploaded by the vehicle, the server normalizes the resource information. The calculation formula is as follows:

[0090]

[0091] Get the vehicle's resource vector

[0092] 2) The server quantifies the similarity of the current training model w of the received vehicle, and the calculation formula is as follows:

[0093]

[0094] where w i and w jare the current local sparse neural network model structures of vehicle i and vehicle j respectively.

[0095] (2) Dynamic clustering algorithm based on Kmeans method: The initial target number of clusters is set to 2, and each cluster is recursively divided. After each division, the actual exploration rate of each cluster of the current division result is calculated. When the total actual exploration rate of the cluster is 100%, the cluster division is terminated, otherwise the cluster division continues.

[0096] 1) Similarity function: Comprehensively consider the kinematic information, computing resource conditions and model similarity of the vehicle client to improve the stability of cluster division results and reduce the impact of challenges such as resource heterogeneity and data heterogeneity on federated learning training. The calculation formula is as follows:

[0097]

[0098] Among them, α, β, and γ are hyperparameters used to adjust the impact of speed, resources, and model similarity on quantifying the similarity between vehicles.

[0099] 2) Expansion exploration rate: The sparse network model received by the cluster from the central server is defined as the basic model. Each cluster first calculates the total number of resources of the current cluster, and then calculates the target expansion exploration rate of the current cluster based on this. Next, the average resource information of the current cluster is calculated, and the maximum model size that can be trained is calculated with this average value. Then, the expansion exploration rate that can be achieved compared with the basic model is calculated, and the actual expansion exploration rate is compared with the target expansion exploration rate. When the actual expansion exploration rate is not less than the target expansion exploration rate, the cluster stops dividing. The cluster expansion exploration rate calculation formula is as follows:

[0100]

[0101] The operator ∪ represents the merging of the exploration results of all clusters, |W| counts the total number of parameters of the original dense network, and ∥·∥0 represents the l0 norm, which is used to count the total number of all explored parameters. The merge operation only represents a logical merge, which is used to represent the set of parameters explored by all clusters.

[0102] according to Figure 6 The experimental results show that the value of R is positively correlated with the performance of the sparse network. A larger parameter exploration space and higher parameter exploration efficiency can more quickly and reliably find a sparse network structure with better performance, thereby significantly improving the performance of dynamic sparse training.

[0103] 3) Target exploration rate: When performing dynamic cluster division, before each cluster division using the Kmeans algorithm, the target exploration rate of the current cluster needs to be calculated. This calculation is achieved by measuring the ratio between the sum of the computing resources owned by all members in the current cluster and the sum of the computing resources in all clusters, and then multiplying this ratio by 100% total exploration rate to get the target exploration rate. The calculation formula is as follows:

[0104]

[0105] Where S i is the computing resources of the current cluster, is the total computing resources of all clusters.

[0106] 4) Actual exploration rate: When performing dynamic cluster division, in addition to calculating the target exploration rate, it is also necessary to calculate the actual exploration rate of the cluster. The normalized resource vector interval in the cluster member is calculated by formula (2) and recorded as The specific calculation formula is as follows:

[0107]

[0108] Where R * It is a predefined parameter that defines the exploration rate that can be achieved per resource vector.

[0109] (3) Neural network model expansion mechanism: After the cluster division is completed, the average model structure of the cluster is compared with the global model structure based on formula (3) for similarity, and sorted in descending order. First, the explorable model θ is obtained. ava ,Then, based on the expansion exploration rate of each cluster and the current cluster model structure, additional parameter connections are randomly generated for each cluster in order to obtain the expansion part θ extend The model structure θ used in the next round of training i , and then update the explorable model θ ava The calculation formulas are as follows:

[0110] θ ava =θ dense \θ extend

[0111] θ i =θ base ∪θ extend

[0112] θ ava =θ ava \θ extend

[0113] where θ dense is the complete dense network model structure, θ baseis the initial sparse network structure on the server side, θ i is the sparse model structure of cluster i.

[0114] A federated dynamic sparse training mechanism based on parallel expansion;

[0115] Based on the parallel expansion of sparse training methods, a new fully sparse federated learning training framework: FedCET is proposed. While reducing the client computing overhead and communication overhead, the performance of the global sparse model is guaranteed not to decrease. Specifically, FedCET divides a large number of vehicle clients into several clusters, allowing different clusters to explore different neural network connections in parallel outside the basic sparse network structure. Through the collaborative cooperation between clusters, it completes the complete exploration of all connections of the dense network structure from a spatial perspective, covers enough connections during the training process, and dynamically optimizes and updates the structure of the global sparse network on this basis, thereby greatly improving the generalization ability of the global sparse model in high-sparse scenarios and achieving higher model accuracy.

[0116] (1) Client-side local sparse training algorithm: Vehicle client i of cluster j uses the corresponding network structure mask M j Perform local sparse training. In the tth communication round, client i uses local data X i , optimized with learning rate η, the calculation formula is as follows:

[0117]

[0118] (2) Parameter expansion exploration: Every ΔT rounds, the sparse network structure is gradually fine-tuned through parallel parameter exploration between clusters. The goal is to obtain the optimal global sparse network structure M through iterative updates while meeting the sparsity requirements. * The server randomly explores new parameters of proportion f for each cluster to form a new sparse network structure. For the jth cluster, its sparse network structure is recorded as M j By merging the network structures of all clusters on the server side, we can logically achieve full coverage of network structure parameters. FedCET uses the cosine decay strategy to gradually fix the structure of the sparse network, and initializes the value of the new exploration parameter ratio f to μ. end The communication round decays to 0. In the tth communication round, the calculation formula of f is as follows:

[0119]

[0120] When f = 0, parameter exploration stops, and the global sparse network structure obtained at this time is the optimal sparse network structure M * , the global sparse network structure remains fixed in subsequent training until the training stops after convergence.

[0121] (3) Model aggregation: Since the sparse network structures trained by different clusters are different, the aggregation brings the challenge of model heterogeneity. In the FedCET process, a new aggregation method is adopted to ensure the personalization of the network structure trained by each cluster during aggregation, while ensuring that the differences between clusters are low. Each network structure is logically regarded as two parts:

[0122] 1) Public basic part w base , these parameters are shared by all clusters.

[0123] 2) Private part w self , corresponding to the newly explored parameters of each cluster, are private to the cluster.

[0124] The reason why the model structure is divided into two parts is that, on the one hand, the public part provides the possibility of aggregating data from all clusters and training a globally representative model, and on the other hand, the private part is used to achieve parameter expansion exploration and thus optimize the structure of the global sparse network. Assume that there are n j clients, the model of cluster j after aggregation is:

[0125]

[0126] Where |D i | is the amount of data owned by client i, and They represent the parameters of the basic public part and private part of the i-th client respectively. All clients participate in the aggregation of the public basic part, while the private part parameters are aggregated and averaged only among the clients that retain the current parameters. After the aggregation is completed, all clients will receive the same public basic part parameters, while the private part parameters are independent among clusters.

[0127] In addition, the proportion f of the newly explored parameters in point (2) also determines the degree of model heterogeneity among clusters. An excessively large f will indirectly affect the stability of sparse training and aggregation. Therefore, the common basic part occupies most of the total training parameters and is shared among clusters as a common structure, which can be fully generalized to all clusters for training.

[0128] (4) Sparse network reconstruction: Every fixed round ΔT, the parameter sensitivity score g defined by formula (14) is selected from the currently explored network structure j Select the most promising parameter connections, comprehensively consider the influence of gradient information and weight size, and reconstruct a new global sparse network structure M with a sparsity of S+(1-S)f G , and serves as the starting point for the next round of parallel expansion sparse training. endWhen f=0 in the round, parameter exploration stops, the sparse network structure of all terminals is exactly the same, and there is no private part anymore, which marks the end of the sparse network parameter exploration. After that, the network structure is used for training until convergence stops.

[0129]

[0130] Where L is the loss function.

[0131] (5) Communication overhead calculation: The communication overhead of training a sparse network is positively correlated with the model size. When the client and server communicate with each other, only the retained parameters are transmitted. The sparse mask representing the network structure is only transmitted when it is updated. Assuming that the original size of the model is L and the target sparsity is S, the total communication overhead of each communication round for a single client is

[0132]

[0133] in is the additional overhead of transmitting the updated mask structure.

[0134] Algorithm 1 below is the pseudo code of the federated dynamic sparse training algorithm based on cluster expansion.

[0135]

[0136]

[0137] Algorithm 2 below is the pseudo code of the vehicle client clustering algorithm based on resource and model similarity.

[0138]

[0139] Output: Client cluster division results And the corresponding sparse model

[0140]

[0141]

[0142] The hardware and software environment of this implementation plan is shown in Table 1.

[0143] Table 1 Software and hardware configuration environment

[0144] name Configuration operating system Ubuntu 18.04 CPU i5-12400F Memory 16G

[0145] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings.

[0146] Step 1: Identify the application scenario: Figure 1As shown in the figure, for the universal Internet of Vehicles scenario, the vehicle collects a large amount of data through its own sensors and has a certain computing power. Due to the serious mismatch between the collection speed and the storage rate of the fuselage or the data transmission rate uploaded to the cloud, under the coordinated management of the edge server deployed near the base station, the vehicle performs federated learning training on the data source side, which can not only save the process of transmitting to the remote cloud, but also protect the privacy of the vehicle's collected data and adapt to the scenario requirements.

[0147] Step 2: System architecture: Figure 2 As shown in the figure, this system is mainly divided into two levels. The edge server deployed at the base station serves as the central node, coordinating and managing the vehicles in its coverage area for federated learning training, and is responsible for receiving and aggregating the model. The second level is the road system, which clusters the vehicle clients. Specifically, there are two roles: 1) Cluster head node: responsible for optimizing training based on local data and aggregating the model within the cluster; 2) Cluster member node: receiving the model from the cluster head node, optimizing training based on its own data and completing model upload. This system is mainly divided into two functional modules, namely the dynamic clustering module based on vehicle computing power and data similarity and the federated dynamic sparse training module based on this module. The clustering module can be divided into three main functional modules: information collection, parameter exploration rate calculation and Kmeans classification; the federated dynamic sparse training module can be divided into: sparse network initialization, local optimization training, network structure reconstruction and update, model aggregation and other functional modules.

[0148] Step 3: Federated dynamic sparse training experiment settings:

[0149] (1) Constructing vehicle flow information: Two types of running vehicle information are set according to system requirements, namely kinematic information (speed and direction) and resource information (available memory, CPU information, and data transmission rate). Specific values ​​that conform to the normal distribution are generated based on the random generation method.

[0150] (2) Model: In order to verify the effectiveness of the proposed mechanism, a typical deep convolutional neural network VGG16 was selected to perform the CIFAR10 classification task.

[0151] (3) Training settings: 200 communication rounds, reconstruction ratio μ = 0.3, ΔT = 25, T end =150, the SGD optimizer is used in local optimization, the learning rate η=0.01, and the batch size of each data sample is 32.

[0152] (4) Comparison method:

[0153] 1) Dense: The original dense model, which is the benchmark algorithm with the highest inference accuracy;

[0154] 2) Static: Train a randomly generated sparse network that meets the target sparsity from scratch, without updating the network structure during training, as the baseline algorithm with the lowest inference accuracy.

[0155] 3) RigL: A dynamic sparse training method that currently works well in centralized scenarios, which guides the update of sparse network structures based on gradient information.

[0156] 4) FedDST: A federated learning dynamic sparse training algorithm with relatively good performance. It allows each client to periodically update the sparse network structure through RigL based on its own local data. The server then aggregates all local sparse network structures and reorganizes a new global sparse network based on the weight amplitude.

[0157] Step 4: Result analysis:

[0158] Generally speaking, the higher the sparsity, the lower the computational and communication overheads during federated learning training, but at the same time the greater the difficulty of training. The goal of this system is to maintain the accuracy loss of the model within a small range at the highest possible sparsity, thereby minimizing the communication and computational resource overheads.

[0159] pass Figure 7 The comparison of experimental results in shows that the pre-training stage on the public dataset and the optimized initialization sparse network method provide a better starting point for the FedCET algorithm proposed in this invention, effectively reducing the sparse network exploration time and significantly improving the final performance. Figure 8 The algorithm comparison in the figure shows the comprehensive comparison between the algorithm FedCET proposed in this system and other baseline algorithms. Thanks to the improvement in parameter exploration rate, the model accuracy obtained by FedCET training is significantly better than other baseline methods. Fig. 9 The final accuracy comparison of various methods at different sparsity levels is shown. When the sparsity S = 0.9, in the case of extreme sparsity, FedCET's performance advantage is particularly prominent, and its performance is only 5% lower than the baseline algorithm, while the performance gap between other algorithms and the baseline algorithm is as high as 16%. This result further proves that increasing the parameter exploration rate can effectively enhance the representativeness and fitting ability of sparse networks.

[0160] In terms of overhead reduction at different sparsity levels, Fig.10The communication cost (Commu_cost) and FLOPs computation cost (FLOPs_cost) of FedCET at different sparsities are shown. Other sparse training methods are basically consistent with this method, because at the same target sparsity, the number of parameters retained by different methods is basically the same, and the total FLOPs computation cost is the sum of all multiplication and addition operations. Thanks to the advantage of dynamic sparse training that trains a sparse network structure throughout the process, as the sparsity of the model increases, the resource cost of the method proposed in this system is significantly reduced, such as Fig.10 As shown, compared with training an original dense model, when S = 0.95, FedCET only requires 0.192× the FLOPs computation overhead and 0.095× the communication overhead on the VGG16 model.

[0161] In summary, the federated dynamic sparse training system based on parallel expansion in the Internet of Vehicles scenario involved in the present invention has been designed and implemented, achieving the goal of maintaining low model performance loss on the basis of significantly reducing communication and computing overheads. Its effectiveness in improving the model accuracy of federated learning training in resource-constrained Internet of Vehicles scenarios has been verified.

Claims

1. A dynamic sparse training method for Internet of Vehicles federation based on cluster expansion, characterized in that: Construct a hierarchical federated learning training architecture in the Internet of Vehicles scenario: The hierarchical federated learning training architecture in the Internet of Vehicles scenario is a two-layer client-server structure, including an edge server and a road system; perform federated dynamic sparse training based on the hierarchical federated learning training architecture in the Internet of Vehicles scenario: under the coordination and management of the edge server, the training node trains the sparse network structure based on local data, and periodically updates the sparse network structure according to a predefined reconstruction update standard; finally, a global network structure that meets the target sparsity is aggregated and reconstructed at the edge server.

2. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 1 is characterized in that: The edge server serves as a central node and is used to: a) Pre-training of neural network models: Pre-training is performed on public datasets based on local computing resources to obtain an initialized sparse network structure, which is then sparsified based on weight standards and sparsity requirements and applied to subsequent formal training; b) Vehicle client cluster division: collect and quantify the kinematic information and vehicle terminal resource information of vehicles entering the area covered by the edge server, and continue to divide the results into two parts until the actual exploration rate between vehicle clusters reaches 100% to obtain the client cluster results; c) Federated training process management: responsible for issuing sparse network structures and aggregating client sparse network structures to form a global sparse network structure, and reconstructing the aggregated model according to the predefined target sparsity.

3. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 2 is characterized in that: The process of obtaining the initialized sparse network model is as follows: the edge server pre-trains the neural network using a public data set, and obtains a sparse network structure that meets the target sparsity as the initialized sparse network structure through a predefined weight amplitude standard; The layer importance factor γ is introduced in the pre-training stage to dynamically determine the sparsity of different network layers in the sparse network structure. The layer importance factor γ of the lth layer is l The calculation formula is as follows: Among them, m l (1) is the number of non-zero parameters in the lth layer, m l (0) is the number of parameters with zero weights in layer l.

4. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 3 is characterized in that: The vehicle client cluster division is based on resource and model similarity; Set the initial target number of clusters to 2, and continue to divide each cluster recursively. After each division, calculate the actual exploration rate of each cluster in the current division result. When the total actual exploration rate of the cluster is 100%, the cluster division is terminated, otherwise continue cluster division; 1) After receiving the content uploaded by the vehicle node, the edge server normalizes the vehicle terminal resource information to obtain the vehicle resource vector The calculation formula is as follows: where f i is the CPU main frequency of the vehicle client i terminal, r i is the data transmission rate of vehicle client i terminal, a i is the available memory size of the vehicle client i terminal; 2) The server quantifies the model similarity of the current training model w of each vehicle node received, so as to measure the similarity of all features in the future. The calculation formula is as follows: where w i is the current local sparse network model of vehicle i and w j is the current local sparse network model of vehicle j; 3) Comprehensively consider the vehicle's kinematic information, vehicle terminal resource information, and the similarity of the current training model to improve the stability of the clustering results and calculate the similarity function. The calculation formula is as follows: Among them, α, β, and γ are hyperparameters used to adjust the impact of speed, resources, and model similarity on quantifying the similarity between vehicles; 4) Expansion exploration rate: The sparse network model received by the vehicle cluster from the central node is defined as the basic model. Each vehicle cluster first calculates the total number of resources of the current vehicle cluster, and based on this, calculates the target expansion exploration rate of the current vehicle cluster; calculates the average resource information of the current cluster, and uses the average value to calculate the size of the trainable cluster local sparse network model, and then calculates the achievable expansion exploration rate compared with the basic model, and compares the actual expansion exploration rate with the target expansion exploration rate. When the actual exploration rate is not less than the target exploration rate, the cluster stops dividing; the cluster expansion exploration rate calculation formula is as follows: The operator ∪ represents the merging of the exploration results of all clusters, M z represents the local sparse network model trained by cluster z, |W| counts the total number of parameters of the original dense network, and ∥·∥0 represents the l0 norm, which is used to count the total number of all explored parameters; After the cluster division is completed, the average model structure of the cluster is compared with the global model structure based on formula (3) for similarity, and sorted in descending order. First, the basic model θ is calculated. base In the dense network model θ dense The complement of the above can be used to obtain the explorable model θ ava Then, based on the actual expansion exploration rate of each cluster and the average sparse neural network model of the current vehicle cluster, some additional parameter connections are randomly generated for each vehicle cluster in sequence to obtain the expansion part θ extend The model structure θ used in the next round of training i , and then update the explorable model θ ava ; The calculation formulas are as follows: i ava =θ dense \i extend (6) i i =θ base ∪θ extend (7) i ava =θ ava \i extend (8) where θ dense is the complete dense network model, given before training; θ base It is the edge server that initializes the sparse network model; θ i is the sparse network model structure of cluster i.

5. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 4 is characterized in that: The target expansion exploration rate is calculated as follows: when performing dynamic cluster division, before each cluster division using the Kmeans algorithm, the target exploration rate of the current cluster needs to be calculated first; by measuring the ratio between the sum of computing resources owned by all members in the current vehicle cluster and the sum of computing resources in all vehicle clusters, the ratio is multiplied by 100% to obtain the target expansion exploration rate; the calculation formula is as follows: Where S i is the computing resources of the current cluster, is the total computing resources of all clusters.

6. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 4 is characterized in that: The actual expansion exploration rate is calculated as follows: When performing dynamic cluster division, in addition to calculating the target expansion exploration rate, the actual expansion exploration rate of the cluster needs to be calculated; the normalized resource vector interval in the cluster member is calculated by formula (2) and recorded as The specific calculation formula is as follows: Where R * It is a predefined parameter that defines the exploration rate that can be achieved per resource vector.

7. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 1 is characterized in that: The road system is composed of vehicle clusters, and vehicle nodes are divided into cluster head nodes and cluster member nodes according to different roles in the vehicle cluster structure; a) Cluster member node: responsible for performing federated training based on locally collected data and uploading the trained sparse network structure to the cluster head node; b) Cluster head node: It is not only responsible for local training, but also for aggregating models from cluster member nodes and uploading the aggregated models to the edge server to assist in updating and reconstructing the global model.

8. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 1 is characterized in that: The federated dynamic sparse training is a parallel and scalable federated dynamic sparse training mechanism FedCET; the parallel and scalable federated dynamic sparse training mechanism FedCET divides a large number of vehicle clients into several clusters, allowing different clusters to explore different neural network connections in parallel outside the basic sparse network structure, and completes the complete exploration of all connections of the dense network structure from a spatial perspective through collaboration between clusters, covering enough connections during the training process, and dynamically optimizing and updating the structure of the global sparse network on this basis, thereby greatly improving the generalization ability of the global sparse model in high-sparse scenarios and achieving higher model accuracy.

9. The method for dynamic sparse training of Internet of Vehicles federation based on cluster expansion according to claim 8 is characterized in that: When performing federated dynamic sparse training, the trained sparse network structure is represented by transmitting a sparse network mask; The method of generating a sparse network mask based on the local sparse network structure trained by the node is as follows: Let W be the sparse neural network structure parameter matrix, and the mask is a binary matrix M with the same shape as it, and each element m in M ​​is a,b By the corresponding weight w in W a,b The specific rules are: if w a,b =0, then m a,b =0; if w a,b ≠0, then m a,b =1; After generating the sparse network masks, the vehicle nodes in each cluster use these masks for local sparse training. The specific method is as follows: the vehicle node i of cluster z uses the corresponding sparse network mask M z Perform local sparse training; In the tth communication round, vehicle node i uses local data X i , optimized with learning rate η, the calculation formula is as follows: In order to further improve the efficiency of parameter exploration through training and optimize the sparse network structure, the parameter expansion exploration phase is introduced. The specific method is: every ΔT rounds, the sparse network structure is gradually fine-tuned through parallel parameter exploration between clusters. The goal is to obtain the optimal global sparse network structure M through iterative updates while meeting the sparsity requirements. * ; The edge server randomly explores new parameters of proportion f for each cluster to form a new sparse network structure; for the zth cluster, its sparse network structure is recorded as M z ,Through merging the sparse network structure of all clusters at the edge servers; The structure of the sparse network is gradually fixed by the cosine decay strategy, and the value of the new exploration parameter ratio f is initialized to μ. end The communication round decays to 0; in the tth communication round, the calculation formula of f is as follows: When f = 0, parameter exploration stops, and the global sparse network structure obtained at this time is the optimal sparse network structure M * ,In the subsequent training, the global sparse network structure remains fixed until the training stops after convergence; During aggregation, the sparse network structure trained for each vehicle cluster is personalized while the differences between clusters are kept low. Each sparse network structure is logically considered as two parts: 1) Public basic part w base , these parameters are shared by all clusters; 2) Private part w self , corresponding to the newly explored parameters of each cluster, which are private to the cluster; Suppose there are n z clients, the model of cluster z after aggregation is: Among them, |D i | is the amount of data owned by vehicle node i, and They represent the parameters of the basic public part and private part of the i-th vehicle node respectively. All clients participate in the aggregation of the public basic part, while the private part parameters are aggregated and averaged only among the clients that retain the current parameters. After the aggregation is completed, all vehicle nodes will receive the same public basic part parameters, while the private part parameters are independent between vehicle clusters. After the central node receives and aggregates the sparse network structures uploaded by all vehicle nodes, it reconstructs the sparse network to meet the target sparsity requirement. The specific method is as follows: every fixed round ΔT, the parameter sensitivity score g defined by formula (14) is selected from the currently explored sparse network structure. j Select the most promising sparse network weights, comprehensively consider the influence of gradient information and weight size, and reconstruct a new global sparse network structure M with a sparsity of S+(1-S)f G , and serves as the starting point for the next round of parallel expansion sparse training; until the preset T end When f=0 in the round, parameter exploration stops, and the sparse network structure of all vehicle terminals is exactly the same, and there is no private part anymore, which marks the end of the sparse network parameter exploration. After that, the network structure is used for training until convergence stops; Where L is the loss function; Communication cost calculation: The communication cost of training a sparse network is positively correlated with the model size. When the client and server communicate with each other, only the retained parameters are transmitted. The sparse mask representing the network structure is only transmitted when it is updated. Assuming that the original size of the model is L and the target sparsity is S, the total communication cost of each communication round for a single client is in is the additional overhead of transmitting the updated mask structure.

Images