Model training method suitable for Internet of Vehicles federal learning environment

Abstract

The invention discloses a model training method suitable for an Internet of Vehicles federated learning environment, and belongs to the technical field of vehicle Internet of Things edge computing, and the method comprises the steps: S1, a center node obtains a corresponding federated learning global model, and loads a deep reinforcement learning decision module; s2, initializing a federated learning global model, and distributing the model to edge nodes in the region; s3, after training the model by each edge node participating in federated learning calculation, sending the model to the central node; s4, determining an aggregation model; s5, repeating the steps S3 and S4 until the center node uploads the trained model to the cloud; and S6, cloud training and optimizing a deep reinforcement learning decision module. According to the model training method suitable for the vehicle internet of things federated learning environment, effective screening of client models participating in aggregation is realized, and the availability and robustness of vehicle internet of things federated learning are improved while the total aggregation time is shortened.

Description

Technical Field

[0001] The present invention relates to the field of vehicle Internet of Things edge computing technology, and in particular to a model training method suitable for a federated learning environment of an Internet of Vehicles. Background Art

[0002] With the development of network infrastructure, edge computing technology has received increasing attention in the field of vehicle Internet of Things. It pushes data processing and analysis to the edge of the network, closer to the data source, which can reduce latency, increase data processing speed, and enhance the real-time response capability of the system. Federated learning, as a paradigm of distributed machine learning, has received increasing attention in the field of edge computing in recent years. It allows multiple participants to collaborate on training models while keeping data local. Its feature of only aggregating models without collecting training data to the cloud significantly improves information security and protects user privacy. In the combined application of edge computing and federated learning, edge devices can not only pre-process and analyze data, but also contribute their computing power and local data as participants in federated learning to jointly train more accurate models. This combination takes advantage of the low latency and high efficiency of edge computing, as well as the advantages of data privacy protection and distributed collaboration of federated learning, providing powerful data analysis and model training capabilities for vehicle Internet of Things.

[0003] Deep reinforcement learning is a technology that combines the perception ability of deep learning with the decision-making ability of reinforcement learning. It obtains rewards through continuous interaction between the intelligent agent and the environment, optimizes its own action strategy, and expects to maximize long-term benefits. Deep reinforcement learning is particularly suitable for sequential decision-making problems, such as applications in autonomous driving, games, robot control, etc. Deep reinforcement learning plays an important role in the federated learning of vehicle Internet of Things. Due to differences in the computing power of vehicles and the performance of sensors such as cameras, the value of models trained by different vehicles and the time consumed for training are different. Indiscriminately waiting for all models to be collected before aggregating not only increases the training time, but also increases the instability of model training. By combining deep reinforcement learning and federated learning and selecting the appropriate time for aggregation, the computing and training efficiency of vehicle Internet of Things applications can be effectively improved.

[0004] In addition, since vehicle models usually have millions or even tens of millions of parameters, it takes too much time to directly screen these parameters. Principal component analysis is a statistical method used to discover the intrinsic structure of data in multidimensional data sets where correlations may exist. It converts raw data into a set of statistically unrelated variables through orthogonal transformation, which are sorted by the size of the variance, thus providing a dimensionality reduction technique. Cooling down the model through principal component analysis can effectively improve the efficiency and robustness of the overall training framework. Summary of the invention

[0005] The purpose of the present invention is to provide a model training method suitable for a federated learning environment of an Internet of Vehicles, so as to solve the problems of low aggregation efficiency and unstable results of federated learning models in the vehicle Internet of Things environment existing in the above-mentioned background technology, and realize a model screening mechanism based on deep reinforcement learning. By analyzing the computing time of edge nodes and the training model parameters, the optimal aggregation time is obtained, so as to achieve improved performance after model aggregation while reducing time consumption.

[0006] To achieve the above object, the present invention provides a model training method suitable for a federated learning environment of an Internet of Vehicles, comprising the following steps:

[0007] S1. Based on input information such as location and vehicle, the central node obtains the corresponding federated learning global model from the cloud and loads the deep reinforcement learning decision module;

[0008] S2: Enter a training cycle, initialize the global federated learning model, and distribute the model to all edge nodes participating in the federated learning calculation in the region;

[0009] S3, entering an aggregation cycle, each edge node participating in the federated learning calculation sends the model to the central node after training the model with its own data set;

[0010] S4. The central node decides which models to aggregate based on the models uploaded by the edge nodes and the time consumed by their training models.

[0011] S5: Repeat steps S3 and S4 for a certain period of time. The central node uploads the trained model to the cloud, ends a training cycle, and starts to repeat the next training cycle.

[0012] S6. After a certain training cycle and a certain amount of experience accumulated in the experience replay pool, the deep reinforcement learning decision module is trained and optimized in the cloud.

[0013] Preferably, step S4 is specifically:

[0014] S41. When the central node receives the model of the edge node for the first time, it does not aggregate the model, but only records the time consumed by the edge node training, and uses the model and time as the status information of the next round of aggregation cycle;

[0015] S42. In the second and subsequent aggregation cycles, the central node performs principal component analysis and dimensionality reduction on the models of all edge nodes received last time, and combines the training consumption time as the state information of deep reinforcement learning;

[0016] S43, the deep reinforcement learning module of the central node selects an edge node number as an action based on the state information, and the action indicates that after receiving the model uploaded by the edge node with the number, the aggregation calculation of the federated learning is immediately started, and the models of all edge nodes received before the aggregation are aggregated, and the models of edge nodes received after the aggregation are not involved in the aggregation;

[0017] S44, calculating the difference between the model of each edge node participating in the aggregation and the federated learning global model as an increment, and accumulating these increments to the federated learning global model in a weighted average manner as a new federated learning global model, thereby ending an aggregation cycle;

[0018] S45. Use the test set to test the score of the new federated learning global model obtained in step S44 as the reward value, and record the state, action and reward of this aggregation cycle, and upload it to the cloud experience replay pool.

[0019] Preferably, the status information of step S41 is defined as:

[0020]

[0021] Among them, ω t represents the global model of federated learning in the tth round of aggregation cycle; Represents the model trained on the nth edge node.

[0022] Preferably, the principal component analysis of step S42 is specifically as follows: first, the covariance matrix of the model parameters needs to be calculated, which is expressed as Where n is the number of models; then the covariance matrix ∑ is decomposed into eigenvalues ​​to obtain the eigenvalue λ i and the eigenvector v i ; The transformation matrix W is formed by the eigenvectors of the covariance matrix, and each column of the transformation matrix W is an eigenvector; the original data set X is transformed to the new space through the transformation matrix W to obtain a new data set X′, and the columns of X′ are the principal components.

[0023] Preferably, the action of step S43 is used to determine the waiting time before the central node is aggregated. The value of the action is the time consumed by the training of the edge node with the selected number. The total training time of the edge node n is expressed as:

[0024]

[0025] in, Indicates the time when the edge node downloads the global model; Indicates the time for edge node to train the model; Indicates the time it takes for the edge node to transmit the trained model back to the central node;

[0026] The waiting time before aggregation is expressed as:

[0027]

[0028] Among them, a t,n is a binary variable indicating whether the car numbered n is selected in the tth aggregation period.

[0029] Preferably, the reward value of step S45 is expressed as:

[0030] r t =λF1(t)-(1-λ)H t

[0031] The reward value is the increment of the F1 score minus the weighted value of the time. The F1 score is expressed as F1 = 2* Precision Prec refers to the proportion of positive categories predicted by the model to be actually positive categories, and recall Recall refers to the proportion of positive categories correctly predicted by the model to be actually positive categories; λ is a value in the interval [0,1], which is used to balance the impact of F1 score and delay on the reward value.

[0032] Preferably, the modeling method for optimizing the deep reinforcement learning decision module is as follows:

[0033] The deep reinforcement learning decision module is based on the deep Q-network architecture, which is characterized by fitting the state value function Q.

[0034]

[0035] Among them, θ represents the parameters of the Q network; the loss function of the fitted state value function is

[0036]

[0037] The gradient descent optimization process of the Q function is expressed as

[0038]

[0039] Among them, α represents the learning rate; Denotes the Q function for parameter θ t The derivative of ; introduce the dual-depth Q network architecture, which uses two sets of network parameters θ and Θ, and the optimization process is expressed as

[0040] Therefore, the present invention adopts the above-mentioned model training method suitable for the federated learning environment of the Internet of Vehicles, which solves the problem of long aggregation time and unstable aggregation results caused by aggregating a large number of client models in the Internet of Vehicles environment. It realizes the effective screening of client models participating in the aggregation, and improves the availability and robustness of the federated learning of the Internet of Vehicles while reducing the total aggregation time.

[0041] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flow chart of a model training method applicable to a federated learning environment of an Internet of Vehicles according to the present invention;

[0043] Figure 2 A schematic diagram showing comparison of image recognition algorithm scores under different numbers of services in a real environment and a simulated environment according to an embodiment of the present invention;

[0044] Figure 3 Schematic diagram of comparison of the completion of the image recognition algorithm under different numbers of services in the real environment and the simulated environment of the embodiment of the present invention. DETAILED DESCRIPTION

[0045] Example

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0047] like Figure 1 As shown, a model training method suitable for a federated learning environment of an Internet of Vehicles includes the following steps:

[0048] S1. Based on input information such as location and vehicle, the central node obtains the corresponding federated learning global model from the cloud and loads the deep reinforcement learning decision module;

[0049] S2: Enter a training cycle, initialize the global federated learning model, and distribute the model to all edge nodes participating in the federated learning calculation in the region;

[0050] S3, entering an aggregation cycle, each edge node participating in the federated learning calculation sends the model to the central node after training the model with its own data set;

[0051] S4. The central node decides which models to aggregate based on the models uploaded by the edge nodes and the time consumed by their training models. Specifically:

[0052] S41. When the central node receives the model of the edge node for the first time, it does not aggregate the model, but only records the time consumed by the edge node training, and uses the model and time as the state information of the next round of aggregation cycle, which is defined as:

[0053]

[0054] Among them, ω t represents the global model of federated learning in the tth round of aggregation cycle; Represents the model trained on the nth edge node.

[0055] S42. In the second and subsequent aggregation cycles, the central node performs principal component analysis on the models of all edge nodes received last time and reduces the dimension, and combines the training consumption time as the state information of deep reinforcement learning; the principal component analysis is as follows: first, the covariance matrix of the model parameters needs to be calculated, which is expressed as Where n is the number of models; then the covariance matrix ∑ is decomposed into eigenvalues ​​to obtain the eigenvalue λ i and the eigenvector v i ; The transformation matrix W is formed by the eigenvectors of the covariance matrix, and each column of the transformation matrix W is an eigenvector; the original data set X is transformed to the new space through the transformation matrix W to obtain a new data set X′, and the columns of X′ are the principal components.

[0056] S43, the deep reinforcement learning module of the central node selects an edge node number as an action based on the state information. The action indicates that after receiving the model uploaded by the edge node with the number, the aggregation calculation of the federated learning is started immediately. The models of all edge nodes received before the aggregation are aggregated. The models of edge nodes received after the aggregation do not participate in the aggregation. The action determines the waiting time before the central node is aggregated. The value of the action is the time consumed by the training of the edge node with the selected number. The total training time of the edge node n is expressed as:

[0057]

[0058] in, Indicates the time when the edge node downloads the global model; Indicates the time for edge node to train the model; Indicates the time it takes for the edge node to transmit the trained model back to the central node;

[0059] The waiting time before aggregation is expressed as:

[0060]

[0061] Among them, a t,n is a binary variable indicating whether the car numbered n is selected in the tth aggregation period.

[0062] S44, calculating the difference between the model of each edge node participating in the aggregation and the federated learning global model as an increment, and accumulating these increments to the federated learning global model in a weighted average manner as a new federated learning global model, thereby ending an aggregation cycle;

[0063] S45. Use the test set to test the score of the new federated learning global model obtained in step S44 as the reward value, and record the state, action and reward of this aggregation cycle, and upload it to the cloud experience replay pool. The reward value is expressed as:

[0064] r t =λF1(t)-(1-λ)H t

[0065] The reward value is the increment of the F1 score minus the weighted value of time. The F1 score is expressed as Precision Prec refers to the proportion of positive categories predicted by the model to be actually positive categories, and recall Recall refers to the proportion of positive categories correctly predicted by the model to be actually positive categories; λ is a value in the interval [0,1], which is used to balance the impact of F1 score and delay on the reward value.

[0066] S5: Repeat steps S3 and S4 for a certain period of time. The central node uploads the trained model to the cloud, ends a training cycle, and starts to repeat the next training cycle.

[0067] S6. After a certain training cycle and a certain amount of experience accumulated in the experience replay pool, the deep reinforcement learning decision module is trained and optimized in the cloud. The modeling method for optimizing the deep reinforcement learning decision module is as follows:

[0068] The deep reinforcement learning decision module is based on the deep Q-network architecture, which is characterized by fitting the state value function Q.

[0069]

[0070] Among them, θ represents the parameters of the Q network; the loss function of the fitted state value function is

[0071]

[0072] The gradient descent optimization process of the Q function is expressed as

[0073]

[0074] Among them, α represents the learning rate; Denotes the Q function for parameter θ t The derivative of ; introduce the dual-depth Q network architecture, which uses two sets of network parameters θ and Θ, and the optimization process is expressed as

[0075] Figure 2 This is a comparison chart of the image recognition algorithm scores under different numbers of services in the real environment and the simulated environment. The model pre-trained based on the algorithm of this embodiment can obtain similar scores in the simulated environment and the real environment. When the number of services is small, the minimum difference between the two does not exceed 1%; when the number of services is large, the maximum difference between the two is 26%. This is because the real environment is subject to more hardware restrictions than the simulated environment, so it is difficult to load too many services, thereby reducing the score.

[0076] Figure 3 This is a comparison chart of the completion of the image recognition algorithm under different numbers of services in the real environment and the simulated environment. We regard the services completed within the tolerance time as completed. It can be seen from the figure that the performance of the pre-trained model in the real environment and the simulated environment is very close, and the maximum difference does not exceed 2.3%. Therefore, it can be shown that the model pre-trained by the algorithm of this embodiment can adapt to the real environment.

[0077] Therefore, the present invention adopts the above-mentioned model training method suitable for the federated learning environment of the Internet of Vehicles, realizes the effective screening of the client models participating in the aggregation, and improves the availability and robustness of the federated learning of the vehicle Internet of Things while reducing the total aggregation time.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A model training method suitable for a federated learning environment of Internet of Vehicles, characterized in that: The following steps are involved: S1. Based on the location and vehicle input information, the central node obtains the corresponding federated learning global model from the cloud and loads the deep reinforcement learning decision module; S2: Enter a training cycle, initialize the global federated learning model, and distribute the model to all edge nodes participating in the federated learning calculation in the region; S3, entering an aggregation cycle, each edge node participating in the federated learning calculation sends the model to the central node after training the model with its own data set; S4. The central node determines the aggregated model based on the model uploaded by the edge node and the time consumed by its training model; S5, repeat steps S3 and S4 until the central node uploads the trained model to the cloud, ends a training cycle, and starts to repeat the next training cycle; S6. After several training cycles and accumulating experience in the experience replay pool, the deep reinforcement learning decision module is trained and optimized in the cloud.

2. A model training method suitable for a federated learning environment of Internet of Vehicles according to claim 1, characterized in that: Step S4 is specifically as follows: S41. When the central node receives the model of the edge node for the first time, it does not aggregate the model, but only records the time consumed by the edge node training, and uses the model and time as the status information of the next round of aggregation cycle; S42. In the second and subsequent aggregation cycles, the central node performs principal component analysis and dimensionality reduction on the models of all edge nodes received last time, and combines the training consumption time as the state information of deep reinforcement learning; S43, the deep reinforcement learning module of the central node selects an edge node number as an action based on the state information, and the action indicates that after receiving the model uploaded by the edge node with the number, the aggregation calculation of the federated learning is started, and the model of the edge node received after the aggregation does not participate in the aggregation; S44, calculating the difference between the model of each edge node participating in the aggregation and the federated learning global model as an increment, and accumulating these increments to the federated learning global model in a weighted average manner as a new federated learning global model, thereby ending an aggregation cycle; S45. Use the test set to test the score of the new federated learning global model obtained in step S44 as the reward value, and record the state, action and reward of this aggregation cycle, and upload it to the cloud experience replay pool.

3. A model training method suitable for a federated learning environment of Internet of Vehicles according to claim 2, characterized in that: The status information of step S41 is defined as: Among them, ω t represents the global model of federated learning in the tth round of aggregation cycle; Represents the model trained on the nth edge node.

4. A model training method suitable for a federated learning environment of Internet of Vehicles according to claim 2, characterized in that: The principal component analysis of step S42 is specifically as follows: first, the covariance matrix of the model parameters is calculated, which is expressed as Where n is the number of models; then the covariance matrix ∑ is decomposed into eigenvalues ​​to obtain the eigenvalue λ i and the eigenvector v i ; The transformation matrix W is formed by the eigenvectors of the covariance matrix, and each column of the transformation matrix W is an eigenvector; the original data set X is transformed to the new space through the transformation matrix W to obtain a new data set X′, and the columns of X′ are the principal components.

5. The model training method applicable to the federated learning environment of Internet of Vehicles according to claim 2 is characterized in that: The action of step S43 is used to determine the waiting time before the central node is aggregated. The action value is the time consumed by the edge node training with the selected number. The total training time of edge node n is expressed as: in, Indicates the time when the edge node downloads the global model; Indicates the time for edge node to train the model; Indicates the time it takes for the edge node to transmit the trained model back to the central node; The waiting time before aggregation is expressed as: Among them, a t,n is a binary variable indicating whether the car numbered n is selected in the tth aggregation period.

6. A model training method suitable for a federated learning environment of Internet of Vehicles according to claim 2, characterized in that: The reward value of step S45 is expressed as: r t =λF1(t)-(1-λ)H t The reward value is the increment of the F1 score minus the weighted value of time. The F1 score is expressed as Precision Prec refers to the proportion of positive categories predicted by the model to be actually positive categories, and recall Recall refers to the proportion of positive categories correctly predicted by the model to be actually positive categories; λ is a value in the interval [0,1], which is used to balance the impact of F1 score and delay on the reward value.

7. The model training method applicable to the federated learning environment of Internet of Vehicles according to claim 1, characterized in that: The modeling method for optimizing the deep reinforcement learning decision module is as follows: The deep reinforcement learning decision module is based on the deep Q-network architecture, which is characterized by fitting the state value function Q. Among them, θ represents the parameters of the Q network; the loss function of the fitted state value function is The gradient descent optimization process of the Q function is expressed as Among them, α represents the learning rate; Denotes the Q function for parameter θ t The derivative of ; introduce the dual-depth Q network architecture, which uses two sets of network parameters θ and Θ, and the optimization process is expressed as

Images