A model training method suitable for the federated learning environment of Internet of Vehicles

Abstract

The present invention discloses a model training method applicable to a federated learning environment of an Internet of Vehicles (IoV), belonging to the field of edge computing technology for the Internet of Vehicles (IoV), comprising steps S1, wherein a central node obtains a corresponding global federated learning model and loads a deep reinforcement learning decision module; S2, wherein the global federated learning model is initialized and distributed to edge nodes within a region; S3, wherein each edge node participating in the federated learning calculation sends the model to the central node after training the model; S4, wherein the aggregated model is determined; S5, wherein steps S3 and S4 are repeated until the central node uploads the trained model to the cloud; and S6, wherein the deep reinforcement learning decision module is trained and optimized in the cloud. The present invention adopts the above-mentioned model training method applicable to an IoV federated learning environment, realizes effective screening of client models participating in aggregation, and improves the availability and robustness of the IoV federated learning while reducing the total aggregation time.

Description

Technical Field

[0001] The present invention relates to the field of vehicle Internet of Things edge computing technology, and in particular to a model training method suitable for a federated learning environment of an Internet of Vehicles. Background Art

[0002] With the development of network infrastructure, edge computing technology is gaining increasing attention in the field of vehicle-to-vehicle (VIoT). This technology pushes data processing and analysis to the edge of the network, closer to the data source, reducing latency, increasing data processing speed, and enhancing the system's real-time responsiveness. Federated learning, a distributed machine learning paradigm, has gained increasing attention in the edge computing field in recent years. It allows multiple participants to collaboratively train models while maintaining data locality. Its ability to aggregate models without collecting training data to the cloud significantly improves information security and protects user privacy. In the combined application of edge computing and federated learning, edge devices not only perform data preprocessing and analysis but also act as participants in federated learning, contributing their computing power and local data to jointly train more accurate models. This combination leverages the low latency and high efficiency of edge computing with the data privacy protection and distributed collaboration advantages of federated learning, providing powerful data analysis and model training capabilities for the VIoT.

[0003] Deep reinforcement learning (DRL) combines the perceptual capabilities of deep learning with the decision-making capabilities of reinforcement learning. It uses continuous interaction between an intelligent agent and its environment to obtain rewards and optimize its action strategy in the hope of maximizing long-term benefits. Deep RL is particularly well-suited for sequential decision-making problems, such as those in autonomous driving, gaming, and robotic control. Deep RL plays a key role in federated learning for the Internet of Vehicles (IoT). Due to differences in vehicle computing power and the performance of sensors like cameras, the value and training time of trained models vary between vehicles. Indiscriminately waiting until all models are collected before aggregating them increases training time and instability. By combining deep RL and federated learning and selecting the appropriate aggregation time, computational and training efficiency can be effectively improved for IoT applications.

[0004] Furthermore, because vehicle models typically have millions or even tens of millions of parameters, directly screening these parameters is prohibitively time-consuming. Principal component analysis is a statistical method used to discover the inherent structure of data in potentially correlated multidimensional datasets. It uses an orthogonal transformation to transform raw data into a set of statistically uncorrelated variables, sorted by their variance, thus providing a dimensionality reduction technique. Cooling the model through principal component analysis can effectively improve the efficiency and robustness of the overall training framework. Summary of the Invention

[0005] The purpose of the present invention is to provide a model training method suitable for the federated learning environment of the Internet of Vehicles, so as to solve the problems of low aggregation efficiency and unstable results of federated learning models in the vehicle Internet of Things environment existing in the above-mentioned background technology, and realize a model screening mechanism based on deep reinforcement learning. By analyzing the computing time of edge nodes and the training model parameters, the optimal aggregation time is obtained, so as to achieve lower time consumption while improving the performance of the model after aggregation.

[0006] To achieve the above objectives, the present invention provides a model training method suitable for a federated learning environment of an Internet of Vehicles (IoV), comprising the following steps:

[0007] S1. Based on input information such as location and vehicle, the central node obtains the corresponding federated learning global model from the cloud and loads the deep reinforcement learning decision module;

[0008] S2: Enter a training cycle, initialize the global federated learning model, and distribute the model to all edge nodes participating in the federated learning calculation in the region;

[0009] S3: Entering an aggregation cycle, each edge node participating in the federated learning calculation sends the model to the central node after training the model using its own data set;

[0010] S4. The central node decides which models to aggregate based on the models uploaded by the edge nodes and the time it takes to train the models.

[0011] S5: Repeat steps S3 and S4 for a certain period of time. The central node uploads the trained model to the cloud, ending one training cycle and starting the next training cycle.

[0012] S6. After a certain training cycle and a certain amount of experience accumulated in the experience replay pool, the deep reinforcement learning decision module is trained and optimized in the cloud.

[0013] Preferably, step S4 is specifically as follows:

[0014] S41. When the central node receives the model of the edge node for the first time, it does not aggregate the model, but only records the time consumed by the edge node training, and uses the model and time as the status information of the next round of aggregation cycle;

[0015] S42. In the second and subsequent aggregation cycles, the central node performs principal component analysis and dimensionality reduction on the models of all edge nodes received last time, and combines the training consumption time as the state information of deep reinforcement learning;

[0016] S43. The deep reinforcement learning module of the central node selects an edge node number as an action based on the state information. This action indicates that after receiving the model uploaded by the edge node with this number, the aggregation calculation of federated learning will be started immediately. The models of all edge nodes received before the aggregation are aggregated. The models of edge nodes received after the aggregation are not involved in the aggregation.

[0017] S44. Calculate the difference between the model of each edge node participating in the aggregation and the federated learning global model as an increment, and add these increments to the federated learning global model in a weighted average manner as a new federated learning global model, thereby ending an aggregation cycle.

[0018] S45. Use the test set to test the score of the new federated learning global model obtained in step S44 as the reward value, and record the state, action, and reward of this aggregation cycle and upload it to the cloud experience replay pool.

[0019] Preferably, the status information of step S41 is defined as:

[0020]

[0021] Among them, ω t represents the global model of federated learning in the tth round of aggregation cycle; Represents the model trained on the nth edge node.

[0022] Preferably, the principal component analysis of step S42 is specifically as follows: first, the covariance matrix of the model parameters needs to be calculated, which is expressed as Where n is the number of models; then the covariance matrix ∑ is decomposed into eigenvalues ​​to obtain eigenvalues ​​λ i and the eigenvector v i The transformation matrix W is formed by the eigenvectors of the covariance matrix, and each column of the transformation matrix W is an eigenvector. The original data set X is transformed into a new space through the transformation matrix W to obtain a new data set X′, and the columns of X′ are the principal components.

[0023] Preferably, the action of step S43 is used to determine the waiting time before the central node aggregates. The value of the action is the time consumed by the training of the edge node with the selected number. The total training time of edge node n is expressed as:

[0024]

[0025] in, Indicates the time when the edge node downloads the global model; Indicates the time for edge node to train the model; Indicates the time it takes for the edge node to transmit the trained model back to the central node;

[0026] The waiting time before aggregation is expressed as:

[0027]

[0028] Among them, a t,n It is a binary variable indicating whether the car numbered n is selected in the tth aggregation period.

[0029] Preferably, the reward value of step S45 is expressed as:

[0030] r t =λF1(t)-(1-λ)H t

[0031] The reward value is the increment of F1 score minus the weighted value of time. The F1 score is expressed as F1=2* Precision Prec refers to the proportion of positive categories predicted by the model to be positive, and recall Recall refers to the proportion of positive categories correctly predicted by the model to be positive. λ is a value in the [0,1] interval, which is used to balance the effects of F1 score and delay on the reward value.

[0032] Preferably, the modeling method for optimizing the deep reinforcement learning decision module is as follows:

[0033] The deep reinforcement learning decision module is based on the deep Q-network architecture, which is characterized by fitting the state value function Q.

[0034]

[0035] Among them, θ represents the parameters of the Q network; the loss function of the fitted state value function is

[0036]

[0037] The gradient descent optimization process of the Q function is expressed as

[0038]

[0039] Among them, α represents the learning rate; Denotes the Q function for parameter θ t The derivative of ; introduce the dual-depth Q network architecture, which uses two sets of network parameters θ and Θ, and the optimization process is expressed as

[0040] Therefore, this invention employs the aforementioned model training method for federated learning in the vehicular Internet of Things (VIoT) environment, addressing the issues of long aggregation times and unstable aggregation results associated with aggregating a large number of client models. This method effectively screens participating client models, reducing overall aggregation time while improving the usability and robustness of VIoT federated learning.

[0041] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of a model training method applicable to a federated learning environment of an Internet of Vehicles (IoV) according to the present invention;

[0043] Figure 2 Schematic diagram comparing the image recognition algorithm scores under different numbers of services in the real environment and the simulated environment according to the embodiment of the present invention;

[0044] Figure 3 Schematic diagram comparing the completion of the image recognition algorithm under different numbers of services in the real environment and the simulated environment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] Example

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0047] like Figure 1 As shown, a model training method suitable for a federated learning environment of an Internet of Vehicles includes the following steps:

[0048] S1. Based on input information such as location and vehicle, the central node obtains the corresponding federated learning global model from the cloud and loads the deep reinforcement learning decision module;

[0049] S2: Enter a training cycle, initialize the global federated learning model, and distribute the model to all edge nodes participating in the federated learning calculation in the region;

[0050] S3: Entering an aggregation cycle, each edge node participating in the federated learning calculation sends the model to the central node after training the model using its own data set;

[0051] S4. The central node decides which models to aggregate based on the models uploaded by the edge nodes and the time consumed by their training models. Specifically:

[0052] S41. When the central node receives the model of the edge node for the first time, it does not aggregate the model, but only records the time consumed by the edge node training, and uses the model and time as the status information of the next round of aggregation cycle, which is defined as:

[0053]

[0054] Among them, ω t represents the global model of federated learning in the tth round of aggregation cycle; Represents the model trained on the nth edge node.

[0055] S42. In the second and subsequent aggregation cycles, the central node performs principal component analysis on the models of all edge nodes received last time and reduces the dimension, and combines the training consumption time as the state information of deep reinforcement learning; principal component analysis is specifically as follows: first, the covariance matrix of the model parameters needs to be calculated, which is expressed as Where n is the number of models; then the covariance matrix ∑ is decomposed into eigenvalues ​​to obtain eigenvalues ​​λ i and the eigenvector v i The transformation matrix W is formed by the eigenvectors of the covariance matrix, and each column of the transformation matrix W is an eigenvector. The original data set X is transformed into a new space through the transformation matrix W to obtain a new data set X′, and the columns of X′ are the principal components.

[0056] S43. The deep reinforcement learning module of the central node selects an edge node number as an action based on the state information. This action indicates that after receiving the model uploaded by the edge node with this number, the aggregation calculation of federated learning will be started immediately. The models of all edge nodes received before the aggregation are aggregated. The models of edge nodes received after the aggregation are not aggregated. The action determines the waiting time before the central node aggregates. The action value is the time consumed by the training of the edge node with the selected number. The total training time of edge node n is expressed as:

[0057]

[0058] in, Indicates the time when the edge node downloads the global model; Indicates the time for edge node to train the model; Indicates the time it takes for the edge node to transmit the trained model back to the central node;

[0059] The waiting time before aggregation is expressed as:

[0060]

[0061] Among them, a t,n It is a binary variable indicating whether the car numbered n is selected in the tth aggregation period.

[0062] S44. Calculate the difference between the model of each edge node participating in the aggregation and the federated learning global model as an increment, and add these increments to the federated learning global model in a weighted average manner as a new federated learning global model, thereby ending an aggregation cycle.

[0063] S45. Use the test set to test the score of the new federated learning global model obtained in step S44 as the reward value. Record the state, action, and reward of this aggregation cycle and upload them to the cloud experience replay pool. The reward value is expressed as:

[0064] r t =λF1(t)-(1-λ)H t

[0065] The reward value is the increment of F1 score minus the weighted value of time. The F1 score is expressed as Precision Prec refers to the proportion of positive categories predicted by the model to be positive, and recall Recall refers to the proportion of positive categories correctly predicted by the model to be positive. λ is a value in the [0,1] interval, which is used to balance the effects of F1 score and delay on the reward value.

[0066] S5: Repeat steps S3 and S4 for a certain period of time. The central node uploads the trained model to the cloud, ending one training cycle and starting the next training cycle.

[0067] S6. After a certain training cycle and a certain amount of experience accumulated in the experience replay pool, the deep reinforcement learning decision module is trained and optimized in the cloud. The modeling method for optimizing the deep reinforcement learning decision module is as follows:

[0068] The deep reinforcement learning decision module is based on the deep Q-network architecture, which is characterized by fitting the state value function Q.

[0069]

[0070] Among them, θ represents the parameters of the Q network; the loss function of the fitted state value function is

[0071]

[0072] The gradient descent optimization process of the Q function is expressed as

[0073]

[0074] Among them, α represents the learning rate; Denotes the Q function for parameter θ t The derivative of ; introduce the dual-depth Q network architecture, which uses two sets of network parameters θ and Θ, and the optimization process is expressed as

[0075] Figure 2 This chart compares the image recognition algorithm scores in real-world and simulated environments with varying numbers of services. Models pre-trained based on the algorithm of this embodiment can achieve similar scores in simulated and real-world environments. The minimum difference between the two is less than 1% when the number of services is small; when the number of services is large, the maximum difference is 26%. This is because real-world environments are subject to greater hardware constraints than simulated environments, making it difficult to handle an excessive number of services, which in turn reduces scores.

[0076] Figure 3 This chart compares the image recognition algorithm's completion rate under different service numbers in real and simulated environments. We consider services completed within the tolerance time as completed. As can be seen from the chart, the pre-trained model in this embodiment performs very similarly in real and simulated environments, with the largest difference not exceeding 2.3%. This demonstrates that the pre-trained model in this embodiment is adaptable to real-world environments.

[0077] Therefore, the present invention adopts the above-mentioned model training method suitable for the federated learning environment of the Internet of Vehicles, which realizes the effective screening of client models participating in aggregation, while reducing the total aggregation time and improving the availability and robustness of the federated learning of the Internet of Vehicles.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A model training method suitable for a federated learning environment of Internet of Vehicles, characterized in that: The following steps are involved: S1. Based on location and vehicle input information, the central node obtains the corresponding federated learning global model from the cloud and loads the deep reinforcement learning decision module; S2: Enter a training cycle, initialize the global federated learning model, and distribute the model to all edge nodes participating in the federated learning calculation in the region; S3: Entering an aggregation cycle, each edge node participating in the federated learning calculation sends the model to the central node after training the model using its own data set; S4. The central node determines the aggregated model based on the models uploaded by the edge nodes and the time consumed by their training models. S5. Repeat steps S3 and S4 until the central node uploads the trained model to the cloud, ending one training cycle and starting the next training cycle. S6. After several training cycles and accumulation of experience in the experience replay pool, the deep reinforcement learning decision module is trained and optimized in the cloud; Step S4 is specifically as follows: S41. When the central node receives the model of the edge node for the first time, it does not aggregate the model, but only records the time consumed by the edge node training, and uses the model and time as the status information of the next round of aggregation cycle; S42. In the second and subsequent aggregation cycles, the central node performs principal component analysis and dimensionality reduction on the models of all edge nodes received last time, and combines the training consumption time as the state information of deep reinforcement learning; S43. The deep reinforcement learning module of the central node selects an edge node number as an action based on the state information. This action indicates that after receiving the model uploaded by the edge node with this number, the aggregation calculation of the federated learning is started. The models of the edge nodes received after the aggregation do not participate in the aggregation. S44. Calculate the difference between the model of each edge node participating in the aggregation and the federated learning global model as an increment, and add these increments to the federated learning global model in a weighted average manner as a new federated learning global model, thereby ending an aggregation cycle. S45. Use the test set to test the score of the new federated learning global model obtained in step S44 as the reward value, and record the state, action, and reward of this aggregation cycle and upload it to the cloud experience replay pool; The principal component analysis of step S42 is as follows: first, the covariance matrix of the model parameters is calculated, which is expressed as Where n is the number of models; then the covariance matrix ∑ is decomposed into eigenvalues ​​to obtain eigenvalues ​​λ i and the eigenvector v i The transformation matrix W is formed by the eigenvectors of the covariance matrix, and each column of the transformation matrix W is an eigenvector. The original data set X is transformed into a new space through the transformation matrix W to obtain a new data set X′, and the columns of X′ are the principal components.

2. A model training method suitable for a federated learning environment of an Internet of Vehicles according to claim 1, characterized in that: The status information of step S41 is defined as: Among them, ω t represents the global model of federated learning in the tth round of aggregation cycle; Represents the model trained on the nth edge node.

3. A model training method suitable for a federated learning environment of an Internet of Vehicles according to claim 1, characterized in that: The action in step S43 is used to determine the waiting time before the central node aggregates. The action value is the time consumed by the edge node training with the selected number. The total training time of edge node n is expressed as: in, Indicates the time when the edge node downloads the global model; Indicates the time for edge node to train the model; Indicates the time it takes for the edge node to transmit the trained model back to the central node; The waiting time before aggregation is expressed as: Among them, a t,n It is a binary variable indicating whether the car numbered n is selected in the tth aggregation period.

4. A model training method suitable for a federated learning environment of an Internet of Vehicles according to claim 1, characterized in that: The reward value of step S45 is expressed as: r t =λF1(t)-(1-λ)H t The reward value is the increment of F1 score minus the weighted value of time. The F1 score is expressed as Precision Prec refers to the proportion of positive categories predicted by the model to be positive, and recall Recall refers to the proportion of positive categories correctly predicted by the model to be positive. λ is a value in the [0,1] interval, which is used to balance the effects of F1 score and delay on the reward value.

5. The model training method applicable to the federated learning environment of Internet of Vehicles according to claim 1, characterized in that: The modeling method for optimizing the deep reinforcement learning decision module is as follows: The deep reinforcement learning decision module is based on the deep Q-network architecture, which is characterized by fitting the state value function Q. Among them, θ represents the parameters of the Q network; the loss function of the fitted state value function is The gradient descent optimization process of the Q function is expressed as Among them, α represents the learning rate; Denotes the Q function for parameter θ t The derivative of ; introduce the dual-depth Q network architecture, which uses two sets of network parameters θ and Θ, and the optimization process is expressed as

Images