A method for offloading computing in Internet of Vehicles based on multi-user game and federated learning

Abstract

The present invention proposes a method for offloading computing in the Internet of Vehicles based on multi-user games and federated learning. It adopts a binary offloading method to select channels and power, and uses a potential game algorithm and a federated learning algorithm to jointly optimize the deep Q network. The channels and transmission power selected in this way can effectively reduce the total overhead of the entire system, that is, reduce service delays and reduce system energy loss. The potential game algorithm used in the present invention can enable user vehicles to obtain near-optimal offloading decisions before training, allowing the deep Q network to converge more effectively. The trained intelligent agent can make the most favorable overall offloading decision in a specific environment, reducing system overhead, and the model trained by the deep Q network after federated learning optimization can be well adapted to more user scenarios and has a certain robustness.

Description

Technical Field

[0001] The present invention relates to a vehicle networking technology, and in particular to a vehicle networking computing offloading method, and more specifically to a vehicle networking computing offloading method based on multi-user game and federated learning. Background Art

[0002] According to statistics, as of the end of 2021, nearly 25 billion objects have been connected to the Internet, of which the Internet of Vehicles accounts for a large proportion and is increasing year by year. As an important part of the Intelligent Traffic System (ITS), this technology can significantly reduce the probability of traffic accidents, promote the development of new business models, and achieve efficient traffic management in many aspects, thereby playing an important role in ensuring safety, saving energy, and improving the environment.

[0003] However, the increase in demand for Internet of Vehicles services means tremendous pressure on communications and computing. In order to capture environmental information in real time, various sensors, radars and cameras installed in driverless vehicles can generate more than 1GB of data per second, and these data need to be processed in real time to ensure the reliability and safety of the vehicle during driving. However, the development speed of computer computing power is far behind the current growth rate of demand for communication services, and the semiconductor process is gradually entering a bottleneck period. The delay caused by insufficient computing power of autonomous driving vehicles cannot meet the reliability and safety requirements of this type of business.

[0004] In the Internet of Vehicles business, the goals that need to be optimized in the user-side computing offloading process are generally latency and energy loss, among which the most important is the optimization of latency. In the traditional computing offloading architecture, the data generated by the user side must be transmitted to the core network through multi-level routing for computing and then returned, which will cause very high latency. In order to solve the above problems, people introduced roadside mobile edge computing (MEC) units to assist user vehicles in completing computing offloading. At the same time, in order to further reduce latency, reduce energy loss, and avoid unreasonable channel allocation, the deep Q network (DQN) in deep reinforcement learning is widely used in the computing offloading process of the Internet of Vehicles, using trained models to guide autonomous driving vehicles to select the offloading scheme, channel, and transmission power. However, the algorithm converges slowly, and the limited local data of each agent will affect the robustness of the network. There is a random exploration process during training, especially when it is applied to multiple agents. It takes a lot of time and cannot protect the privacy of user data. Therefore, the multi-user game process is introduced to solve the problem of slow algorithm convergence, and federated learning is introduced to improve user data privacy and improve the generalization ability of the algorithm. Before training, all user vehicles gradually obtain a set of action spaces that can minimize the overall cost in the current state based on their current position and environment through a multi-user game process, and substitute them as initial values ​​into the deep Q network for training to determine whether the user offloads the calculation; during the training process, each user uploads the network parameters to the roadside computing unit RU for summary and aggregation after a certain step interval, and then sends them back to each user.

[0005] Therefore, the present invention proposes a computing task offloading algorithm based on multi-user game and federated learning. The algorithm targets the multi-user intelligent offloading process in the Internet of Vehicles scenario, with the goal of improving the quality of Internet of Vehicles services and protecting user data privacy, minimizing total latency and energy loss, and improving the generalization ability and robustness of model applications. As long as the environment does not change significantly, no retraining is required for a long period of time. Summary of the invention

[0006] Purpose of the invention: In view of the above problems existing in the prior art, a distributed computing offloading method for Internet of Vehicles based on multi-user game principle and federated learning is proposed. The method intends to adopt a multi-task binary offloading method to accelerate the convergence speed of deep Q learning, make the trained model more robust, achieve lower computing delay, and minimize energy loss.

[0007] Technical solution: Taking into account the local computing delay threshold, the purpose is to minimize the total system delay and energy loss by reasonably selecting the offloading unit, channel and transmission power. In order to improve the generalization ability of the training network model while reducing the system overhead, a binary offloading method is used to allocate computing resources, so that user vehicles can gradually obtain an offloading solution that minimizes the overall system overhead during the game iteration process. A deep reinforcement learning model is constructed based on the environmental state information, user vehicle action space, reward function, etc. in the Internet of Vehicles scenario, and a federated learning algorithm is used to optimize the deep Q network. The optimized network is used to train user vehicles. Each trained user vehicle, as an intelligent agent, can obtain location information and channel occupancy information from the current environment to determine whether to perform computing offloading and select the best offloading method, channel and transmission power. The above invention is achieved through the following technical solution: A method for computing offloading of Internet of Vehicles based on multi-user games and federated learning, comprising the following steps:

[0008] (1) Construct a vehicle network system model including user vehicles, service vehicles and roadside units. The user vehicle VC can offload computing tasks to the service vehicle VS and roadside unit RU to reduce the total system overhead C. sum ;

[0009] (2) Based on binary offloading, the unloading node, channel, and transmission power of the user vehicle are jointly selected, that is, only one action variable parameter is needed You can get all the unloading information of the user's vehicle VC;

[0010] (3) Decomposing the optimization of multi-user overall overhead into the optimization of each device overhead, a distributed unloading algorithm based on multi-user game is used to assist user vehicles in the Internet of Vehicles scenario to select the optimal unloading solution;

[0011] (4) The uninstall actions made by k users after the game Composition Collection As the initial offloading scheme, to minimize the total system overhead C sum Build a deep Q network for the target;

[0012] (5) In order to improve the generalization ability of the offloading model trained by deep reinforcement learning and make it easier to apply to more user vehicle scenarios without retraining, the deep Q network is optimized through federated learning;

[0013] (6),During the test, the user vehicle VC loads the pre-trained network model and selects the most suitable offloading channel and transmission power according to the current environmental information;

[0014] Furthermore, the step (1) comprises the following specific steps:

[0015] (1a), in this offloading model, the ultimate goal of optimization is to reduce the total system overhead C sum , which is composed of the total delay T sum and total energy loss E sum According to a certain proportion, the calculation formula can be expressed as:

[0016] C sum =λ1T sum +λ2E sum Expression 1

[0017] Among them, λ1 and λ2 are the total delay T sum and total energy loss E sum The proportion weights of , they also satisfy:

[0018]

[0019] In the business that pursues to minimize the delay, the value of λ1 is larger, and in the business with relatively limited energy, the value of λ2 is larger;

[0020] (1b), total delay T sum Includes the total data transmission delay of the entire system and total computational delay When the user vehicle chooses to transfer all data to local computing,

[0021] (1c), define the user vehicle set as: {0, 1, 2, ..., k, ..., K}, and the unloading channel set as: {0, 1, 2, ..., l, ..., L}. When the user vehicle chooses to calculate unloading, Use the following formula to calculate:

[0022]

[0023] Among them, d k represents the total amount of data that needs to be calculated by user vehicle k, L represents the total number of channels, Φ l Indicates whether the lth channel is occupied by user k. When Φ l The value of 0 indicates that the channel is not occupied. l The value of 1 indicates that the channel is occupied, B indicates the bandwidth allocated to each channel, and P k is the transmission power of the kth user vehicle, g C，S Indicates the channel gain generated between VC and VS;

[0024] (1d), total computation delay Use the following formula to calculate:

[0025]

[0026] Among them, ω k Indicates the ratio of computing resources allocated to user k by the auxiliary computing unit. Here, the computing resources are evenly divided according to the number of channels in each auxiliary computing unit. k =1, v represents the calculation speed of the corresponding calculation unit;

[0027] (1e), total energy loss E sum It consists of two parts: selecting the local calculation user's total energy consumption Total energy consumption of users who choose to calculate offload The formula is:

[0028]

[0029] (1f), the total energy loss of the local computing user is the computing energy consumption itself, and the vehicle that chooses to unload is not included in the calculation. The expression is:

[0030]

[0031] Select Calculate the total energy loss of unloaded users The energy loss of transmitting data to the user side is calculated. The vehicle selected for local calculation is not included in the calculation. The energy loss of data transmission back to the server is not included in the system overhead. It can be calculated using the following formula:

[0032]

[0033] Among them, χ is the energy coefficient, its value depends on the chip structure of the CPU, v k is the calculated speed of the user vehicle, and the others are as defined in (1c);

[0034] Furthermore, the step (2) includes the following specific steps:

[0035] (2a), define the number of service vehicles as n s , the number of roadside assistance computing units is n r , the number of subchannels that can be allocated to each link is L n , the transmission power can be selected as n P species, the size of the action space is:

[0036] A n =(n s +n r )×L n ×n P , A n ∈Z Expression 8

[0037] The value range is [0, (ns +n r )×L n ×n P ], The kth user vehicle only needs one variable parameter It can contain all the unloading information of a user's vehicle;

[0038] (2b), when When , it means that the user vehicle chooses to execute all computing tasks locally without offloading; when When , it means that the user vehicle chooses to offload the computing task to the auxiliary computing unit for computing;

[0039] (2c), when the user vehicle chooses to offload the computing task, its transmission power sequence number The choice can be calculated using the following formula:

[0040]

[0041] in, is rounded down, Mod is the remainder operation, and the value calculated by expression 2 is The number for selecting the transmit power starts from 0, indicating the number in the power array. The power array starts to store values ​​from the position with index 0.

[0042] (2d), similarly, the transmission channel L (k) The choice can be calculated using the following formula:

[0043]

[0044] Among them, Mod is the remainder operation, and L calculated by expression 3 is (k) The number of the channel, starting from 0, indicates that the Lth channel in the channel array is selected. (k) The array starts to store values ​​from the position with index 0, and arranges each channel option of each unloading channel in sequence;

[0045] Assuming that the channel of the roadside assistance computing unit RU is arranged in front of the array, and the channel of the service vehicle VS is arranged immediately after it, then through L (k) The value of Mod l can be used to know the unloading location selected by the user's vehicle. When this value is less than n r When the user vehicle chooses to offload the calculation to the roadside computing unit, when the value is greater than or equal to n r When ,the user vehicle chooses to offload the computation to the service vehicle;

[0046] Furthermore, the step (3) includes the following specific steps:

[0047] (3a), initialize the offload action set A = {0, 0, ..., 0}, indicating that all calculations are performed locally;

[0048] (3b), each user vehicle VC uses distributed computing to form a linked list of all user vehicles. Each game iteration cycle operates on a VC. When the last user vehicle is reached, the game returns to the first user vehicle and continues the game process until a VC is found that can make the overall cost C sum Minimum per user action And update to the uninstall action set;

[0049] (3c), for each multi-user game iteration, using C in step (1) sum The formula calculates the overall cost of the system under the current action set A. Then the user vehicle VC currently performing the operation obtains its own position, observes the system action set A, and calculates C when its own action is replaced by all other possible actions in turn. sum value, take the smallest C sum Action corresponding to the value As the new unloading decision of the user vehicle, it is updated to the system unloading decision set A at the same time, and the game iteration process of the next user vehicle begins, looping step (3c). The exit condition of the loop is: the total system cost C sum are equal in two consecutive game iteration cycles;

[0050] (3d), after several game iterations, the overall cost of all user vehicles is C sum When a value is reached that cannot be further reduced, the new set of unloading actions obtained after the multi-user game process is updated The cost value of each user's vehicle can maximize the benefits of the entire system without deviating from its own interests, that is, to reach the Nash equilibrium state. At this time, the action set It is the initial unloading scheme used for deep learning training after being optimized through the multi-user game process;

[0051] Furthermore, the step (4) comprises the following specific steps:

[0052] (4a), establish a state space set S, which contains the observed states of k user vehicles The number of training iterations episode and the probability of random action selection in the ε-greedy strategy ε, d k represents the amount of computing task data contained in user vehicle k, G k represents the status information of each channel formed by user vehicle k, including the channel status information related to unloading and the status information of auxiliary computing equipment, that is,

[0053] (4b), as defined in (2a), the number of service vehicles is n s , the number of roadside assistance computing units is n r , the number of subchannels that can be allocated to each link is L n , the transmission power can be selected as n P The size of the action space can be calculated using Expression 8. The action of each user vehicle Construct the action space set A, according to step (2), through The unloading object, the selected channel and the transmission power can be reversely and uniquely obtained;

[0054] (4c), define the reward function R(t), and set the maximum benefit value to The goal is to reduce delay and energy consumption. In order to minimize the total cost, the reward function of the user vehicle at step t is set as:

[0055]

[0056] in, It is a fixed value during the training process and can be adjusted freely to facilitate training. The reward function is set so that when the total system cost C sum When (t) is closer to the minimum value, the reward value R(t) obtained by the training network is larger and closer to the maximum value. Failure to meet the requirements for high latency and high energy consumption will penalize the system.

[0057] (4d) Based on the state space set S, action space set A and reward function R(t), the unloading scheme obtained by the multi-user game process As the initial decision, it is brought into the established deep Q network model for training;

[0058] (4e), each agent performs network prediction asynchronously, and the data obtained during the training process is stored in the Replay Memory. The training data generated are: Current Moment Action The reward value R obtained by executing this action k (t), next state value

[0059] Furthermore, the step (5) includes the following specific steps:

[0060] (5a), set a fixed step interval aggr, and every time the user vehicle passes this step interval, the local training network parameter λ k Upload to the roadside unit RU for federated aggregation;

[0061] (5b) The roadside unit RU aggregates all network parameters uploaded by users and takes the weighted average of all user vehicle training models participating in federated learning according to their contribution to obtain the best aggregation effect. The calculation formula is:

[0062]

[0063] in, is the global network parameter after federation aggregation, λ k is the parameter of the local network predicted by the kth user vehicle during the deep Q learning process, η k is the contribution of the kth user’s vehicle model parameters;

[0064] (5c), the roadside unit RU aggregates the global network parameters after federated learning Sent to each user car to update the local network parameter λ k If the set episode training period is not exceeded, the training will continue.

[0065] Further, the step (6) comprises the following specific steps:

[0066] (6a) The deep Q network model optimized by the multi-user game process and federated learning is loaded into the actual running vehicle networking scenario, and then the multi-user game process is performed to select the initial unloading action, and then the user agent selects the initial unloading action according to the environmental state information at this time. Determine the action to be applied to the next state with the trained network model

[0067] (6b), according to the action Get the current status The best unloading channel L (k) and the optimal transmission power to match it

[0068] Beneficial effect: The present invention proposes a method for unloading computing in the Internet of Vehicles based on multi-user game and federated learning, which selects channels and power in a binary unloading manner, and uses the potential game algorithm and the federated learning algorithm to jointly optimize the deep Q network. The channels and transmission power selected in this way can effectively reduce the total overhead of the entire system, that is, reduce service delay and reduce system energy loss. The potential game algorithm used in the present invention can enable user vehicles to obtain near-optimal unloading decisions before training, allowing the deep Q network to converge more effectively. The trained intelligent agent can make the most favorable unloading decision for the whole in a specific environment, reducing system overhead, and the model trained by the deep Q network after federated learning optimization can be well adapted to the scenarios of more users and has a certain robustness.

[0069] In summary, the vehicle network computing offloading method based on multi-user game and federated learning proposed in the present invention is superior in optimizing the convergence of deep learning networks, reducing system latency, and reducing energy loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of gradually reaching a Nash equilibrium in a multi-user game process provided by an embodiment of the present invention;

[0071] Figure 2 A schematic diagram of gradually reducing system overhead provided by an embodiment of the present invention;

[0072] Figure 3 A graph showing the convergence results of deep Q network training provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0073] The core idea of ​​the present invention is to use binary unloading, regard each user vehicle as an intelligent agent to establish a deep Q network model, and train the intelligent agent by jointly optimizing the deep Q network through a multi-user game algorithm plus a federated learning algorithm. Load the trained deep Q network model to guide the actual user to select the most appropriate unloading channel and transmission power.

[0074] The present invention is described in further detail below.

[0075] Step (1): In the IoV scenario, a system model including user vehicles, service vehicles and roadside units is constructed. The user vehicle VC can offload computing tasks to the service vehicle VS and roadside unit RU to reduce the total system overhead C. sum ;

[0076] The roadside unit RU is fixed in position, usually at the core of the constructed scene, specializing in providing computing services and possessing strong computing power, while the service vehicle VS is movable to cope with more complex and changeable scenes.

[0077] The specific steps include:

[0078] (1a), in this offloading model, the ultimate goal of optimization is to reduce the total system overhead C sum , which is composed of the total delay T sum and total energy loss E sum According to a certain ratio, it can be calculated by the following formula:

[0079] C sum =λ1T sum +λ2E sum Expression 1

[0080] Among them, λ1 and λ2 are the total delay Tsum and total energy loss E sum The proportion weights of , they also satisfy:

[0081]

[0082] In the business that pursues to minimize the delay, the value of λ1 is larger, and in the business with relatively limited energy, the value of λ2 is larger. The values ​​of λ1 and λ2 are adjusted dynamically and flexibly according to the specific business needs;

[0083] (1b), total delay T sum Contains the total data transmission delay of the entire system and total computational delay When the user vehicle chooses to delegate all data to local computing, since no data transmission is required,

[0084] (1c), the set of user vehicles with a total number of K is defined as: {0, 1, 2, ..., k, ..., K}, the set of unloading channels with a total number of L is defined as: {0, 1, 2, ..., l, ..., L}. When a user vehicle chooses to calculate unloading, Use the following formula to calculate:

[0085]

[0086] Among them, d k represents the total amount of data that needs to be calculated by user vehicle k, L represents the total number of channels, Φ l Indicates whether the lth channel is occupied by user k. When Φ l The value of 0 indicates that the channel is not occupied. l The value of 1 indicates that the channel is occupied, B indicates the bandwidth allocated to each channel, and P k is the transmission power of the kth user vehicle, g C，S Indicates the channel gain generated between VC and VS;

[0087] (1d), total computation delay Use the following formula to calculate:

[0088]

[0089] Among them, ω k Indicates the ratio of computing resources allocated to user k by the auxiliary computing unit. Here, the computing resources are evenly divided according to the number of channels in each auxiliary computing unit. When calculating locally, ω k =1, v represents the computing speed of the corresponding computing unit, which is differentiated between local computing, VS-assisted computing, and RU-assisted computing. Generally speaking, the computing power of RU is greater than that of VS and greater than that of VC.

[0090] (1e), total energy loss E sum It consists of two parts: selecting the local calculation user's total energy consumption Total energy consumption of users who choose to calculate offload The formula is:

[0091]

[0092] (1f), the total energy loss of the local computing user is the computing energy consumption itself, and the vehicle that chooses to unload is not included in the calculation. The expression is:

[0093]

[0094] Select Calculate the total energy loss of unloaded users The energy loss of transmitting data to the user side is calculated. The vehicle selected for local calculation is not included in the calculation. The energy loss of data transmission comes from the server side and is not included in the system overhead. It can be calculated using the following formula:

[0095]

[0096] Among them, χ is the energy coefficient, its value depends on the chip structure of the CPU, v k is the calculated speed of the user vehicle, and the others are as defined in (1c);

[0097] The above calculation method will be used for calling the following steps;

[0098] Step (2), based on binary unloading, jointly select the unloading node, channel, and transmission power of the user vehicle, that is, only one action variable parameter is needed You can get all the unloading information of the user's vehicle VC;

[0099] The binary offloading method provides two options for the user vehicle: offloading all computing tasks or not offloading all computing tasks. If partial offloading is used, the computational complexity will increase, and the offloading scheme will also be more complicated, which is not conducive to the convergence of deep learning network training.

[0100] The steps include:

[0101] (2a), define the number of service vehicles as n s , the number of roadside assistance computing units is n r , the number of subchannels that can be allocated to each link is L n , the transmission power can be selected as n P species, the size of the action space is:

[0102] A n =(n s +nr )×L n ×n P , A n ∈Z Expression 8

[0103] The value range is [0, (n s +n r )×L n ×n P ], The kth user vehicle only needs one variable parameter It can contain all the unloading information of a user's vehicle, which will be discussed in the following different situations;

[0104] (2b), when When , it means that the user vehicle chooses to execute all computing tasks locally without offloading; when When , it means that the user vehicle chooses to offload the computing task to the auxiliary computing unit for computing;

[0105] (2c), when the user vehicle chooses to offload the computing task, its transmission power sequence number The choice can be calculated using the following formula:

[0106]

[0107] in, is rounded down, Mod is the remainder operation, and the value calculated by expression 2 is The number for selecting the transmit power starts from 0, indicating the number in the power array. The power array starts to store values ​​from the position with index 0.

[0108] (2d), similarly, the transmission channel L (k) The choice can be calculated using the following formula:

[0109]

[0110] Among them, Mod is the remainder operation, and L calculated by expression 3 is (k) The number of the channel, starting from 0, indicates that the Lth channel in the channel array is selected. (k) The array starts to store values ​​from the position with index 0, and arranges each channel option of each unloading channel in sequence;

[0111] Assuming that the channel of the roadside assistance computing unit RU is arranged in front of the array, and the channel of the service vehicle VS is arranged immediately after it, then through L (k)The value of Mod l can be used to know the unloading location selected by the user's vehicle. When this value is less than n r When the user vehicle chooses to offload the calculation to the roadside computing unit, when the value is greater than or equal to n r When ,the user vehicle chooses to offload the computation to the service vehicle;

[0112] Step (3), decomposing the optimization of multi-user overall overhead into the optimization of each device's overhead, and using a multi-user game distributed unloading algorithm to assist user vehicles in the Internet of Vehicles scenario to select the optimal unloading solution;

[0113] This method can not only be used to obtain the initial decision before training, but also can be used to determine the initial decision in the actual test phase. It can also be used as a new algorithm to replace the deep Q network to determine each unloading decision.

[0114] The steps include:

[0115] (3a), initialize the offload action set A = {0, 0, ..., 0}, indicating that all calculations are performed locally;

[0116] (3b), each user vehicle VC uses distributed computing to form a linked list of all user vehicles. Each game iteration cycle operates on a VC. When the last user vehicle is reached, the game returns to the first user vehicle and continues the game process until a VC is found that can make the overall cost C sum Minimum per user action And update to the uninstall action set;

[0117] (3c), for each multi-user game iteration, using C in step (1) sum The formula calculates the overall cost of the system under the current action set A. Then the user vehicle VC currently performing the operation obtains its own position, observes the system action set A, and calculates C when its own action is replaced by all other possible actions in turn. sum value, take the smallest C sum Action corresponding to the value As the new unloading decision of the user vehicle, it is updated to the system unloading decision set A at the same time, and the game iteration process of the next user vehicle begins, looping step (3c). The exit condition of the loop is: the total system cost C sum are equal in two consecutive game iteration cycles;

[0118] (3d), after several game iterations, the overall cost of all user vehicles is C sum When a value is reached that cannot be further reduced, the new set of unloading actions obtained after the multi-user game process is updated The cost value of each user's vehicle can maximize the benefits of the entire system without deviating from its own interests, that is, to reach the Nash equilibrium state. At this time, the action set It is the initial unloading scheme used for deep learning training after being optimized through the multi-user game process;

[0119] Step (4): The uninstall actions made by k users after the game Composition Collection As the initial offloading scheme, to minimize the total system overhead C sum Build a deep Q network for the target;

[0120] The steps include:

[0121] (4a), establish a state space set S, which contains the observed states of k user vehicles The number of training iterations episode and the probability of random action selection in the ε-greedy strategy ε, d k represents the amount of computing task data contained in user vehicle k, G k represents the status information of each channel formed by user vehicle k, including the channel status information related to unloading and the status information of auxiliary computing equipment, that is,

[0122] (4b), as defined in (2a), the number of service vehicles is n s , the number of roadside assistance computing units is n r , the number of subchannels that can be allocated to each link is L n , the transmission power can be selected as n P The size of the action space can be calculated using Expression 8. The action of each user vehicle Construct the action space set A, according to the calculation process described in step (2), through Reversely and uniquely determine the offloading object, the selected channel and the transmission power;

[0123] (4c), define the reward function R(t), and set the maximum benefit value to The goal is to reduce delay and energy consumption. In order to minimize the total cost, the reward function of the user vehicle at step t is set as:

[0124]

[0125] in, It is a fixed value during the training process and can be adjusted freely to facilitate training. The reward function is set so that when the total system cost C sum When (t) is closer to the minimum value, the reward value R(t) obtained by the training network is larger and closer to the maximum value. Failure to meet the requirements for high latency and high energy consumption will penalize the system.

[0126] (4d) Based on the state space set S, action space set A and reward function R(t), the unloading scheme obtained by the multi-user game process As the initial decision, it is brought into the established deep Q network model for training;

[0127] (4e), each agent performs network prediction asynchronously, and the data obtained during the training process is stored in the Replay Memory. The training data generated are: Current Moment Action The reward value R obtained by executing this action k (t), next state value

[0128] Step (5), in order to improve the generalization ability of the offloading model trained by deep reinforcement learning, so that it can be easily applied to more user vehicle scenarios without retraining, the deep Q network is optimized through federated learning;

[0129] Since federated learning involves the process of aggregating and then distributing network parameters, the network model trained by it can be applied to scenarios with multiple users. For general deep Q networks, when the number of users is small, training can be performed based on the number of users. However, when the number of users increases, the time required for training in multi-user scenarios will increase dramatically, and even the problem of non-convergence of training results may occur. The model trained by federated learning can be directly applied to multi-user scenarios, which can save a lot of training time.

[0130] The steps include:

[0131] (5a), set a fixed step interval aggr, and every time the user vehicle passes this step interval, the local training network parameter λ k Upload to the roadside unit RU for federated aggregation;

[0132] (5b) The roadside unit RU aggregates all network parameters uploaded by users and takes the weighted average of all user vehicle training models participating in federated learning according to their contribution to obtain the best aggregation effect. The calculation formula is:

[0133]

[0134] in, is the global network parameter after federation aggregation, λ k is the parameter of the local network predicted by the kth user vehicle during the deep Q learning process, η k is the contribution of the kth user’s vehicle model parameters;

[0135] (5c), the roadside unit RU aggregates the global network parameters after federated learning Sent to each user car to update the local network parameter λ k If the set episode training period is not exceeded, the training will continue.

[0136] Step (6), during the test, the user vehicle VC loads the pre-trained network model and selects the most suitable unloading channel and transmission power according to the current environment information;

[0137] The steps include:

[0138] (6a) The deep Q network model optimized by the multi-user game process and federated learning is loaded into the actual running vehicle networking scenario, and then the multi-user game process is performed to select the initial unloading action, and then the user agent selects the initial unloading action according to the environmental state information at this time. Determine the action to be applied to the next state with the trained network model

[0139] (6b), according to the action Get the current status The best unloading channel L (k) and the optimal transmission power to match it

[0140] exist Figure 1 In the figure, it is described that in eight user vehicle scenarios, after multiple game iteration cycles, the total cost of each user vehicle VC gradually tends to be constant, and the cost of individual VC is not too high or too low, reaching a Nash equilibrium state.

[0141] exist Figure 2 In the multi-user game process, it is described that the total cost of the system continues to decrease due to the update of the unloading selection of each user's vehicle VC, and finally stabilizes at a minimum fixed value.

[0142] exist Figure 3 In the figure, it is described how the optimized deep Q network gradually converges as the number of iterations increases.

[0143] Based on the description of the present invention, it should not be difficult for those skilled in the art to see that the vehicle network computing offloading method based on multi-user game and federated learning of the present invention can reduce the overall system overhead and solve the problem of non-convergence of multi-user training networks.

[0144] The contents not described in detail in the present application belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A method for offloading computing in Internet of Vehicles based on multi-user game and federated learning, characterized in that: The steps include: (1) Construct a vehicle network system model that includes user vehicles, service vehicles, and roadside units. The user vehicle VC can offload computing tasks to the service vehicle VS and roadside unit RU to reduce the total system overhead C. sum ; (2) Based on the user vehicle, all computing tasks are executed locally without offloading or offloading to the auxiliary computing unit for calculation. The offloading node, channel, and transmission power of the user vehicle are jointly selected, that is, only one action variable parameter is needed. You can get all the unloading information of the user's vehicle VC; (3) The optimization of multi-user overall overhead is decomposed into the optimization of the overhead of each device, and a distributed unloading algorithm based on multi-user game is used to assist user vehicles in the Internet of Vehicles scenario to select the optimal unloading solution; (4) Uninstall actions made by K users after game Composition Collection As the initial offloading scheme, to minimize the total system overhead C sum Build a deep Q network for the target; (5) To improve the generalization ability of the offloading model trained by deep reinforcement learning, the deep Q network is optimized through federated learning; (6) During the test, the user vehicle VC loads the pre-trained network model and selects the most suitable offloading channel and transmission power according to the current environment information; Furthermore, the step (1) comprises the following specific steps: (1a), in this offloading model, the ultimate goal of optimization is to reduce the total system overhead C sum , which is composed of the total delay T sum and total energy loss E sum According to a certain proportion, the calculation formula can be expressed as: C sum =λ1T sum +λ2E sum Among them, λ1 and λ2 are the total delay T sum and total energy loss E sum The proportion weights of , they also satisfy: In the business that pursues to minimize the delay, the value of λ1 is larger, and in the business with relatively limited energy, the value of λ2 is larger; (1b), total delay T sum Contains the total data transmission delay of the entire system and total computational delay When the user vehicle chooses to transfer all data to local computing, (1c), define the user vehicle set as: {0, 1, 2, ..., k, ..., K}, and the unloading channel set as: {0, 1, 2, ..., l, ..., L}. When the user vehicle chooses to calculate unloading, Use the following formula to calculate: Among them, d k represents the total amount of data that needs to be calculated by user vehicle k, L represents the total number of channels, Φ l Indicates whether the lth channel is occupied by user k. When Φ l The value of 0 indicates that the channel is not occupied. l The value of 1 indicates that the channel is occupied, B indicates the bandwidth allocated to each channel, and P k is the transmission power of the kth user vehicle, g C,S Indicates the channel gain generated between VC and VS; (1d), total computation delay Use the following formula to calculate: Among them, ω k Indicates the ratio of computing resources allocated to user k by the auxiliary computing unit. Here, the computing resources are evenly divided according to the number of channels in each auxiliary computing unit. k =1, v represents the calculation speed of the corresponding calculation unit; (1e), total energy loss E sum It consists of two parts: selecting the local calculation user's total energy consumption Total energy consumption of users who choose to calculate offload The formula is: (1f), the total energy loss of the local computing user is the computing energy consumption itself, and the vehicle that chooses to unload is not included in the calculation. The expression is: Select Calculate the total energy loss of unloaded users The energy loss of transmitting data to the user side is calculated. The vehicle selected for local calculation is not included in the calculation. The energy loss of data transmission back to the server is not included in the system overhead. It can be calculated using the following formula: Among them, χ is the energy coefficient, its value depends on the chip structure of the CPU, v k is the calculated speed of the user vehicle, and the others are as defined in (1c); Further, the step (3) comprises the following specific steps: (3a) Initialize the offload action set A = {0, 0, ..., 0}, indicating that all calculations are performed locally; (3b) Each user vehicle VC performs distributed computing to find a solution that reduces the overall cost C sum Minimum single user action And update to the uninstall action set; (3c) After several game iterations, the total cost of all user vehicles is C sum When a value that cannot be further reduced is reached, it reaches the Nash equilibrium. The unloading action set obtained at this time is the initial unloading plan after optimization through the multi-user game process. Further, the step (4) comprises the following specific steps: (4a) Establish a state space set S, which contains the observed states of k user vehicles The number of training iterations episode and the probability of random action selection in the ε-greedy strategy ε, d k represents the amount of computing task data contained in user vehicle k, G k The gains between the channels formed by the user vehicle k; (4b) Define the number of service vehicles as n s , the number of roadside assistance computing units is n r , the number of subchannels that can be allocated to each link is L n , the transmission power can be selected as n P species, the size of the action space is: A n =(n s +n r )×L n ×n P ,A n ∈Z The actions of each user's vehicle Construct an action space set A through Reversely and uniquely determine the offloading object, the selected channel and the transmission power; (4c) Define the reward function R(t) and set the maximum benefit value to The goal is to reduce delay and energy consumption. In order to minimize the total cost, the reward function of the user vehicle at step t is set as: in, It is a fixed value during the training process and can be adjusted freely to facilitate training; (4d) Based on the state space set S, the action space set A and the reward function R(t), the unloading plan obtained in the multi-user game process is used as the initial decision and brought into the established deep Q network model for training; Further, the step (5) comprises the following specific steps: (5a) Set a fixed step interval aggr. Every time the user vehicle passes this step interval, the local training network parameter λ k Upload to the roadside unit RU for federated aggregation; (5b) The roadside unit RU aggregates all network parameters uploaded by users and takes the weighted average of all user vehicle training models participating in federated learning according to their contribution to obtain the best aggregation effect. The calculation formula is: in, is the global network parameter after federation aggregation, λ k is the parameter of the local network predicted by the kth user vehicle during the deep Q learning process, η k is the contribution of the kth user’s vehicle model parameters; (5c) The roadside unit RU aggregates the global network parameters after federated learning Sent to each user car to update the local network parameter λ k If the set episode training period is not exceeded, the training will continue.

Images