Distributed vehicle task unloading method and system based on A2C

Abstract

The embodiment of the invention discloses a distributed vehicle task unloading method and system based on A2C, and relates to the technical field of data processing. The method comprises the following steps: acquiring current task information and current vehicle information of a vehicle; historical task information, current task information, current vehicle information and historical unloading actions serve as state variables and are input into a reinforcement learning agent, the agent outputs the current unloading actions, the unloading actions comprise task unloading nodes, vehicle-mounted sending equipment and the unloading proportion of an edge server, and the unloading proportion of the task unloading nodes, the vehicle-mounted sending equipment and the edge server is determined. The unloading node comprises a vehicle local server and an edge server; the current task is unloaded according to the current unloading action, an award is calculated according to the total unloading time delay and the total energy consumption, the network parameters of the intelligent agent are updated according to the award, and the updated intelligent agent is used for outputting the unloading action of the next task. According to the embodiment of the invention, the distributed vehicle task unloading method under the incomplete information condition is realized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a distributed vehicle task offloading method and system based on A2C. Background Art

[0002] With the development of intelligent connected technology, new in-vehicle applications are constantly emerging. Relying solely on the vehicle's own computing power can no longer meet the real-time requirements of such applications. The task offloading method based on mobile edge computing is a promising solution at present.

[0003] Existing methods usually collect global resource information first, such as the computing power, bandwidth and load of each node, and then use relevant algorithms for global scheduling according to task requirements. This type of method has the following disadvantages. First, due to node resource fluctuations or privacy requirements, global resource information cannot be obtained accurately and timely. Second, additional communication overhead is required before formulating each offloading strategy, which will have a greater impact on the real-time performance of the algorithm. Real-world communication scenarios are often more complex. Once signal obstruction occurs, the offloading strategy cannot be formulated in time. Therefore, the traditional complete information algorithm is not suitable for actual application scenarios. Summary of the invention

[0004] The embodiment of the present invention provides a distributed vehicle task offloading method and system based on A2C, which effectively realizes vehicle task offloading under incomplete information.

[0005] In a first aspect, an embodiment of the present invention provides a distributed vehicle task offloading method based on A2C, comprising:

[0006] S110, obtaining the current task information and current vehicle information of the vehicle;

[0007] S120, historical task information, current task information, current vehicle information and historical unloading actions are input as state variables into a reinforcement learning agent, and the agent outputs a current unloading action, wherein the unloading action includes a task unloading node, a vehicle-mounted sending device, and an unloading ratio at an edge server, and the unloading nodes include a local vehicle and an edge server;

[0008] S130, unloading the current task according to the current unloading action, calculating the reward according to the total unloading delay and the total energy consumption, updating the network parameters of the agent according to the reward, and the updated agent is used to output the unloading action of the next task.

[0009] In a second aspect, an embodiment of the present invention provides a distributed vehicle task offloading method based on A2C, comprising:

[0010] Each vehicle unloads the vehicle task of each time slot through the method of the above embodiment, and updates the reinforcement learning agent deployed in each vehicle, wherein each agent includes an action network and an evaluation network; at the same time,

[0011] Each vehicle forms a cluster with neighboring vehicles, and performs local aggregation and distribution of action networks within the cluster;

[0012] Vehicles in different areas upload local action network parameters to the macro base station to aggregate and distribute intelligent agents on a global scale.

[0013] In a third aspect, an embodiment of the present invention provides an A2C-based distributed vehicle task offloading system, comprising:

[0014] Multiple vehicles, each vehicle is used to unload the vehicle task of each time slot through the method described in the above embodiment, and update the reinforcement learning agent deployed on each vehicle, wherein each agent includes an action network and an evaluation network; at the same time, each vehicle is used to form a cluster with neighboring vehicles, and perform local aggregation and distribution of the action network within the cluster;

[0015] The macro base station is used to aggregate and distribute intelligent agents globally after vehicles in different areas upload local action network parameters to the macro base station.

[0016] The embodiment of the present invention proposes a distributed vehicle task unloading method based on A2C. Different from the traditional centralized task unloading method, the method uses historical task information, current task information, current vehicle information and historical unloading actions as state variables, outputs the current unloading action through the reinforcement learning agent, calculates the reward according to the total unloading delay and total energy consumption, and updates the network parameters of the agent according to the reward. The whole process does not require any node resource information, reduces unnecessary communication overhead, and can adaptively adjust the unloading strategy according to the vehicle energy consumption budget, and can achieve instant vehicle unloading under incomplete information without relying on any edge information. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 is an architecture diagram of an A2C-based distributed vehicle task offloading system provided by an embodiment of the present invention;

[0019] Figure 2is a flow chart of a distributed vehicle task unloading method based on A2C provided by an embodiment of the present invention;

[0020] Figure 3 It is a structural diagram of an Actor-Critic network provided by an embodiment of the present invention;

[0021] Figure 4 is a flow chart of another A2C-based distributed vehicle task unloading method provided by an embodiment of the present invention;

[0022] Figure 5 is a flowchart of another A2C-based distributed vehicle task unloading method provided by an embodiment of the present invention; Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0024] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0025] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0026] An embodiment of the present invention provides a distributed vehicle task offloading method based on A2C. To illustrate the method, a system architecture supporting the implementation of the method is first described. Figure 1is an architecture diagram of an A2C-based distributed vehicle task offloading system provided by an embodiment of the present invention, such as Figure 1 As shown, the system includes multiple vehicles, multiple micro base stations (Small-Cell Base Station, SBS) and macro base stations (Macro-Cell Base Station, MBS), and is suitable for highways or other vehicle networking traffic scenarios.

[0027] Taking a typical highway scenario as an example, base stations in a certain area are connected by optical fiber, and each micro base station is equipped with an edge server. The micro base station is responsible for task offloading, and the macro base station is responsible for regional management. In this embodiment, the edge server set under the coverage of the macro base station is recorded as Each server is denoted as variable m and The characteristics of a server can be represented by a tuple (f m ,b m ,q m ), where each term represents the computing power, wireless bandwidth, and storage capacity of edge server m. The time variable can be discretized into a set of equally spaced time slots, denoted as Each time slot

[0028] Vehicle u is moving at speed v u Driving on the road at a constant speed, the initial position is The vehicle can choose to perform local computation or offload some tasks to the edge server due to the limitation of communication range. However, due to the limitation of communication range, the vehicle can only select base stations within the signal coverage range. For the convenience of description, this embodiment represents the optional offloading action set at time t as A(t), and the task set as Each task And can only be processed in sequence, task r has three characteristics D r Indicates the amount of data for the task, C r represents the amount of computation required for the task, is the maximum acceptable delay of task r. The offloading action corresponding to task r can be expressed as a r , when task r is offloaded to server m, there is a r =m, that is otherwise If you select Local Calculation, the variable And a r =0.

[0029] The total delay of the task includes three parts: transmission time, calculation time and relay time. Among them, the relay time means that when the task is calculated at the base station, the vehicle has left the coverage of the base station. At this time, several macro base stations are required to transmit the calculation results of the task, and additional processing time is required in this process. The total energy consumption of the task includes the transmission energy consumption of sending the task to the base station and the calculation energy consumption of the local part. Since the calculation result is relatively small, the receiving energy consumption is not considered. This embodiment decomposes the above process into two cases: local calculation and edge calculation, and performs problem modeling separately.

[0030] For local computing, when task r chooses local computing, use f local represents the calculation frequency of the vehicle, then the calculation time T local for:

[0031]

[0032] Energy consumption E caused by the calculation process local for:

[0033]

[0034] where κ is the effective switched capacitance, which depends on the device’s chip architecture.

[0035] For edge computing, when task r is offloaded to server m and the offload ratio is θ r When the transmission time is expressed as T trans , the corresponding calculation method is as follows:

[0036]

[0037] Among them, R mu is the channel data rate between the vehicle and the edge server m at time t, which is calculated as follows:

[0038]

[0039] Where ρ0 is the channel power gain at this distance, β is the path loss exponent, and p r is the transmission power, σ 2 is the power noise, d m (t) is the distance between the vehicle and m at time t.

[0040] The computing resources allocated to this task on server m are In the case of cmp The calculation formula is:

[0041]

[0042] If the vehicle has driven out of the communication range of base station m when the task calculation is completed, the variable z r =1, otherwise z r = 0. In this case, the calculation result needs to be returned by several macro base stations as relay points, and the additional relay time is:

[0043] T ext =kλ (6)

[0044] Where k is the number of macro base stations, and λ is the processing time of data in the macro base station, which is defined as a fixed constant. Since the calculation result of the task is often much smaller than the size of the original data, the delay of the task calculation result being transmitted back to the vehicle is not considered.

[0045] The task unloading delay T off It can be expressed as the maximum value of the local computing delay and the offloading delay:

[0046]

[0047] At the same time, offloading tasks to edge servers will result in energy loss. The longer the transmission time, the greater the energy consumption. Vehicles are usually equipped with multiple transmitting devices with different powers. The transmission power of the selected transmitting device is represented by p r , we can get the energy E consumed by task r off :

[0048]

[0049] Combining the above two actions, the total unloading delay of task r is T r for:

[0050]

[0051] Total energy consumption E r for:

[0052]

[0053] The remaining energy of the vehicle at this time is represented by E res , the energy consumption of local computing cannot exceed the remaining power, so there is a constraint:

[0054] E local <E res (11)

[0055] After each unloading task, the vehicle updates the remaining energy at the next moment:

[0056] E res ←E res -E r (12)

[0057] In this embodiment, the vehicle task unloading strategy is determined with the goal of minimizing the total delay and total energy consumption. The weights of the total delay and total energy consumption in the minimum goal are first determined. Optionally, the task to be calculated for the vehicle is recorded as r, and the calculation amount is C r , the maximum tolerable delay is This embodiment improves the weighting method, uses the delay requirement to divide the importance of tasks, and additionally uses the vehicle speed v to adjust the weight. When the task calculation amount is larger, the delay requirement is shorter or the vehicle speed is higher, the delay weight of the task is higher. Therefore, the delay weight The calculation method is as follows:

[0058]

[0059] Among them, α is a scaling factor used to adjust the weight size.

[0060] Next, determine the weight coefficient of energy consumption. In this embodiment, take a new energy vehicle as an example and express the maximum mileage of the vehicle under full charge as L u , the remaining distance to the destination is S u , the expected power consumption is roughly expressed as E p , the current power is E res , the amount of electricity used for task offloading is E ext , then E ext The calculation method is as follows:

[0061]

[0062] E ext =E res -E p (15)

[0063] When E ext When the energy consumption is relatively sufficient, a constant form is used to weight the energy consumption. In this case, delay is used as the main optimization target. When E ext When it is less than the threshold value H, it is weighted in logarithmic form. At this time, the weight of energy consumption is increased. The calculation method is as follows:

[0064]

[0065] Among them, ω is the weight coefficient, N is a constant, H is the energy consumption threshold, and H = e -N In practical applications, the size of N can be flexibly adjusted to achieve personalized energy consumption bias.

[0066] In summary, the optimization target OV of this embodiment can be defined as:

[0067]

[0068] So we get the final optimization problem:

[0069]

[0070] Among them, constraint C1 is the energy constraint, C2 is the unloading action constraint, the vehicle can only choose one unloading action, C3 and C4 are constraints on the value range of the action, C5 means that the unloading ratio can only take optional discrete values ​​to reduce the complexity of the problem, and C6 means that the task needs to be completed within the specified time.

[0071] Based on the above system architecture and optimization issues, Figure 2 is a flow chart of a distributed vehicle task offloading method based on A2C provided by an embodiment of the present invention. The method is mainly used to solve the problem of vehicle task offloading under incomplete information, and aims to obtain an efficient task offloading strategy without relying on any edge resource information. The method can not only achieve lower latency and energy loss, but also achieve a higher task offloading success rate. The method can be executed by the vehicle in the above system, and can also be executed by other electronic devices, such as Figure 2 As shown, the specific steps include:

[0072] S110: Acquire the current task information and current vehicle information of the vehicle.

[0073] The current task information refers to the relevant information of the task to be unloaded and executed at the current moment, and the current vehicle information refers to the information such as the location of the vehicle at the current moment. This embodiment obtains the task information and vehicle information at each moment in real time as the data source of the entire method.

[0074] S120, historical task information, current task information, current vehicle information and historical unloading actions are input as state variables into a reinforcement learning agent, and the agent outputs the current unloading action, wherein the unloading action includes a task unloading node, a vehicle-mounted sending device, and an unloading ratio at the edge server, and the unloading nodes include the vehicle local and the edge server.

[0075] This embodiment is based on a reinforcement learning agent to select the unloading action for the current task. Optionally, due to the unknown environment node resource information, the characteristics and unloading results of the two most recent tasks in the past can be used as references, and the characteristics of the task to be unloaded at the current moment, the vehicle position, etc. can be introduced as elements of the state space. Specifically, the state space can be represented as S u (t), which is The four parts are as follows:

[0076]

[0077] The specific meaning and definition of each part are as follows:

[0078] Historical mission information

[0079] Current task information

[0080] Current vehicle information

[0081] Historical Actions and Feedback

[0082] Each variable can be solved by the calculation formula in the above system architecture, and the solved variables together constitute the state S at time t u (t). The agent observes the state S at time t. u (t), and output the action as the current unloading strategy. Optionally, the action space is represented as A, and the specific action taken by the agent is a(t), where the optional actions include local unloading, edge unloading, sending device, and unloading ratio. Usually a vehicle has π sending devices with different powers and τ optional unloading ratios, then the action space can be defined as:

[0083]

[0084] Among them, local means the action or strategy offloaded to the vehicle. Used to identify whether to uninstall the index edge servers, dev0,…,dev π They are used to identify whether to use the vehicle-mounted sending device with index 0,…,π to send the unloading task to the edge server, ratio0,…,ratio τ They represent the τ proportions of offloading tasks to edge servers respectively, and the × here corresponds to the multiplication of sets, i.e., the Cartesian product.

[0085] S130, unloading the current task according to the current unloading action, and calculating the reward according to the total unloading delay and the total energy consumption; updating the network parameters of the agent according to the reward, and the updated agent is used to output the unloading action of the next task.

[0086] The agent is based on the state S u (t) takes action a(t), and the environment will give a reward R u (t) is used to evaluate the quality of the action and guide the network to update parameters. Optionally, the vehicle can set a waiting threshold locally If the task is completed within the specified time, the task is considered to be unloaded successfully, and the negative of the target value is used as a reward. Otherwise, a smaller constant -G is returned as a penalty, and the task is calculated locally. Therefore, the reward function can be in the following form:

[0087]

[0088] In a specific implementation, the above-mentioned reinforcement learning agent adopts an Actor-Critic network structure, and improves the traditional Actor-Critic method from three levels: exploration strategy, action mask, and collision request. At the same time, it combines action feedback and dynamic reward function to obtain the task offloading strategy through online learning. In order to illustrate the improvements of this embodiment, the basic principles of the Actor-Critic method are first introduced. The Actor-Critic method combines the policy gradient and value function estimation methods to solve decision-making problems in discrete or continuous action spaces. The algorithm consists of two neural networks: an action network (Actor) and an evaluation network (Critic). The network structure is as follows: Figure 3 shown.

[0089] The A2C algorithm introduces an advantage function to measure the pros and cons of taking an action relative to the average level. The Actor network update formula can be expressed as follows:

[0090]

[0091] in, and They represent the network parameters of the Actor network at time t+1 and time t, β represents the learning rate, Indicates that in state s t Select each action a t The probability vector, A(s t ,a t ) indicates state s t and action a t When the advantage value is positive, it means that the current action is feasible, and its probability of being selected is gradually increased through gradient ascent. On the contrary, when the advantage value is negative, the action probability will decrease. A(s t ,a t ) is defined as follows:

[0092] A(s t ,a t )=Q(s t ,a t )-V(s t ) (twenty two)

[0093] The above formula includes two parts: state action value Q and state value V. To simplify the calculation process, the A2C algorithm uses the Critic network to estimate the state value function and approximates the Q function according to the Bellman equation as follows:

[0094] Q(s t ,a t )≈r t +γV(s t+1 ) (twenty three)

[0095] Among them, r t Represents the reward value at the current moment.

[0096] Therefore, the advantage function A(s t ,a t ) can be expressed as:

[0097] A(s t ,a t )=r t +γV(s t+1 )-V(s t ) (twenty four)

[0098] This conversion may introduce a certain estimation error, but the impact of this error can be ignored. The loss function of the Critic network can be expressed as:

[0099]

[0100] The algorithm does not set a clear exploration strategy, and only selects actions based on the output of the current policy network. However, if an agent only focuses on the currently known optimal strategy, it will not be able to explore other potentially better strategies. This is the so-called "exploration-exploitation" dilemma. To solve this problem, this embodiment adds parameterized noise to the linear layer of the neural network to guide the Actor network to select an action different from the current strategy and enhance its exploration ability. The following describes the process of adding parameterized noise and the changes in agent learning after adding noise.

[0101] Optionally, the linear layer of the Actor network can be decomposed into three parts: mean, standard deviation and random noise; and the decomposition method can be used to reduce the amount of noise parameters and reduce the computational complexity; the improved Actor network is used for online training and learning, random noise is added once at the beginning of each iteration, and the balance between exploration and utilization is adaptively adjusted during training.

[0102] In a specific implementation, the weight of the linear layer in the Actor network is represented as w, and the bias is represented as b, then the output of the network is y=wx+b. In order to add parameterized noise, each weight and bias parameter of the linear layer can be decomposed into three parts: mean μ, standard deviation σ and random noise. At this time, the linear layer is defined as:

[0103] y=(μ w +σ w ⊙ε w )x+μ b +σ b ⊙ε b (26)

[0104] Among them, ε w and ε b It refers to Gaussian noise that obeys the standard normal distribution, and ⊙ is the multiplication of the corresponding elements.

[0105] Correspondingly, when the network parameters of the agent are updated according to the reward, the network will automatically adjust the proportion of exploration and utilization according to the value of different states and actions, and use the gradient descent method to update μ w , σ w , μ b and σ b , adaptively increase or decrease exploration; at the same time, random noise can be extracted from the Gaussian distribution to update ε w and ε b , by the updated μ w +σ w ⊙ε w and μ b +σ b ⊙ε b As the final network parameter. Furthermore, if the input dimension of the linear layer is p and the output dimension is q, a total of pq+q noise variables are required. To reduce the number of noise elements, the noise term can be decomposed. Specifically, ε w is a matrix of size p×q, ε b is a vector of dimension b. w Each item of is expressed as ε b Each item of is expressed as Then we can use two variables ε i and ε j The calculation method is:

[0106]

[0107] Among them, f is a function, there is no fixed requirement, generally choose Through the above decomposition process, only p noise variables ε need to be extracted at this time iand q noise variables ε j By multiplying the p-dimensional noise vector and the q-dimensional noise vector, we can get the p×q noise matrix ε w , using q as the noise vector as ε b .

[0108] The above is the improvement of the Actor-Critic network in the exploration strategy of this embodiment. The following is the improvement of the action mask of this embodiment. Due to the limitation of the communication range, the vehicle cannot unload tasks to the base station outside the communication range, but the Actor network will only select the unloading action according to the action probability, so it may output such invalid unloading strategy. To avoid this situation, this embodiment uses a mask to filter its action probability. The definition of the mask can be expressed as M(t):

[0109] M(t)={m1,m2,…,m |A|} (29)

[0110] The mask corresponding to the optional node is 1, and the others are 0. r-2 or a r-1 If the task fails to be unloaded, the mask of the corresponding position or It is also 0. Assume that the action probability output by the Actor network model is π t ={p1,p2,...,p |A|}, then the output action probability after the mask operation is π′ t :

[0111] π′ t ={p1m1,p2m2,…,p |A| m |A|} (30)

[0112] At this time, the probability of unselectable actions is set to 0, thus avoiding the selection of such strategies.

[0113] In addition, in terms of request collision, since the unloading action of the vehicle is random, any SBS may receive a large number of tasks in a certain time slot. However, due to the limited number of resources, the SBS cannot meet the resource requirements of all tasks. To solve this problem, this embodiment sets a task upper limit for each base station, and the micro base station will perform the following operations on the tasks arriving in each time slot:

[0114] S1-1. Each micro base station allocates necessary computing and storage resources to the incoming tasks in sequence. Optionally, the task set received by SBSm in time slot t is represented as Due to resource limitations, the base station must meet the following two constraints:

[0115]

[0116] SBSm will pre-allocate necessary storage and computing resources for each task according to n=1,2,..,U in the order in which the tasks are received.

[0117] S1-2. When the remaining resources cannot meet the resource requirements of a task, all subsequent tasks including the current task are discarded, and the remaining resources are evenly distributed to the previously accepted tasks. or The edge server of the micro base station discards subsequent tasks including task k, and distributes the excess idle resources evenly to the first k-1 tasks.

[0118] S1-3. For abandoned tasks, the vehicle will subsequently observe high latency (no response after a certain period of time), and trigger the penalty mechanism of the reward function:

[0119] R u (t)=-G (33)

[0120] The vehicle can update the Actor network based on the penalty item, reduce the probability of selecting the base station, and complete the task calculation locally.

[0121] Figure 4 This is a flowchart of another A2C-based distributed vehicle task offloading method provided by an embodiment of the present invention, which is executed by the vehicle and macro base station in the above system. This method can further solve the problem of vehicle task offloading under incomplete information, aiming to obtain an efficient task offloading strategy without relying on any edge resource information. This method can not only achieve lower latency and energy consumption loss, but also achieve a higher task offloading success rate. Figure 4 As shown, the specific steps include:

[0122] S210. Each vehicle unloads the vehicle task of each time slot and updates the reinforcement learning agent deployed in each vehicle by the method described in any of the above embodiments.

[0123] Each vehicle is equipped with a reinforcement learning agent described in any of the above embodiments. Figure 3 The A2C network structure shown includes an action network and an evaluation network.

[0124] S220. While training locally, combined with the federated learning method, models of the intelligent agents in each vehicle are aggregated from both the global and local levels to improve the efficiency of algorithm training.

[0125] Due to vehicle mobility and limited communication range, the exploration range of the agent is often limited, and the exploration time is also limited, which brings challenges to the training of the algorithm. To solve this problem, each vehicle in this embodiment forms a cluster with neighboring vehicles, and performs local aggregation and distribution of action networks within the cluster to complete the aggregation of agent models at the local level; at the same time, vehicles in different areas upload local action network parameters to the macro base station to aggregate and distribute agents globally.

[0126] Combine the following Figure 5 , respectively explain the specific methods of local training, global aggregation and local aggregation:

[0127] 1) Local training. Each vehicle locally deploys the deep reinforcement agent described in any of the above embodiments. The agent is based on A2C, uses a noise layer instead of an ordinary linear layer, and uses an action mask to filter invalid strategies. This method learns the distribution of environmental resources and updates local parameters based on historical action feedback. Specifically, when a vehicle generates a new task, the user inputs the corresponding state vector into the Actor network. After filtering the action mask, the agent selects the action according to the action probability and performs task offloading. The environment will give a reward value as feedback, and then the Critic network calculates the advantage function based on the reward value and updates the network parameters. The Actor network will use policy gradients to update the network parameters based on the action probability and advantage value.

[0128] 2) Global aggregation. To solve the problem of limited communication range, this embodiment integrates the Actor networks of vehicles in different areas, and uses the learning experience and advantages of different vehicles to obtain a more global and comprehensive model to speed up the learning efficiency of the model (i.e., the agent). Optionally, the improved A2C algorithm is trained online locally, and when a certain number of iterations is met, the vehicle uploads the local Actor network parameters and the last reward value to the macro base station. The macro base station will use the Softmax function and the reward value to calculate the corresponding weights for the received model, and perform weighted aggregation and distribution.

[0129] In a specific implementation, the user uploads the local Actor network parameters and the last reward value to the MBS, and each MBS is responsible for the global aggregation and distribution of all vehicle network models in the area. Considering the different user contributions, the MBS will use the size of the reward value uploaded by the user as the selection criterion to aggregate and distribute the collected sub-models. The calculation formula is as follows:

[0130]

[0131]

[0132] Among them, w u and They represent the agent weight and network parameters of vehicle u, Th3 represents the number of agents participating in global aggregation, and R i represents the network parameters of the agent of vehicle i participating in the global aggregation, θ g It represents the agent network parameters after local aggregation, which will be updated to the agents of all vehicles in the MBS responsible area.

[0133] 3) Local aggregation. Since the resource distribution in the adjacent area has a certain similarity, in order to avoid repeated learning and reduce the global communication overhead, short-range communication technology can be used to directly perform fast local aggregation between vehicles. Optionally, the vehicle can set a timer locally. When the timing meets a certain time threshold, a broadcast request is sent to neighboring vehicles. Among them, vehicles that meet the relative distance difference and relative speed difference form a cluster. All vehicles are scored based on the two factors of computing power and distance. The vehicle with the highest score serves as the cluster head and is responsible for the aggregation and distribution of local models within the cluster.

[0134] In a specific implementation, any vehicle u can proactively send a broadcast request with a radius of R to a set of neighboring vehicles u′, and communicate with all the neighboring vehicles in the area that meet the requirements. and The vehicles are merged into a cluster, where and v i are the two-dimensional position coordinates and speed of vehicle i respectively, I is the maximum reliable communication distance, and J is the maximum acceptable speed difference.

[0135] Taking into account the differences in vehicle performance and average relative distance, the following evaluation function can be constructed:

[0136]

[0137] Among them, G(u) represents the evaluation score of vehicle u, k1 and k2 are positive weight coefficients, which represent the influence of relative distance and computing power respectively. Each vehicle u can calculate the score of the current user according to G(u) and share it with all nodes in the cluster. The node with the highest score will be elected as the cluster head, and the remaining vehicles will send the local model parameters to the cluster head node. After the cluster head node receives the model parameters of multiple vehicles, it uses the average aggregation algorithm to quickly aggregate these parameters and send them down. The calculation formula is as follows;

[0138]

[0139]

[0140] in, represents the agent network parameters of vehicle i, Represents the network parameters after local aggregation.

[0141] Optionally, because the communication overhead between vehicles is relatively low, local aggregation between vehicles can usually be performed more frequently. Local aggregation can be triggered once when the vehicle first enters the MBS range, and then triggered once every fixed time interval. For global aggregation, a quantity threshold can be used, which is triggered when the number of models participating in the aggregation reaches the threshold. The specific value of the threshold can be set and adjusted according to the actual scenario and algorithm effect.

[0142] In summary, this embodiment takes a realistic high-speed scenario as an example to construct a vehicle task unloading system architecture based on a hybrid architecture of a micro base station and a macro base station, wherein the micro base station is responsible for calculating the task, the macro base station is responsible for managing the area, and cooperates with the vehicle to complete the unloading of the vehicle task. Based on this system architecture, this embodiment provides a distributed vehicle task unloading method based on A2C. Unlike the traditional centralized task unloading method, this method does not require any node resource information, reduces unnecessary communication overhead, and can adaptively adjust the unloading strategy according to the vehicle energy consumption budget. Specifically, this method is based on the A2C algorithm, uses the noise layer to improve the Actor network linear layer, and adaptively adjusts the proportion of exploration and learning during the training process, without manually setting the probability of exploration; considering the problem of communication range, this embodiment sets a dynamic mask for the output layer to filter invalid actions and avoid the selection of some actions; due to the lack of information exchange, the vehicle may have resource competition problems, so this embodiment uses the necessary resource allocation strategy to accept some tasks and reject tasks that cannot meet resource requirements, and uses penalty items to reduce the selection probability of the action.

[0143] In addition, in order to improve the training efficiency of the algorithm, this embodiment combines the hierarchical federated learning method with the A2C algorithm, reduces repeated learning processes through local aggregation, expands the vehicle exploration range through global aggregation, and improves global perception capabilities. Among them, at the local level, due to the similarity of resource distribution in adjacent areas, in order to avoid repeated learning, the vehicle will set a local timer and regularly send broadcast requests to neighboring vehicles. The vehicles that meet the speed and distance differences will form a cluster, and the cluster head will be selected according to the computing power and relative distance, and local model aggregation will be performed within the cluster. At the global level, in order to solve the problem of vehicle communication range limitation, the vehicle will upload the last reward value and its own Actor network parameters to the macro base station. When the macro base station receives a certain number of models, the Softmax function is used for weighted aggregation and distribution.

[0144] The embodiment of the present invention also provides a distributed vehicle task unloading system based on A2C, the basic structure of the system is as follows: Figure 1 As shown, multiple vehicles and macro base stations are included.

[0145] Among them, each vehicle is used to unload the vehicle tasks of each time slot and update the reinforcement learning agent deployed on each vehicle through the method described in any of the above embodiments, wherein each agent includes an action network and an evaluation network; at the same time, each vehicle is used to form a cluster with neighboring vehicles and perform local aggregation and distribution of the action network within the cluster.

[0146] The macro base station is used to aggregate and distribute intelligent agents on a global scale after vehicles in different areas upload local action network parameters to the macro base station.

[0147] The system and any of the above method embodiments are based on the same inventive concept. Any limitations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects as any of the above embodiments, which will not be repeated here.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A distributed vehicle task offloading method based on A2C, characterized in that: include: S110, obtaining the current task information and current vehicle information of the vehicle; S120, historical task information, current task information, current vehicle information and historical unloading actions are input as state variables into a reinforcement learning agent, and the agent outputs a current unloading action, wherein the unloading action includes a task unloading node, a vehicle-mounted sending device, and an unloading ratio at an edge server, and the unloading nodes include a local vehicle and an edge server; S130, unloading the current task according to the current unloading action, calculating the reward according to the total unloading delay and the total energy consumption, updating the network parameters of the agent according to the reward, and the updated agent is used to output the unloading action of the next task.

2. The method according to claim 1, characterized in that The historical task information Among them, D r-2 and D r-1 Represents the data volume of the two most recent tasks in the past, C r-2 and C r-1 They represent the amount of computation required for the two most recent tasks in the past. and Respectively represent the delay weights of the two most recent tasks in the past, and Respectively represent the energy consumption weights of the two most recent tasks in the past; Current task information Among them, D r , C r , and Respectively represent the data volume, computing volume, delay weight and energy consumption weight of the current task; Current vehicle information in, S u , L u and E res They respectively represent the vehicle's current two-dimensional position coordinates, the remaining distance to the destination, the maximum mileage under power conditions, and the current power level; Historical uninstall actions Among them, a r-2 and a r-1 They represent the offloading actions of the two most recent tasks in the past, T r-2 and T r-1 They represent the total unloading delay of the two most recent tasks in the past, E r-2 and E r-1 Respectively represent the total energy consumption of the two most recent tasks in the past; The action space A of the unloading action is: Among them, local means the action that is offloaded to the vehicle. Used to identify whether to uninstall the index edge servers, dev0,…,dev π They are used to identify whether to use the vehicle-mounted sending device with index 0,…,π to send the unloading task to the edge server, ratio0,…,ratio τ They represent the τ proportions of offloading tasks to edge servers; Reward function R u (t) is: Among them, T r and E r They represent the total offloading delay and total energy consumption of the current task respectively, and -G is a negative constant.

3. The method according to claim 1, characterized in that The intelligent agent includes an action network and an evaluation network; Before the historical task information, current task information, current vehicle information and historical unloading actions are input as state variables into the reinforcement learning agent, the method further includes: adding a noise component to the linear layer of the action network so that the action network becomes the following form: y=(μ w +s w ⊙e w )x+μ b +s b ⊙e b Among them, y represents the action variable, x represents the state variable, μ w , σ w and ε w Respectively represent the mean, standard deviation and random noise of each linear parameter in the action network, μ b , σ b and ε b Respectively represent the mean, standard deviation and random noise of the bias term in the action network, ε w and ε b All are Gaussian noises that obey the standard normal distribution, and ⊙ is the multiplication of the corresponding elements; Accordingly, updating the network parameters of the agent according to the reward includes: updating μ according to the reward using a gradient descent method. w , σ w , μ b and σ b , and extract random noise from the Gaussian distribution to update ε w and ε b , by the updated μ w +σ w ⊙ε w and μ b +σ b ⊙ε b as the final network parameters.

4. The method according to claim 1, characterized in that: The outputting of the current uninstallation action by the agent includes: The agent outputs the probabilities of all unloading actions in the action space; Define a mask set of the same size as the action space, where the mask element corresponding to the optional action is 1 and the mask element corresponding to the non-optional action is 0; For micro base stations outside the vehicle communication range, set their mask elements to 0; In the vehicle's recent unloading, if a certain unloading action leads to unloading failure, the mask element of the action is set to 0;. The final mask is used to filter the probability of each uninstall action, and the uninstall action with the highest probability after filtering is selected as the current uninstall action.

5. The method according to claim 1, characterized in that The reward is calculated based on the total unloading delay and the total energy consumption, including: The offloaded edge server allocates necessary computing and storage resources to the incoming tasks in order. When the remaining resources cannot meet the resource requirements of the current task, all subsequent tasks including the current task are discarded, and the remaining resources are evenly distributed to the previously accepted tasks. The vehicle whose task was abandoned observes high latency, triggering the penalty mechanism of the following reward function and completing the calculation of the task locally: R u (t)=-G Among them, -G is a negative constant.

6. A distributed vehicle task unloading method based on A2C, characterized in that: include: Each vehicle unloads the vehicle task of each time slot through the method described in claims 1 to 4, and updates the reinforcement learning agent deployed in each vehicle, wherein each agent includes an action network and an evaluation network; at the same time, Each vehicle forms a cluster with neighboring vehicles, and performs local aggregation and distribution of action networks within the cluster; Vehicles in different areas upload local action network parameters to the macro base station to aggregate and distribute intelligent agents on a global scale.

7. The method according to claim 6, characterized in that Each vehicle and neighboring vehicles form a cluster, and local aggregation and distribution of action networks are performed within the cluster, including: The vehicle sends a broadcast request to neighboring vehicles at regular intervals, forming a cluster with vehicles that meet the relative distance difference and relative speed difference; The vehicles in the cluster are scored according to computing power and distance, and the vehicle with the highest score is used as the cluster head, which performs local aggregation and distribution of the action network within the cluster.

8. The method according to claim 7, characterized in that The scoring of the vehicles in the cluster according to the computing power and the distance includes: According to the vehicle computing capability and the average relative distance, the following evaluation function is defined: Where G(u) represents the evaluation score of vehicle u; k1 and k2 are positive weighting coefficients, representing the influence of the relative distance and computing power of the vehicle, respectively; represents the set of other vehicles that belong to the same cluster as vehicle u, represents the number of other vehicles that belong to the same cluster as vehicle u, and They represent the two-dimensional geographic location coordinates of vehicle u, and They represent the two-dimensional geographic location coordinates of other vehicles i that belong to the same cluster as vehicle u, and f u Represents the computing capability of vehicle u.

9. The method according to claim 6, characterized in that The vehicles in different areas upload local action network parameters to the macro base station, and aggregate and distribute the intelligent bodies globally, including: After the local action network of vehicles in different areas meets a certain number of iterations, the local action network parameters and the last reward value are uploaded to the macro base station; The macro base station uses the Softmax function and each reward value to respectively calculate the weights corresponding to each received action network, and performs weighted aggregation and sends down the parameters of each action network according to each weight.

10. A distributed vehicle task unloading system based on A2C, characterized in that: include: A plurality of vehicles, each vehicle is used to unload the vehicle task of each time slot and update the reinforcement learning agent deployed on each vehicle by the method according to claims 1 to 4, wherein each agent includes an action network and an evaluation network; at the same time, each vehicle is used to form a cluster with neighboring vehicles and perform local aggregation and distribution of the action network within the cluster; The macro base station is used to aggregate and distribute intelligent agents on a global scale after vehicles in different areas upload local action network parameters to the macro base station.

Images