Calculation unloading and resource allocation method based on multi-agent reinforcement learning
By introducing multi-agent reinforcement learning in smart vehicles and MEC servers, distributed intelligent decision-making is realized, and the problems of low computing resource utilization efficiency and high task processing delay in multi-vehicle dynamic environment are solved, and the utilization efficiency and service quality of computing resources are improved.
Patent Information
- Application Number
- CN202411399651.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-03-11
AI Technical Summary
In the dynamic environment where multiple vehicles move rapidly, existing computing unloading and resource allocation methods are difficult to effectively deal with uncertainty and dynamics between vehicles, resulting in low computing resource utilization efficiency and high task processing delays.
Using the computational offloading and resource allocation method based on multi-agent reinforcement learning, intelligent vehicles and MEC servers are used as agents to make decisions through their respective neural networks, realize distributed intelligent decisions, and dynamically perform computation offloading decisions and resource allocation strategies.
Through this method, intelligent vehicles can make dynamic calculation and offload decisions based on real-time environmental status, and MEC servers can effectively allocate resources, reduce task processing delays, improve computing resource utilization efficiency, and ensure service quality.
Smart Images
Figure CN119597443A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of intelligent transportation, and in particular relates to a computation offloading and resource allocation method based on multi-agent reinforcement learning. Background Art
[0002] With the rapid development of the Internet of Things and artificial intelligence (AI) technologies, smart vehicles are gradually becoming mobile terminals that integrate multiple sensors and computing capabilities. These smart vehicles are usually equipped with in-vehicle infotainment systems, advanced driver assistance systems (ADAS), and vehicle-to-everything (V2X) communication modules, enabling smart vehicles to perform complex computing-intensive tasks such as real-time driving monitoring, high-precision navigation, video entertainment, and environmental monitoring. However, these computing-intensive tasks place higher demands on the computing resources of smart vehicles, especially in application scenarios with high real-time requirements, such as real-time driving monitoring scenarios.
[0003] Although the performance of smart vehicle hardware is constantly improving under the continuous optimization of researchers, the limitation of computing resources is still an issue that cannot be ignored. In order to solve this challenge, researchers have proposed a variety of computing offloading and resource allocation strategies, including optimizing on-board computing resources, introducing efficient algorithms, and utilizing cloud computing technology. After the concept of vehicle edge network (VEN) was proposed, mobile edge computing (MEC) became popular because it can provide more intelligent computing offloading and resource allocation by using perception capabilities. However, although the existing computing offloading and resource allocation methods using mobile edge computing technology have alleviated the limitation of computing resources to a certain extent, how to effectively deal with the uncertainty and dynamics between vehicles in a dynamic environment with multiple vehicles moving rapidly to achieve efficient computing offloading and resource allocation is still an unresolved problem. For example: some methods treat MEC servers as intelligent agents and train the neural networks corresponding to MEC servers. Vehicles do not have corresponding neural networks. Vehicles select the MEC servers closest to themselves to transmit computing-intensive tasks. This method ignores the mobility of vehicles and the dynamic changes in computing requirements, which may cause multiple vehicles to repeatedly select the same MEC server, thereby affecting the effect of computing offloading and resource allocation. For example, although some methods regard both vehicles and MEC servers as intelligent agents and train their corresponding neural networks, they only consider a single reward mechanism in algorithm design, which limits the ability of MEC servers to learn effective resource allocation strategies. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a computational offloading and resource allocation method based on multi-agent reinforcement learning, which utilizes mobile edge computing technology in a dynamic environment with multiple vehicles moving rapidly to effectively handle the uncertainty and dynamics between vehicles, so as to achieve efficient computational offloading and resource allocation, while optimizing task processing delays and improving the performance of intelligent vehicles in processing computationally intensive tasks, thereby ensuring service quality.
[0005] The technical solution adopted by the present invention to solve the above technical problems is: a method for computing offloading and resource allocation based on multi-agent reinforcement learning, characterized in that in a vehicle edge network, multiple smart vehicles and multiple base stations are arranged, each base station is equipped with a MEC server, the smart vehicles and the MEC server are both smart agents, all smart vehicles are driving fast on the road, all base stations are dispersed along the road, the signal of each base station covers part of the road, the signals of all base stations cover the whole road, the base station communicates with the smart vehicles within its signal coverage, and the base stations communicate with each other through optical fibers; for any smart vehicle, a first neural network consisting of a first actor network and a first critic network is constructed; for any MEC server, a second neural network consisting of a second actor network and a second critic network is constructed; the method comprises the following steps:
[0006] Step 1: Set i to represent the global round number, and set K to represent the global maximum round number, where the initial value of i is 1, i = 1, 2, ..., K;
[0007] Step 2: Set t to represent the single round time step, and set T to represent the single round maximum time step, where the initial value of t is 1, t = 1, 2, ..., T;
[0008] Step 3: At the tth time step, the intelligent vehicle that generates computationally intensive tasks is defined as a task vehicle, and the time when the computationally intensive tasks are generated follows a Poisson distribution; each task vehicle senses the environmental state, which includes its own vehicle position and the computationally intensive tasks generated by itself; at the same time, each MEC server also senses the environmental state, which includes the vehicle positions of all task vehicles within the signal coverage of its corresponding base station and the computationally intensive tasks generated by them;
[0009] Step 4: At the tth time step, for any task vehicle, the environmental state perceived by the task vehicle is input into the first actor network of the corresponding first neural network, the computational offloading decision of the task vehicle is obtained through the action output by the first actor network, the computational offloading decision of the task vehicle is input into the first critic network of the corresponding first neural network, and the quality of the computational offloading decision of the task vehicle is judged by the action output by the first critic network; for any MEC server, the environmental state perceived by the MEC server is input into the second actor network of the corresponding second neural network, the resource allocation strategy of each task vehicle within the signal coverage of the base station corresponding to the MEC server is obtained through the action output by the second actor network, the resource allocation strategy of each task vehicle within the signal coverage of the base station corresponding to the MEC server is input into the second critic network of the corresponding second neural network, and the quality of the resource allocation strategies of these task vehicles is judged by the action output by the second critic network; wherein, the resource allocation strategy includes communication resource allocation and computing resource allocation;
[0010] Step 5: At the tth time step, for any task vehicle, the task vehicle determines the MEC server to which it wants to offload computation based on its computation offloading decision, and determines the allocated communication resources and computing resources based on its resource allocation strategy; then the task vehicle uses the allocated communication resources to transmit the generated computation-intensive tasks to the determined MEC server, and the MEC server calculates the received computation-intensive tasks based on the allocated computing resources, wherein a transmission delay is generated when transmitting the computation-intensive tasks, and a computing delay is generated when calculating the computation-intensive tasks; then the task processing delay is calculated, and the task processing delay is the sum of the transmission delay and the computing delay; finally, the global reward is calculated based on the task processing delay, and the local reward is calculated based on the resource allocation strategy of the task vehicle;
[0011] Step 6: Update the environment state; then randomly extract data from the experience pool to train the first neural network corresponding to each task vehicle and the second neural network corresponding to each MEC server, and update the network parameters of the first neural network and the second neural network respectively;
[0012] Step 7: Determine whether t is less than T. If so, set t=t+1, and then return to step 3 to continue execution, where the "=" in t=t+1 is an assignment symbol; otherwise, execute step 8;
[0013] Step 8: Determine whether i is less than K. If so, set i=i+1 and t=1, and then return to step 2 to continue execution, where the "=" in i=i+1 is an assignment symbol; otherwise, end.
[0014] The first actor network includes three fully connected layers and an output layer, one of which is used as an input layer, and the other two are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 1024, and the number of neurons in the second hidden layer is 512; the first critic network is used as a global critic network. The first critic network includes four fully connected layers and an output layer, one of which is used as an input layer, and the other three are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, the third hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 1024, the number of neurons in the second hidden layer is 512, and the number of neurons in the third hidden layer is 256.
[0015] The second actor network includes three fully connected layers and an output layer, one of the fully connected layers is used as the input layer, and the other two fully connected layers are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 1024, and the number of neurons in the second hidden layer is 512; the second critic network is used as a local critic network. The second critic network includes three fully connected layers and an output layer, one of the fully connected layers is used as the input layer, and the other two fully connected layers are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 512, and the number of neurons in the second hidden layer is 256.
[0016] Compared with the prior art, the advantages of the present invention are:
[0017] 1) As intelligent entities, smart vehicles and MEC servers can independently perceive and respond to their respective environmental states and formulate decision-making strategies through their respective neural networks (including actor networks and critic networks), namely, computational offloading decisions and resource allocation strategies, respectively.
[0018] This distributed intelligent decision-making method enables each mission vehicle (intelligent vehicle that generates computing-intensive tasks) to dynamically make computing offloading decisions based on the real-time environmental status and its own computing needs, and select the best MEC server for computing offloading, thereby ensuring efficient use of resources; at the same time, the MEC server can comprehensively consider the needs of all mission vehicles within the signal coverage area, flexibly formulate and adjust resource allocation strategies, including the allocation of communication resources and computing resources, and effectively deal with the uncertainty and dynamics in multi-vehicle environments.
[0019] 2) By determining the computational offloading decision in the discrete action space and obtaining the resource allocation strategy in the continuous action space, the task processing delay (including transmission delay and computational delay) is directly optimized. The global reward is evaluated by calculating the task processing delay, and the local reward is calculated according to the resource allocation strategy of the task vehicle. These reward feedbacks are used to train and optimize the parameters of the neural network. This method helps to reduce task processing delay, improve the performance of intelligent vehicles in performing computationally intensive tasks, and ensure that the quality of service can be maintained in a dynamic environment.
[0020] 3) In the decision-making process, the intelligent agents only need to rely on their own perceived environmental states without the need for information from other intelligent agents, which enhances the robustness and autonomy of the system. This design is more in line with actual application scenarios, where intelligent vehicles and MEC servers may not be able to continuously and reliably exchange information, and simplifies the complexity of the system, making the application scope of the method of the present invention wider and more adaptable. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a flowchart of the implementation process of the method of the present invention;
[0022] Figure 2 This is a model diagram of the vehicle edge network;
[0023] Figure 3 A schematic diagram showing a curve comparison of task processing delays of two traditional algorithms and the method of the present invention as a function of the ratio of the number of intelligent vehicles to the number of MEC servers;
[0024] Figure 4 A schematic diagram of a curve comparison of the resource consumption of two traditional algorithms and the method of the present invention as the ratio of the number of intelligent vehicles to the number of MEC servers changes;
[0025] Figure 5 Schematic diagram of curve comparison of system costs of two traditional algorithms and the method of the present invention as the ratio of the number of smart vehicles to the number of MEC servers changes. DETAILED DESCRIPTION
[0026] The present invention is further described in detail below with reference to the accompanying drawings.
[0027] The present invention proposes a method for computation offloading and resource allocation based on multi-agent reinforcement learning, which is used in vehicle edge networks, such as Figure 2As shown, there are multiple smart vehicles and multiple base stations, each base station is equipped with a MEC (mobile edge computing) server, the smart vehicles and MEC servers are both intelligent entities, all smart vehicles are driving fast on the road, all base stations are scattered along the road, the signal of each base station covers part of the road, the signal of all base stations covers the whole road, the base station communicates with the smart vehicles within its signal coverage, and the base stations communicate with each other through optical fiber; for any smart vehicle, a first neural network consisting of a first actor network and a first critic network is constructed; for any MEC server, a second neural network consisting of a second actor network and a second critic network is constructed. The implementation flow chart of this method is shown in the following figure. Figure 1 As shown, the following steps are included:
[0028] Step 1: Initialize the environment parameters, including the global maximum number of rounds and the maximum time step of a single round; initialize the network parameters, including the network parameters of the first actor network and the first critic network, the second actor network and the second critic network, and the learning rate; set i to represent the global number of rounds, and set K to represent the global maximum number of rounds, where the initial value of i is 1, i = 1, 2, ..., K.
[0029] Step 2: Set t to represent the single-round time step, and set T to represent the single-round maximum time step, where the initial value of t is 1, t=1, 2,…,T.
[0030] Step 3: At the tth time step, the intelligent vehicle that generates computationally intensive tasks is defined as a task vehicle, and the time when the computationally intensive tasks are generated follows a Poisson distribution; each task vehicle perceives the environmental state, and the environmental state perceived by the task vehicle includes its own vehicle position and the computationally intensive tasks generated by itself; at the same time, each MEC server also perceives the environmental state, and the environmental state perceived by the MEC server includes the vehicle positions of all task vehicles within the signal coverage range of its corresponding base station and the computationally intensive tasks they generate.
[0031] Step 4: At the tth time step, for any mission vehicle, the environmental state perceived by the mission vehicle is input into the first actor network of the corresponding first neural network, and the computational offloading decision of the mission vehicle (the decision of which MEC server the mission vehicle selects for computational offloading) is obtained through the action output by the first actor network. The computational offloading decision of the mission vehicle is input into the first critic network of the corresponding first neural network, and the quality of the computational offloading decision of the mission vehicle is judged by the action output by the first critic network; for any MEC server, the environmental state perceived by the MEC server is input into the second actor network of the corresponding second neural network, and the resource allocation strategy of each mission vehicle within the signal coverage of the base station corresponding to the MEC server is obtained through the action output by the second actor network, and the resource allocation strategy of each mission vehicle within the signal coverage of the base station corresponding to the MEC server is input into the second critic network of the corresponding second neural network, and the quality of the resource allocation strategies of these mission vehicles is judged by the action output by the second critic network; wherein, the resource allocation strategy includes communication resource allocation and computing resource allocation.
[0032] In actual implementation, assuming there are 5 MEC servers, the first actor network outputs a 5-dimensional vector, and then through the processing of existing technology, the computational offloading decision of the task vehicle can be obtained, and the first critic network outputs a value to judge the quality of the computational offloading decision of the task vehicle; assuming that there are 3 task vehicles within the signal coverage range of the base station corresponding to a MEC server, the second actor network outputs a 6-dimensional vector, of which 3 dimensions are communication resources and the other 3 dimensions are computing resources. Then, through the processing of existing technology, the resource allocation strategies of the three task vehicles can be obtained, and the second critic network outputs a 3-dimensional vector, and each value in the vector is used to judge the quality of the resource allocation strategy of a task vehicle.
[0033] Step 5: At the tth time step, for any task vehicle, this task vehicle determines the MEC server to which it wants to offload computation based on its computation offloading decision, and determines the allocated communication resources and computing resources based on its resource allocation strategy; then this task vehicle uses the allocated communication resources to transmit the generated computation-intensive tasks to the determined MEC server, and the MEC server calculates the received computation-intensive tasks based on the allocated computing resources, wherein a transmission delay is generated when transmitting the computation-intensive tasks, and a computing delay is generated when calculating the computation-intensive tasks; then the task processing delay is calculated, and the task processing delay is the sum of the transmission delay and the computing delay; finally, the global reward is calculated based on the task processing delay, and the local reward is calculated based on the resource allocation strategy of this task vehicle.
[0034] Here, both the global reward and the local reward are calculated using existing techniques.
[0035] Step 6: Update the environment status; then randomly extract data from the experience pool to train the first neural network corresponding to each task vehicle and the second neural network corresponding to each MEC server, and update the network parameters of the first neural network and the second neural network respectively.
[0036] Step 7: Determine whether t is less than T. If so, set t=t+1, and then return to step 3 to continue execution, where the "=" in t=t+1 is an assignment symbol; otherwise, execute step 8.
[0037] Step 8: Determine whether i is less than K. If so, set i=i+1 and t=1, and then return to step 2 to continue execution, where the "=" in i=i+1 is an assignment symbol; otherwise, end.
[0038] In this specific embodiment, the first actor network includes three fully connected layers and an output layer, one of which is used as an input layer, and the other two fully connected layers are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 1024, and the number of neurons in the second hidden layer is 512; the first critic network is used as a global critic network, and the first critic network includes four fully connected layers and an output layer, one of which is used as an input layer, and the other three fully connected layers are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, the third hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 1024, the number of neurons in the second hidden layer is 512, and the number of neurons in the third hidden layer is 256.
[0039] In this specific embodiment, the second actor network includes three fully connected layers and an output layer, one of the fully connected layers is used as the input layer, and the other two fully connected layers are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 1024, and the number of neurons in the second hidden layer is 512; the second critic network is used as a local critic network, and the second critic network includes three fully connected layers and an output layer, one of the fully connected layers is used as the input layer, and the other two fully connected layers are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 512, and the number of neurons in the second hidden layer is 256.
[0040] In order to further illustrate the feasibility and effectiveness of the method of the present invention, a simulation experiment is carried out on the method of the present invention, and the experimental results are evaluated to verify the performance of the method of the present invention.
[0041] In the simulation experiment, the importance simulation parameters and neural network parameters are set as shown in Table 1.
[0042] Table 1. Important simulation parameters and neural network parameters
[0043]
[0044]
[0045] In the simulation experiment, each episode is set to 60 time steps, that is, T = 60, 5000 episodes are trained, that is, K = 5000, and evaluation is performed every 100 episodes. During the evaluation process, the intelligent vehicle and the MEC server take actions using their respective actor networks.
[0046] In the simulation experiment, two traditional algorithms were selected to compare their performance with the method of the present invention. Traditional Algorithm 1 regards the MEC server as an intelligent agent, trains the neural network corresponding to the MEC server, uses all resources for calculation, does not consider the efficiency of resource use, and the vehicle does not have a corresponding neural network. The vehicle selects the MEC server closest to itself to transmit computing-intensive tasks. Traditional Algorithm 2 regards both the vehicle and the MEC server as intelligent agents, trains their corresponding neural networks, and designs a single reward mechanism.
[0047] Figure 3 The task processing delay of two traditional algorithms and the method of the present invention is shown when the number of intelligent vehicles and the number of MEC servers are in different ratios. Figure 3 It can be seen that when the ratio of the number of smart vehicles to the number of MEC servers is low, the difference in task processing delay between the method of the present invention and the traditional algorithm 1 is small, but both are higher than the traditional algorithm 2. This is because there are enough MEC servers to handle computationally intensive tasks. However, as the ratio of the number of smart vehicles to the number of MEC servers increases, computationally intensive tasks increase. These tasks need to be offloaded to the MEC servers for calculation, which will use more resources and increase the load on the MEC servers. Resources are limited. Since the traditional algorithm 1 uses all resources to calculate the MEC server for training, the task processing delay will be slightly lower than the method of the present invention. Since the traditional algorithm 2 has only one reward setting, the training of the MEC server is not very good, which greatly increases the task processing delay.
[0048] Figure 4 The resource consumption of two traditional algorithms and the proposed method is shown when the number of intelligent vehicles and the number of MEC servers are in different proportions. Figure 4 It can be seen that compared with traditional algorithms 1 and 2, the method of the present invention can achieve higher resource efficiency. The method of the present invention trains the MEC server to allocate resources, which will make the resources of the MEC server more effectively utilized, reduce resource consumption, and thus prevent unnecessary waste of resources. Since traditional algorithm 2 has only one reward setting, it will lead to inadequate training of the MEC server and low resource allocation efficiency. Since traditional algorithm 1 uses all the resources of the MEC server without considering the efficiency of resource use, the resource usage of traditional algorithm 1 is all resources are used.
[0049] Figure 5 The system cost of two traditional algorithms and the proposed method is shown for different ratios of the number of intelligent vehicles to the number of MEC servers. The system cost is the weighted sum of task processing delay and resource usage. Figure 5 It can be seen that as the ratio of the number of smart vehicles to the number of MEC servers increases, the method of the present invention is always better than the traditional algorithm, because the method of the present invention trains smart vehicles and MEC servers at the same time, and uses a dual reward mechanism to enable smart vehicles to learn excellent computing offloading strategies and MEC servers to learn excellent resource allocation strategies, so smart vehicles can find suitable offloading targets, and MEC servers can also reasonably allocate resources to reduce system costs. Traditional algorithm 1 will greatly increase system costs due to the use of all resources, and traditional algorithm 2 will lead to inadequate training of vehicles and MEC servers due to only one reward setting, resulting in system costs that are not as good as the method of the present invention. The increase in the number of smart vehicles also shows the effectiveness of the method of the present invention in high-load scenarios, achieving the best balance between efficient resource utilization and minimizing task processing delays.
[0050] Comprehensive analysis Figure 3 , Figure 4 and Figure 5 ,The performance of the three algorithms under different ratios of the number of intelligent vehicles to the number of MEC servers, the method of the ,invention shows comprehensive advantages in terms of task processing delay, resource consumption and ,system cost.
[0051] 1. Task processing delay analysis ( Figure 3 ):
[0052] When the ratio of the number of smart vehicles to the number of MEC servers is low, that is, when the number of MEC servers is large, the difference in task processing delay between the method of the present invention and traditional algorithm 1 is small, and both are higher than traditional algorithm 2. As the number of smart vehicles increases, the load of the MEC server increases. Traditional algorithm 1 uses all resources to train the MEC server, while the method of the present invention considers the efficiency of resource use. Under the premise that the task processing delay is not too high, it tries to ensure efficient resource use, resulting in a slightly higher task processing delay than traditional algorithm 1. However, because traditional algorithm 2 only sets a single reward mechanism, the training effect is not good, and the MEC server fails to effectively allocate resources, resulting in a significant increase in task processing delay. In contrast, the method of the present invention uses a dual reward mechanism to train smart vehicles and MEC servers at the same time, which can ensure a low task processing delay even under high load conditions, showing good adaptability.
[0053] 2. Resource consumption analysis ( Figure 4 ):
[0054] From the perspective of resource consumption, the method of the present invention shows higher resource efficiency than traditional algorithms 1 and 2. Traditional algorithm 1 does not optimize the resource usage of the MEC server, resulting in the resource utilization rate reaching the upper limit. Although its task processing delay is low, the resource waste is serious. Traditional algorithm 2 has only a single reward mechanism, which cannot effectively guide the reasonable allocation of MEC server resources, resulting in low resource allocation efficiency. The method of the present invention avoids excessive resource consumption and achieves more efficient resource utilization by training MEC servers to perform reasonable resource allocation.
[0055] 3. System cost analysis ( Figure 5 ):
[0056] The system cost is the weighted sum of task processing delay and resource consumption. As the ratio of the number of smart vehicles to the number of MEC servers increases, the system cost gap between the method of the present invention and the traditional algorithm gradually narrows, but the method of the present invention always shows the lowest system cost. Although the traditional algorithm 1 performs well under low load conditions, the system cost increases significantly when the load increases due to the failure to effectively manage resources. Because the traditional algorithm 2 has only a single reward mechanism, the training effect is not good in high-load scenarios, and the system cost remains high. In contrast, the method of the present invention uses a dual reward mechanism and joint training of smart vehicles and MEC servers, so that smart vehicles can select the best unloading target, and MEC servers can also reasonably allocate resources, achieving a better performance balance, especially in high-load scenarios. Keep the system cost low, showing good robustness and efficiency.
[0057] Comprehensive analysis: The method of the present invention is superior to traditional algorithms in terms of task processing delay, resource consumption and system cost. Its dual reward mechanism and joint training strategy enable intelligent vehicles to autonomously select the best unloading target, and the MEC server can efficiently allocate resources and maintain good performance in different load scenarios. In particular, it achieves the best balance between minimizing task processing delay and maximizing resource utilization under high load conditions, which fully verifies the effectiveness and robustness of the method of the present invention.
Claims
1. A computation offloading and resource allocation method based on multi-agent reinforcement learning, characterized in that In the vehicle edge network, multiple smart vehicles and multiple base stations are set, each base station is equipped with a MEC server, the smart vehicles and the MEC server are both intelligent entities, all smart vehicles are driving fast on the road, all base stations are scattered beside the road, the signal of each base station covers part of the road, the signal of all base stations covers the whole road, the base station communicates with the smart vehicles within its signal coverage, and the base stations communicate with each other through optical fibers; for any smart vehicle, a first neural network consisting of a first actor network and a first critic network is constructed; for any MEC server, a second neural network consisting of a second actor network and a second critic network is constructed; the method comprises the following steps: Step 1: Set i to represent the global round number, and set K to represent the global maximum round number, where the initial value of i is 1, i = 1, 2, ..., K; Step 2: Set t to represent the single round time step, and set T to represent the single round maximum time step, where the initial value of t is 1, t = 1, 2, ..., T; Step 3: At the tth time step, the intelligent vehicle that generates computationally intensive tasks is defined as a task vehicle, and the time when the computationally intensive tasks are generated follows a Poisson distribution; each task vehicle senses the environmental state, which includes its own vehicle position and the computationally intensive tasks generated by itself; at the same time, each MEC server also senses the environmental state, which includes the vehicle positions of all task vehicles within the signal coverage of its corresponding base station and the computationally intensive tasks generated by them; Step 4: At the tth time step, for any task vehicle, the environmental state perceived by the task vehicle is input into the first actor network of the corresponding first neural network, the computational offloading decision of the task vehicle is obtained through the action output by the first actor network, the computational offloading decision of the task vehicle is input into the first critic network of the corresponding first neural network, and the quality of the computational offloading decision of the task vehicle is judged by the action output by the first critic network; for any MEC server, the environmental state perceived by the MEC server is input into the second actor network of the corresponding second neural network, the resource allocation strategy of each task vehicle within the signal coverage of the base station corresponding to the MEC server is obtained through the action output by the second actor network, the resource allocation strategy of each task vehicle within the signal coverage of the base station corresponding to the MEC server is input into the second critic network of the corresponding second neural network, and the quality of the resource allocation strategies of these task vehicles is judged by the action output by the second critic network; wherein, the resource allocation strategy includes communication resource allocation and computing resource allocation; Step 5: At the tth time step, for any task vehicle, the task vehicle determines the MEC server to which it wants to offload computation based on its computation offloading decision, and determines the allocated communication resources and computing resources based on its resource allocation strategy; then the task vehicle uses the allocated communication resources to transmit the generated computation-intensive tasks to the determined MEC server, and the MEC server calculates the received computation-intensive tasks based on the allocated computing resources, wherein a transmission delay is generated when transmitting the computation-intensive tasks, and a computing delay is generated when calculating the computation-intensive tasks; then the task processing delay is calculated, and the task processing delay is the sum of the transmission delay and the computing delay; finally, the global reward is calculated based on the task processing delay, and the local reward is calculated based on the resource allocation strategy of the task vehicle; Step 6: Update the environment state; then randomly extract data from the experience pool to train the first neural network corresponding to each task vehicle and the second neural network corresponding to each MEC server, and update the network parameters of the first neural network and the second neural network respectively; Step 7: Determine whether t is less than T. If so, set t=t+1, and then return to step 3 to continue execution, where "=" in t=t+1 is an assignment symbol; otherwise, execute step 8; Step 8: Determine whether i is less than K. If so, set i=i+1 and t=1, and then return to step 2 to continue execution, where the "=" in i=i+1 is an assignment symbol; otherwise, end.
2. According to claim 1, a method for computing offloading and resource allocation based on multi-agent reinforcement learning is characterized in that The first actor network includes three fully connected layers and an output layer, one of which is used as an input layer, and the other two are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 1024, and the number of neurons in the second hidden layer is 512; the first critic network is used as a global critic network. The first critic network includes four fully connected layers and an output layer, one of which is used as an input layer, and the other three are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, the third hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 1024, the number of neurons in the second hidden layer is 512, and the number of neurons in the third hidden layer is 256.
3. A method for computing offloading and resource allocation based on multi-agent reinforcement learning according to claim 1 or 2, characterized in that The second actor network includes three fully connected layers and an output layer, one of the fully connected layers is used as the input layer, and the other two fully connected layers are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 1024, and the number of neurons in the second hidden layer is 512; the second critic network is used as a local critic network. The second critic network includes three fully connected layers and an output layer, one of the fully connected layers is used as the input layer, and the other two fully connected layers are used as hidden layers. The input layer, the first hidden layer, the second hidden layer, and the output layer are connected in sequence. The number of neurons in the first hidden layer is 512, and the number of neurons in the second hidden layer is 256.