Internet of vehicles multi-hop offloading and resource allocation method based on multi-agent reinforcement learning

Abstract

This invention discloses a multi-hop offloading and resource allocation method for vehicle-to-everything (V2X) networks based on multi-agent reinforcement learning, comprising: S1: The task vehicle observes its local state and obtains a set of candidate service vehicles; S2: A multi-agent collaborative decision-making framework based on deep reinforcement learning is constructed, with the task vehicle acting as an agent. The Actor network generates hybrid actions including service vehicle selection, channel allocation, and power control based on the local state, while the Critic network generates the expected reward value based on the global state and global actions; S3: Gumbel-Softmax is introduced to make discrete actions continuous, and Sigmoid is used to constrain continuous actions; S4: A dynamic agent management mechanism is designed to adapt to dynamic changes in vehicles through an agent pool and mask vector, achieving parameter sharing and ensuring training stability; S5: After training, the task vehicle generates and executes distributed actions through the Actor network based on its local state.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mobile communication, Internet of Vehicles and edge computing, and particularly relates to a multi-hop offloading and resource allocation method for Internet of Vehicles based on multi-agent reinforcement learning. BACKGROUND

[0002] With the deep integration of new generation communication technology and intelligent networked vehicles, the demand for high vehicle-mounted computing capability and real-time processing of emerging applications is growing exponentially. Vehicle edge computing (VEC) forms a distributed edge computing resource pool by deploying roadside units on both sides of the road or using road vehicles with idle computing capability to relieve the computing load of vehicle-mounted terminals and reduce task processing latency. However, in remote road sections or rural areas without cellular network coverage, task vehicles may not be able to directly communicate with edge servers deployed on infrastructure and must rely on other road vehicles as service nodes or relay nodes for multi-hop transmission to offload tasks to remote service nodes. Multi-hop offloading faces the following challenges: the rapid change of network topology caused by high-speed vehicle movement, the short and unstable connection time between vehicles, and how to select reliable service vehicles and transmission paths in a dynamic environment is one of the difficulties; task offloading decisions need to determine service vehicle selection, channel allocation, power allocation, etc., which contain both discrete and continuous variables, forming a mixed integer nonlinear programming problem, and traditional optimization methods are difficult to solve in real time; the number of vehicles in the network changes dynamically over time, making it difficult to directly apply multi-agent reinforcement learning methods based on a fixed number of agents.

[0003] In recent years, deep reinforcement learning has shown great potential in the fields of wireless communication, Internet of Vehicles and edge computing. The multi-agent deep deterministic policy gradient (MADDPG) algorithm has become a powerful tool for solving multi-vehicle collaborative decision-making problems in the Internet of Vehicles scenario due to its centralized training and distributed execution paradigm. However, existing research has failed to effectively handle mixed action spaces, making it difficult to directly adapt to dynamic Internet of Vehicles environments. Therefore, there is an urgent need for a multi-hop offloading and resource allocation method that can adapt to network dynamics, handle mixed action spaces and achieve multi-vehicle collaborative optimization. SUMMARY

[0004] The purpose of the present application is to provide a multi-hop offloading and resource allocation method for Internet of Vehicles based on multi-agent reinforcement learning to overcome the shortcomings of the prior art. Through dynamic agent management, mixed action approximation and centralized training and distributed execution mechanism, the joint optimization of multi-hop offloading and resource allocation in dynamic Internet of Vehicles scenarios is achieved to minimize the average service latency of tasks.

[0005] To achieve the above purpose, the present application provides a multi-hop offloading and resource allocation method for Internet of Vehicles based on multi-agent reinforcement learning, comprising the following steps: S1: Considering that the vehicular network operates on a time-slot system, the network operating period is divided as follows: A discrete time slot, represented as a set In the time slot Inside, there exists A set of task vehicles. , Indicates the first One mission vehicle; have A set of tasks , Indicates the mission vehicle The One task; for Its communication range has A service vehicle is represented as a set. , Indicates the mission vehicle The One service vehicle; Observe local status Local status This includes the status information of the task vehicle itself, task set information, candidate service vehicle status information, and validity mask vector; S2: Construct a collaborative decision-making model based on multi-agent deep deterministic policy gradient (MADDPG), including an actor network. Critic Network Target Actor Network Target Critic Network And the experience replay pool; S3: Introduce the Gumbel-Softmax method to perform continuous approximation processing on discrete action components, and use the Sigmoid function to constrain continuous action components to solve the problem of mixed action space; S4: Design a dynamic agent management mechanism, pre-defining the capacity as... A pool of intelligent agents; in each time slot Inside, the mission vehicle dynamically binds to idle agents in the agent pool, and uses agent mask vectors. The activation state of the agent is marked to ensure a fixed input dimension for the neural network. A one-to-one correspondence is established between the task vehicle and the agent, using a unified [method / approach]. This indicates the corresponding task vehicle and intelligent agent; S5: Employs a centralized training and distributed execution paradigm; during the centralized training phase, the Critic network... Based on global state With global actions Generate expected reward value And define a loss function to update the Critic network. Training parameters Actor Network Based on expected reward value Update the Actor network using gradient descent. Training parameters Use a soft update method, based on the Actor network. Training parameters and Critic Network Training parameters Update the target Actor network Training parameters and target Critic network Training parameters During the distributed execution phase, the task vehicle Based on local status Through the Actor network Generate and execute actions This enables multi-hop unloading and resource allocation decisions.

[0006] Furthermore, in step S1, the mission vehicle Its own state information includes location ,speed Maximum transmission power and local computing power ;Task Information includes the amount of task data. Calculate the demand and maximum tolerable latency Candidate service vehicles Status information includes relative position, relative velocity, transmission power, computing power, and relationship with the mission vehicle. Multi-hop link connection time The validity mask vector includes the task validity mask. Service vehicle validity mask Task validity mask Indicates in time slot Mission vehicles With task ,otherwise Service vehicle validity mask Indicates in time slot Mission vehicles Service vehicles ,otherwise .

[0007] Furthermore, the dynamic agent management mechanism in step S4 includes: all activated agents sharing the Actor network. Training parameters Critic Network Training parameters Target Actor Network Training parameters and the target Critic network Training parameters By using an agent pool and mask vectors to adapt to dynamic changes in the vehicle, parameter sharing is achieved and training stability is ensured.

[0008] Furthermore, in step S5, the action... Includes: Service vehicle selection variables Channel allocation variables and power allocation variables ; Indicates the mission vehicle Uninstall task To service vehicles , Indicates task In the transmission link The first road segment was allocated One channel, Indicates task In the transmission link The power values ​​allocated to each road segment.

[0009] Furthermore, the centralized training process in step S5 includes: S5.1: Random sampling from the experience replay pool A number of empirical samples, among which the sample Includes global state Global Actions Instant rewards Global state of the next time slot and agent mask vector ; S5.2: For the sample Target Actor Network Based on sample information and the global state of the next time slot Generate target action Target Critic Network Based on sample information and target actions Generate the target Q value : , in, Discount factor; S5.3: Minimize the loss function to update the training parameters of the Critic network loss function Represented as: , in, For the sample Medium mission vehicles The corresponding agent mask; S5.4: Update the training parameters of the Actor network using gradient descent. Policy gradient function for: , in, The policy gradient function For Actor networks Training parameters gradient, For Critic Network For action gradient, For Actor Network For Actor networks Training parameters The gradient; S5.5: Update the target Actor network using a soft update method. Training parameters and target Critic network Training parameters : , in, These are soft update coefficients. After training, the task vehicle generates and executes distributed actions through the Actor network based on its local state. This invention reduces the average service latency of the task, improves the task completion rate, and is suitable for edge computing scenarios in the Internet of Vehicles (IoV).

[0010] Furthermore, instant rewards The negative of the average service latency for a task is represented as: , in, For the current time slot Within, the average service latency of all tasks.

[0011] Furthermore, multi-hop link connection time Defined as a mission vehicle With service vehicles The minimum connection time of each single-hop link in a multi-hop path.

[0012] The present invention also discloses a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the steps of a multi-agent reinforcement learning-based method for multi-hop offloading and resource allocation in vehicle networking.

[0013] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of a multi-agent reinforcement learning-based vehicle networking multi-hop offloading and resource allocation method.

[0014] This invention discloses a multi-hop offloading and resource allocation method for vehicular networks based on multi-agent reinforcement learning. By constructing a MADDPG model, it achieves joint optimization of multi-hop offloading and resource allocation. Compared with existing methods, it has the following advantages: 1. Dynamic adaptability: Through the dynamic intelligent agent management mechanism, this invention can flexibly respond to changes in the number of vehicles without redesigning the network structure, thus improving the practicality of the algorithm in real dynamic scenarios.

[0015] 2. Mixed Action Processing: By introducing a method combining Gumbel-Softmax and Sigmoid, the learning of mixed discrete and continuous actions is realized, avoiding the suboptimal nature of hierarchical decision-making.

[0016] 3. Collaborative optimization: Utilizing the centralized training characteristics of MADDPG, the Critic network can evaluate strategies based on global information, promote multi-vehicle collaboration, and effectively reduce task conflicts and resource competition.

[0017] 4. Strong model generalization: The service vehicle state input based on multi-hop link connection time sorting improves the model's adaptability to different network topologies.

[0018] 5. Low-complexity online decision-making: The execution phase only requires local observation and forward propagation, with low computational overhead, making it suitable for real-time operation of vehicle terminals. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a vehicle-mounted network scenario without cellular coverage, as described in a specific embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the overall process of the multi-hop offloading and resource allocation method for vehicle networking based on multi-agent reinforcement learning of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the embodiments of this invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] This embodiment provides a multi-agent reinforcement learning-based multi-hop offloading and resource allocation method for vehicular networks in scenarios without cellular coverage, such as... Figure 1 As shown.

[0023] like Figure 2 As shown, this method specifically includes the following steps: S1: Considering that the vehicular network operates on a time-slot system, the network operating period is divided as follows: A discrete time slot, represented as a set In the time slot Inside, there exists A set of task vehicles. , Indicates the first One mission vehicle; mission vehicle have A set of tasks , Indicates the mission vehicle The One task; for the task vehicle Its communication range has A service vehicle is represented as a set. , Indicates the mission vehicle The Service vehicles; mission vehicles Observe local status Local status This includes the status information of the task vehicle itself, task set information, candidate service vehicle status information, and validity mask vector; mission vehicles Its own state information includes location ,speed Maximum transmission power and local computing power ;Task Information includes the amount of task data. Calculate the demand and maximum tolerable latency Candidate service vehicles Status information includes relative position, relative velocity, transmission power, computing power, and relationship with the mission vehicle. Multi-hop link connection time The validity mask vector includes the task validity mask. Service vehicle validity mask Task validity mask Indicates in time slot Mission vehicles With task ,otherwise Service vehicle validity mask Indicates in time slot Mission vehicles Service vehicles ,otherwise .

[0024] Step S2: Construct a collaborative decision-making model based on multi-agent deep deterministic policy gradient (MADDPG), with the task vehicle acting as an agent and including an Actor network. Critic Network Target Actor Network Target Critic Network And the experience replay pool.

[0025] Step S3: Introduce the Gumbel-Softmax method to perform continuous approximation processing on discrete action components, and use the Sigmoid function to constrain continuous action components to solve the mixed action space problem.

[0026] Step S4: Predefine the capacity size as The pool of intelligent agents, in each time slot Inside, the mission vehicle dynamically binds to idle agents in the agent pool, and uses agent mask vectors. The activation state of the agent is marked to ensure a fixed input dimension for the neural network. A one-to-one correspondence is established between the task vehicle and the agent, using a unified [method / approach]. This represents the corresponding task vehicle and agent. The dynamic agent management mechanism includes: all activated agents sharing the Actor network. Training parameters Critic Network Training parameters Target Actor Network Training parameters and the target Critic network Training parameters By using an agent pool and mask vectors to adapt to dynamic changes in the number of vehicles, parameter sharing is achieved and training stability is ensured.

[0027] Step S5: Employ a centralized training and distributed execution paradigm. During the centralized training phase, the Critic network... Based on global state With global actions Generate expected reward value Define the loss function and update the Critic network. Training parameters Actor Network Based on expected reward value Update the Actor network using gradient descent. Training parameters Using a soft update method, based on the Actor network... Training parameters and Critic Network Training parameters Update the target Actor network Training parameters and target Critic network Training parameters During the distributed execution phase, the task vehicle Based on local status Through the Actor network Generate and execute actions This enables multi-hop unloading and resource allocation decisions. (Action) Specifically, this includes: service vehicle selection variables. Channel allocation variables and power allocation variables . Indicates the mission vehicle Uninstall task To service vehicles , Indicates task In the transmission link The first road segment was allocated One channel, Indicates task In the transmission link The power values ​​allocated to each road segment.

[0028] The intensive training process includes: S5.1: Random sampling from the experience replay pool A number of empirical samples, among which the sample Includes global state Global Actions Instant rewards Global state of the next time slot and agent mask vector : S5.2: For the sample Target Actor Network Based on sample information and the global state of the next time slot Generate target action Target Critic Network Based on sample information and target actions Generate the target Q value : ; in, Discount factor; S5.3: Minimize the loss function to update the training parameters of the Critic network loss function Represented as: ; in, For the sample Medium mission vehicles The corresponding agent mask; S5.4: Update the training parameters of the Actor network using gradient descent. Policy gradient function for: ; in, The policy gradient function For Actor networks Training parameters gradient, For Critic Network For action gradient, For Actor Network For Actor networks Training parameters The gradient; S5.5: Update the target Actor network using a soft update method. Training parameters and target Critic network Training parameters : , in, This is the soft update coefficient.

[0029] After training, the task vehicle generates and executes distributed actions through the Actor network based on its local state. This invention reduces the average service latency of tasks, improves the task completion rate, and is suitable for edge computing scenarios in vehicle-to-everything (V2X) networks.

[0030] Multi-hop link connection time Defined as a mission vehicle With service vehicles The minimum connection time of each single-hop link in a multi-hop path.

[0031] The present invention also discloses a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it implements the steps of a multi-agent reinforcement learning-based method for multi-hop offloading and resource allocation in vehicle networking.

[0032] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of a multi-agent reinforcement learning-based vehicle networking multi-hop offloading and resource allocation method.

[0033] The method of this invention can be deployed in roadside units or vehicle-mounted units within a vehicle-to-everything (V2X) edge computing system. After obtaining the Actor network model through pre-training, online decision-making can be completed with low latency during actual runtime, making it suitable for vehicle task offloading scenarios with high real-time requirements, such as autonomous driving and augmented reality navigation. Furthermore, this invention can also be extended to other dynamic multi-hop network environments such as UAV ad hoc networks and mobile edge computing.

[0034] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for multi-hop offloading and resource allocation in vehicle-to-everything (V2X) networks based on multi-agent reinforcement learning, characterized in that, Includes the following steps: S1: Considering that the vehicular network operates on a time-slot system, the network operating period is divided as follows: A discrete time slot, represented as a set In the time slot Inside, there exists A set of task vehicles. , Indicates the first One mission vehicle; mission vehicles have A set of tasks , Indicates the mission vehicle The One task; for the task vehicle Its communication range has A service vehicle is represented as a set. , Indicates the mission vehicle The Service vehicles; mission vehicles Observe local status The local state This includes the status information of the task vehicle itself, task set information, candidate service vehicle status information, and validity mask vector; S2: Construct a collaborative decision-making model based on multi-agent deep deterministic policy gradient (MADDPG), including an actor network. Critic Network Target Actor Network Target Critic Network And the experience replay pool; S3: Introduce the Gumbel-Softmax method to perform continuous approximation processing on discrete action components, and use the Sigmoid function to constrain continuous action components to solve the problem of mixed action space; S4: Design a dynamic agent management mechanism, pre-defining the capacity as... A pool of intelligent agents; in each time slot Inside, the mission vehicle dynamically binds to idle agents in the agent pool, and uses agent mask vectors. Mark the activation state of the agent to ensure that the input dimension of the neural network is fixed; S5: Employs a centralized training and distributed execution paradigm; during the centralized training phase, the Critic network... Based on global state With global actions Generate expected reward value And define a loss function to update the Critic network. Training parameters Actor Network Based on expected reward value Update the Actor network using gradient descent. Training parameters Use a soft update method, based on the Actor network. Training parameters and Critic Network Training parameters Update the target Actor network Training parameters and target Critic network Training parameters ; During the distributed execution phase, the task vehicle Based on local status Through the Actor network Generate and execute actions This enables multi-hop unloading and resource allocation decisions.

2. The method for multi-hop offloading and resource allocation in vehicle networking based on multi-agent reinforcement learning according to claim 1, characterized in that, In step S1, the task vehicle Its own state information includes location ,speed Maximum transmission power and local computing power ;Task Information includes the amount of task data. Calculate the demand and maximum tolerable latency Candidate service vehicles Status information includes relative position, relative velocity, transmission power, computing power, and relationship with the mission vehicle. Multi-hop link connection time ; The validity mask vector includes the task validity mask. Service vehicle validity mask Task validity mask Indicates in time slot Mission vehicles With task ,otherwise Service vehicle validity mask Indicates in time slot Mission vehicles Service vehicles ,otherwise .

3. The method for multi-hop offloading and resource allocation in vehicle networking based on multi-agent reinforcement learning according to claim 2, characterized in that, The dynamic agent management mechanism in step S4 includes: all activated agents sharing an Actor network. Training parameters Critic Network Training parameters Target Actor Network Training parameters and the target Critic network Training parameters .

4. The method for multi-hop offloading and resource allocation in vehicle networking based on multi-agent reinforcement learning according to claim 3, characterized in that, In step S5, the action Includes: Service vehicle selection variables Channel allocation variables and power allocation variables ; Indicates the mission vehicle Uninstall task To service vehicles , Indicates task In the transmission link The first road segment was allocated One channel, Indicates task In the transmission link The power values ​​allocated to each road segment.

5. The method for multi-hop offloading and resource allocation in vehicle networking based on multi-agent reinforcement learning according to claim 4, characterized in that, The centralized training process in step S5 includes: S5.1: Random sampling from the experience replay pool A number of empirical samples, among which the sample Includes global state Global Actions Instant rewards Global state of the next time slot and agent mask vector ; S5.2: For the sample Target Actor Network Based on sample information and the global state of the next time slot Generate target action Target Critic Network Based on sample information and target actions Generate the target Q value : , in, Discount factor; S5.3: Minimize the loss function to update the training parameters of the Critic network loss function Represented as: ； in, For the sample Medium mission vehicles The corresponding agent mask; S5.4: Update the training parameters of the Actor network using gradient descent. Policy gradient function for: ； in, The policy gradient function For Actor networks Training parameters gradient, For Critic Network For action gradient, For Actor Network For Actor networks Training parameters The gradient; S5.5: Update the target Actor network using a soft update method. Training parameters and target Critic network Training parameters : , in, This is the soft update coefficient.

6. The method for multi-hop offloading and resource allocation in vehicle networking based on multi-agent reinforcement learning according to claim 5, characterized in that, The instant reward The negative of the average service latency for a task is represented as: , in, For the current time slot Within, the average service latency of all tasks.

7. The method for multi-hop offloading and resource allocation in vehicle networking based on multi-agent reinforcement learning according to claim 6, characterized in that, The multi-hop link connection time Defined as a mission vehicle With service vehicles The minimum connection time of each single-hop link in a multi-hop path.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-agent reinforcement learning-based multi-hop offloading and resource allocation method for vehicle networking as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multi-agent reinforcement learning-based multi-hop offloading and resource allocation method for vehicle networking as described in any one of claims 1 to 7.

Images