A resource allocation method based on MARL in 5G vehicle networking scenario

Abstract

The present invention belongs to the field of intelligent transportation technology, and specifically relates to a resource allocation method based on MARL in a 5G vehicle networking scenario, including: establishing a cellular vehicle networking communication environment model; modeling the vehicle networking allocation problem into a partially observable Markov decision process; treating each V2V link as an intelligent agent for interaction, obtaining sample information and storing it in a sample buffer pool; each intelligent agent uses a multi-head attention mechanism to encode the state characteristics of local observation information collected from the buffer pool, and splices it to obtain a global state; constructing a target value function, and selecting the minimum value to construct the final target value; using the constructed value function to regress the target and learn the corresponding action value function; training the value function according to the final target value to obtain the optimal strategy; interacting the optimal strategy with the environment to allocate resources for the vehicle networking. The present invention increases the estimation capability of the global action value function and improves the resource allocation utilization rate of the system.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and specifically relates to a resource allocation method based on MARL in a 5G vehicle networking scenario. Background Art

[0002] With the rapid popularization of 5G technology, the Internet of Vehicles system, as a core component of intelligent transportation, has ushered in unprecedented development opportunities. However, traditional Internet of Vehicles systems are unable to cope with challenges such as massive data processing, network congestion, and high latency. In order to cope with these challenges, multi-agent reinforcement learning (MARL) technology has emerged. MARL technology optimizes the overall performance of the system through collaborative learning and decision-making among multiple agents. The high speed, low latency, and large connectivity of 5G technology provide a strong basic support for the application of MARL technology. The resource allocation of traditional Internet of Vehicles systems is often based on static rules and is difficult to adapt to dynamic and complex traffic environments. The resource allocation method based on MARL can intelligently allocate network, computing, and storage resources according to real-time traffic conditions, vehicle requirements, and network status, thereby improving the resource utilization and performance stability of the entire system. In addition, the application of MARL technology in intelligent transportation systems can not only reduce traffic congestion and improve traffic fluency, but also improve driving safety and reduce the incidence of traffic accidents. Therefore, MARL resource allocation technology based on 5G Internet of Vehicles has become a hot topic in current research. It can not only promote the performance improvement of the Internet of Vehicles system, but also provide new ideas and methods for the construction of future intelligent transportation systems. In this context, this patent came into being, dedicated to introducing MARL technology in the Internet of Vehicles system, realizing intelligent allocation of resources, and improving the resource allocation efficiency and communication efficiency of the Internet of Vehicles.

[0003] Problems with existing technologies: The mainstream approach to wireless resource management for intelligent connected vehicles is to model the problem from a numerical optimization perspective and then find solutions with specific optimization requirements. However, in Internet of Vehicles applications, the rapidly changing dynamic channels caused by the high-speed movement of vehicles make the optimization results only have a short-term effect. In addition, in the algorithm implementation process, it is difficult to simultaneously consider the different goals of heterogeneous vehicle links, and it is impossible to accurately model the quality of service (QoS). Importantly, the current resource scheduling scheme based on reinforcement learning does not consider the dependencies between multiple vehicles from a global perspective, which weakens the ability to encode the observation information of each intelligent agent. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a resource allocation method based on MARL in a 5G vehicle networking scenario, comprising:

[0005] S1. Establish a cellular vehicle network communication environment model;

[0006] S2, modeling the VIOT allocation problem as a partially observable Markov decision process;

[0007] S3. In the Markov decision process, each V2V link is regarded as an intelligent agent, which interacts in the vehicle networking environment model, obtains sample information and stores it in the sample buffer pool;

[0008] S4, randomly collect multiple local observation information from the sample buffer pool, each agent uses the multi-head attention mechanism to encode the state features of the collected local observation information to obtain the global state information, and splice the global state information obtained by each agent to obtain the global state;

[0009] S5. Based on the global state, construct the target value function based on multiple different perspective states, and select the smallest value to construct the final target value;

[0010] S6. Use the constructed value function to regress the target and learn the corresponding action value function;

[0011] S7, perform value function training according to the constructed final target value, update the strategy parameter θ, and obtain the optimal strategy;

[0012] S8. Use the learned optimal strategy to interact with the environment and allocate resources for the Internet of Vehicles.

[0013] Beneficial effects of the present invention:

[0014] 1. The present invention optimizes the value function based on local observation and global state and then uses it for strategy optimization. During the execution process, the V2V link interacts with the environment through its own local observation, so that the vehicles can maintain a certain degree of autonomy while cooperating. The efficient collaboration between links will also effectively improve the utilization rate of wireless resources.

[0015] 2. The present invention constructs global state features by adaptively fusing multiple local observations using the attention mechanism, and proposes corresponding value function update methods based on global features from different perspectives. While improving the global feature encoding capability of the model, it increases the estimation capability of the global action value function and improves the resource allocation utilization of the system. It has broad application prospects in the resource allocation scenarios of Internet of Vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an overall flow chart of a resource allocation method based on MARL in a 5G vehicle networking scenario of the present invention;

[0017] Figure 2Schematic diagram of partially observable multi-agent Markov decision making constructed in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] A resource allocation method based on MARL in 5G vehicle networking scenarios, such as Figure 1 As shown, including:

[0020] S1. Establish a cellular vehicle network communication environment model;

[0021] S2, modeling the VIOT allocation problem as a partially observable Markov decision process;

[0022] S3. In the Markov decision process, each V2V link is regarded as an intelligent agent, which interacts in the vehicle networking environment model, obtains sample information and stores it in the sample buffer pool;

[0023] S4, randomly collect multiple local observation information from the sample buffer pool, each agent uses the multi-head attention mechanism to encode the state features of the collected local observation information to obtain the global state information, and splice the global state information obtained by each agent to obtain the global state;

[0024] S5. Based on the global state, construct the target value function based on multiple different perspective states, and select the smallest value to construct the final target value;

[0025] S6. Use the constructed value function to regress the target and learn the corresponding action value function;

[0026] S7, perform value function training according to the constructed final target value, update the strategy parameter θ, and obtain the optimal strategy;

[0027] S8. Use the learned optimal strategy to interact with the environment and allocate resources for the Internet of Vehicles.

[0028] In an embodiment of the present invention, a cellular 5G vehicle networking environment is considered, which includes vehicles with V2I and V2V links. The vehicles move along the road, and the distance between the vehicles changes dynamically. The V2I link requires a large link capacity to support high-speed data transmission services, and the V2V link requires high link reliability to transmit safety-related information.

[0029] The set of V2I links is denoted as M = {1, 2, …, M}, and the set of V2V links is denoted as K = {1, 2, …, K}. By adopting an orthogonal spectrum allocation scheme, the V2V link can reuse the channel resources of the V2I link to improve the utilization efficiency of the sub-channel, and allocate M V2I links to the corresponding sub-bands, that is, the m-th V2I link occupies the m-th sub-band. Therefore, the design goal is to design an effective resource allocation scheme for the V2V link and achieve the required performance requirements with minimal signaling overhead. The Orthogonal Frequency Division Multiple Access (OFDMA) technology is used to convert multiple parallel channels of the frequency-selective wireless channel on different sub-carriers. Each sub-carrier is assigned to a spectrum sub-band, and the channel power gain of the k-th V2V link transmitted in the m-th sub-band is g. k [m] = α k h k [m], where the link gain is composed of the frequency-dependent small-scale fading h (path loss) and the frequency-independent large-scale fading factor α k (Shadow Effect) Jointly determined.

[0030] On the mth subband, the signal-to-noise ratio of the mth V2I link is determined by the transmission power, the channel gain of the V2V link, and the interference of the V2V link. Similarly, on the mth subband, the signal-to-noise ratio of the kth V2V link is also affected by the transmission power, the channel gain of the V2V link, and the interference factor. By calculating the signal-to-noise ratio, the capacity of the mth V2I link on the mth subband can be obtained. and the capacity of the kth V2V link The signal-to-noise ratio of the V2I link is The signal-to-noise ratio of the V2V link is The V2I link capacity is The V2V link capacity is Among them, α k represents the large-scale fading factor of the V2V link, h k [m] represents the path loss of the V2V link, Respectively represent the transmission power of the V2I link and the V2V link, represents the interference channel from the mth V2I link to the kth V2V link, σ 2 represents the noise power, ρ k [m] indicates whether the kth V2V link is transmitted on the mth subband. If yes, it is 1, otherwise it is 0. k,B [m] represents the interference channel from the kth V2V link to the base station of the mth subband, g k [m] means I k [m] represents the interference power.

[0031] The design goal of V2I links is to maximize the total capacity To meet the needs of high data rate entertainment services. The design goal of the V2V link is to achieve the transmission of periodic safety information, by establishing rate constraints to ensure that the transmission rate of a packet of size B can be successfully transmitted within a given time, that is:

[0032]

[0033] Where Pr{} represents the probability, B represents the size of the periodic message generated by V2V, and Δ T Denoted as channel coherence time, ρ k [m] is a Boolean spectrum selection variable, which indicates whether the kth V2V link is transmitted on the mth subband. If yes, it is 1, otherwise it is 0. T represents time, M represents the number of V2I links, represents the channel capacity of the V2V link, Δ T represents the channel coherence time, and K represents the number of V2V links.

[0034] B represents the size of the periodic message generated by V2V, Δ T Denoted as channel coherence time, ρ k [m] is a Boolean spectrum selection variable, which indicates whether the k-th V2V link is transmitted on the m-th subband, if yes, it is 1, otherwise it is 0.

[0035] In the embodiment of the present invention, the multi-agent reinforcement learning algorithm QMIX (Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning) is used to model the vehicle resource allocation problem as a partially observable Markov decision process (Decentralized Partially Observable Markov Decision Process, DEC-POMDP). The specific architecture is as follows Figure 2 As shown, multiple distributed intelligent agents interact with the Internet of Vehicles resource allocation model, collect samples into a buffer pool, and then use a centralized training method for value function training based on the attention mechanism model.

[0036] In the resource allocation problem based on multi-agent reinforcement learning, each V2V link acts as an agent. At time step t, the global state S tAll channel states and agent actions need to be considered, but for each agent, this information is hidden, and agent i can only understand the current environment through the observation function. Given the current environment state S t , the observation function O of the V2V link receiving environment, taking action a t , forming a joint action, after which the agent receives the next reward R t+1 And the environment state becomes S t+1 , and each agent will receive a new observation space Z t+1 Therefore, the observation space of agent i includes the channel information G of the current agent i [m](where ), the remaining V2V link payload B i , the remaining delay T i , and the interference power I on subband m i [m]. So the observation function of agent i is: The observation space associated with the channel is

[0037] In the embodiment of the present invention, the QMIX algorithm is used to train a distributed execution mechanism through centralized training. The model consists of two main parts: a decision network for vehicle resource allocation, which uses local observations τ i The action a corresponding to the input and output i ; Strategy evaluation model, which outputs the corresponding value function Q for a given local observation and the corresponding action i (τ i ,a i ). QMIX adopts a joint hybrid network to accept the global Q i (τ i ,a i ) as input and output as joint Q tot (τ,a), and add global state information to assist in the training and learning process to improve algorithm performance:

[0038]

[0039] QMIX introduces a hypernetwork to calculate the parameter weights of the hybrid network: by integrating the value function Q of each agent in a nonlinear way i (τ i ,a i ). At the same time, in order to make the weights greater than zero, a linear network and an absolute value activation function are used to ensure that the output is not negative.

[18] This constraint can make the learned value function meet the monotonicity requirement. The QMIX algorithm implements Q tot (τ,a) and each Q i (τi ,a i ) between:

[0040]

[0041] Then the final loss function of QMIX is:

[0042]

[0043] Where b represents the number of samples sampled from the experience memory, Q tot is the global value function, τ is the local observation, θ represents the parameters of the corresponding value function network, y tot for:

[0044]

[0045] Among them, θ - Represents the parameters of the network corresponding to the objective value function.

[0046] Based on the QMIX algorithm, this paper proposes a QMIX reinforcement learning method (QMIXAtt) based on the attention mechanism. A global state feature encoding model based on the attention mechanism is designed. Given the current multiple local observation information (τ 1 ,τ 2 ...τ n ), use the multi-head attention mechanism model to construct the corresponding state feature encoding. The multi-head attention mechanism can autonomously analyze the dependency information between each local observation and autonomously extract the most important observation information to construct the global state information. The corresponding global state feature encoding is:

[0047]

[0048] Among them, MulAtt represents the multi-head attention mechanism model. Represents the global state encoding result corresponding to the i-th agent.

[0049] In the multi-head attention mechanism, τ i As the query information, τ k As key information, it is randomly sampled from multiple agents, (τ 1 ...τ n ) is the specific result of the splicing of all local observations. Each local observation agent uses the attention mechanism to construct the corresponding new local observation information, and then splices it to construct the corresponding global state. The final global state is expressed as

[0050] This state encoding model based on the attention mechanism can adaptively fuse multiple local observation information from a global observation perspective, and design a relatively stable and robust global state feature to learn the global action value function. In order to further improve the state encoding capability, τ k Two random selections were made, and two different sets of global states were designed from two perspectives, denoted as S and att,1 ,S att,2 Furthermore, the objective function corresponding to the value function of the proposed QMIXAtt algorithm is expressed as:

[0051]

[0052] in Representation based on global state The constructed value function, Representation based on global state The constructed value function, where θ represents the parameters of the corresponding value function network, θ - Represents the parameters of the network corresponding to the objective value function.

[0053] The minimum value function is selected between the value functions constructed in two different view states to construct the final target value for the regression of the value function. The proposed value function update rule can be applied to the parameter update of the attention mechanism at the same time, and can also be used to construct a relatively stable target for the regression of the value function. The value function training objective function of QMIXAtt is as follows:

[0054]

[0055] It involves regression of two value functions, represents the target value estimated from different perspectives, and b represents the number of samples sampled from the experience memory. and Represents the value function estimated at different perspectives. These two value functions are based on the state encoding learned at different perspectives.

[0056] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A resource allocation method based on MARL in a 5G vehicle networking scenario, characterized in that: include: S1. Establish a cellular vehicle network communication environment model; Establish a cellular vehicle network communication environment model, including: The cellular vehicle network communication environment includes vehicles with V2I and V2V links. The set of V2I links is recorded as M = {1, 2, ..., M}, and the set of V2V links is recorded as K = {1, 2, ..., K}. The V2V link can reuse the channel resources of the V2I link. In the V2I link, the Uu interface is used to connect the vehicle and the base station to provide high-speed data transmission services; the V2V link periodically sends safety information through the PC5 interface; An orthogonal spectrum allocation scheme is adopted to allocate M V2I links to corresponding subbands; The frequency selective channel is converted into a form with flat channels on different subcarriers by using OFDM multiple access technology; Each subcarrier is assigned a spectrum subband, and the channel power gain of the kth V2V link transmitted in the mth subband is g k [m] = α k h k [m]; where α k represents the large-scale fading factor of the V2V link, h k [m] represents the path loss of the V2V link; On the mth subband, the signal-to-noise ratio of the mth V2I link is determined by the transmission power, the channel gain of the V2V link, and the interference of the V2V link. The signal-to-noise ratio of the V2I link is The signal-to-noise ratio of the V2V link is The V2I link capacity is The V2V link capacity is in, Respectively represent the transmission power of the V2I link and the V2V link, represents the interference channel from the mth V2I link to the kth V2V link, σ 2 represents the noise power, ρ k [m] indicates whether the kth V2V link is transmitted on the mth subband. If yes, it is 1, otherwise it is 0. k,B [m] represents the interference channel from the kth V2V link to the base station of the mth subband, I k [m] represents the interference power; For the V2V link, a rate constraint is established to ensure that a packet of size B can be successfully transmitted within a given time. For a V2V link, the rate constraints for successfully transmitting packets of size B within a given time include: Where Pr{} represents the probability, B represents the size of the periodic message generated by V2V, and Δ T It is represented as the channel coherence time, T represents time, represents the channel capacity of the V2V link, Δ T represents the channel coherence time, K represents the number of V2V links; S2, modeling the VIOT allocation problem as a partially observable Markov decision process; S3. In the Markov decision process, each V2V link is regarded as an intelligent agent, which interacts in the vehicle networking environment model, obtains sample information and stores it in the sample buffer pool; S4, randomly collect multiple local observation information from the sample buffer pool, each agent uses the multi-head attention mechanism to encode the state features of the collected local observation information to obtain the global state information, and splice the global state information obtained by each agent to obtain the global state; S5. Based on the global state, construct the target value function based on multiple different perspective states, and select the smallest value to construct the final target value; The objective value functions are constructed based on multiple different view states, including: Among them, y tot Indicates that the target value function is constructed based on different viewing angle states. Representation based on global state The constructed value function, Representation based on global state The constructed value function, θ represents the parameters of the corresponding value function network, θ - represents the parameters of the corresponding target value function network, r represents the reward, γ represents the discount factor, τ t+1 、a t+1 Respectively represent the local observation and action of the next state; S6. Use the constructed value function to regress the target and learn the corresponding action value function; Use the constructed value function to regress the target and learn the corresponding action value function, including: Where L(θ) represents the action value function, represents the target value estimated at the view angle i, b represents the number of samples sampled from the experience memory, and represents the value function estimated at different perspectives, τ, a t Represent the local observation and the action at the current moment, respectively. Represent the first and second global states respectively; S7, perform value function training according to the constructed final target value, update the strategy parameter θ, and obtain the optimal strategy; S8. Use the learned optimal strategy to interact with the environment and allocate resources for the Internet of Vehicles.

2. According to claim 1, a resource allocation method based on MARL in a 5G vehicle networking scenario is characterized in that: In the Markov decision process, each V2V link is regarded as an intelligent agent, which interacts in the vehicle network environment model and obtains sample information and stores it in the sample buffer pool, including: At each time step t, given the current environment state S t , the observation function O of the V2V link receiving environment, taking action a t , to form a joint action; the agent performs the joint action and receives the next reward R t+1 And the environment state becomes S t+1 , and each agent will receive a new observation space Z t+1 ; All observation results are stored in the sample buffer pool; the observation results include: current environment state, local observation, joint action, next moment reward, next moment environment state.

3. According to claim 1, a resource allocation method based on MARL in a 5G vehicle networking scenario is characterized in that: Randomly collect multiple local observation information from the sample buffer pool. Each agent uses a multi-head attention mechanism to encode the state features of the collected local observation information to obtain global state information, including: in, Indicates that agent i encodes state features to obtain global state information, MulAtt represents the multi-head attention mechanism model, τ i represents the query information, τ k Represents key information, (τ 1 ...τ n ) represents the n local observation information after splicing.

4. According to claim 1, a resource allocation method based on MARL in a 5G vehicle networking scenario is characterized in that: According to the constructed final target value, the value function is trained, the policy parameter θ is updated, and the optimal policy is obtained, including: According to the action value function, the parameters θ in the network are updated through the gradient direction of the neural network, and the parameters are updated a preset number of times to obtain the optimal strategy.

5. According to claim 1, a resource allocation method based on MARL in a 5G vehicle networking scenario is characterized in that Use the learned optimal strategy and environment interaction to allocate resources for the Internet of Vehicles, including: The communication performance of V2I and V2V links is considered at the same time. According to the obtained optimal strategy, the V2V link selects its own parameters, i.e., the occupied subchannel and the adopted transmission power for resource allocation.

Images