A resource optimization method for Internet of Vehicles based on composite priority experience replay sampling

Abstract

The present invention provides a vehicle network resource optimization method based on composite priority experience replay sampling, including: step 1, problem modeling and goal definition; step 2, designing state space and action space; step 3, designing reward function to guide the agent to obtain maximum return to select the optimal strategy; step 4, establishing a resource allocation optimization algorithm based on multi-agent deep deterministic gradient strategy MADDPG to improve the load transmission success rate of the vehicle-to-vehicle V2V communication link and maximize the total capacity of the vehicle-to-infrastructure V2I communication link; step 5, establishing a multi-agent deep deterministic gradient strategy algorithm based on composite priority experience replay sampling to improve the convergence and stability of the MADDPG algorithm and further optimize the power control strategy. The simulation results show that the present invention has better performance in terms of the load transmission success rate of the V2V communication link and the total capacity of the V2I communication link.

Description

Technical Field

[0001] The present invention belongs to the field of Internet of Vehicles, and in particular relates to an Internet of Vehicles resource optimization method based on composite priority experience playback sampling. Background Art

[0002] With the rapid development of 5G technology and the rapid increase in the number of vehicles in my country, the outstanding problems in modern transportation still need to be solved. Vehicle to Everything (V2X) brings people a more intelligent and convenient travel experience. However, there are still some problems in the vehicle to everything system, such as performance optimization in path planning, energy management, vehicle control and vehicle sharing, which need further research and exploration. For example, Xie Renchao et al. proposed a review of mobile edge computing offloading technology (Reference: Xie Renchao, Lian Xiaofei, Jia Qingmin et al. Review of Mobile Edge Computing Offloading Technology [J]. Journal of Communications, 2018, 39(11): 138-155), and conducted an in-depth analysis of the current development status of the technology. This review introduces the application of mobile edge computing in the vehicle to everything system in detail, including its advantages and challenges.

[0003] The Internet of Vehicles (IoV) technology originated from the Internet of Things (IoT), and its main goal is to reduce traffic congestion, enhance drivers' safety awareness, and improve passengers' travel comfort. The traditional definition of this technology is that vehicles use on-board electronic devices or electronic tags, and use advanced communication technologies such as RFID to identify vehicles and simply extract dynamic and static information (reference: Jing Li. Research on spectrum allocation and AoI of IoV based on reinforcement learning [D]. Chongqing: Chongqing University, 2020). This method not only allows drivers to grasp real-time road conditions, but also provides efficient and high-quality communication services for vehicles. With the widespread application of 5G technology, the continuous advancement of related technologies, and the continuous updating and iteration of on-board equipment, the definition of IoV has become increasingly rich, and the traditional definition can no longer fully meet the needs of modern society. IoV mainly consists of two types. The first is the communication between vehicle-to-infrastructure (V2I), which enables vehicles to communicate with roadside facilities (such as electronic road signs, signal base stations, traffic lights, etc.). The communicating parties obtain current real-time road conditions, danger warnings and other information through data transmission. The second is the vehicle-to-vehicle (V2V) communication method (reference: Li Hongyang. Simulation and research on resource allocation algorithm based on LTE vehicle network [D]. Beijing Jiaotong University, 2018). This communication method allows vehicles on the road to communicate with each other through self-organizing networks. By combining the above two communication methods, the application of vehicle network technology enables real-time information exchange between vehicles, obtains traffic information beyond the human perspective, improves the vehicle's ability to perceive the environment, and thus reduces the probability of traffic accidents. In addition, the vehicle network also provides an effective way to access information resources in the cloud (reference: Song Bo, Ye Wei, Meng Xianghui. A review of dynamic spectrum allocation methods based on multi-agent reinforcement learning [J]. Systems Engineering and Electronics, 2021, 43(11): 3338-3351), such as real-time traffic reports, sensor data, and map information. This information can not only improve the current driving experience and driving safety, but also play a key role in the development of future automotive autonomous driving technology. To a certain extent, the emergence of vehicle networking technology has redefined the way modern urban transportation operates, and plays an indispensable role in improving urban transportation efficiency and road safety.

[0004] Due to the high complexity of modern urban traffic, the Internet of Vehicles system also has corresponding characteristics: First, as the vehicle is the basic unit, it is very easy to cause changes in network topology during high-speed driving, which reduces the communication quality; second, the Internet of Vehicles communication system is easily affected by roads, traffic signals, signs and emergencies, resulting in changes in the vehicle driving environment. Therefore, Internet of Vehicles applications can be divided into two categories: safety and non-safety: safety applications are mainly used to prevent accidents and ensure driving safety, such as hazard warnings and collision detection; while non-safety applications are mainly used to improve traffic efficiency, such as traffic congestion relief and audio-visual entertainment. These two types of applications have different requirements for the timeliness of information transmission. For example, safety applications have more precise requirements for the real-time nature of data interaction. Therefore, the Internet of Vehicles system needs to ensure ultra-reliable low latency for real-time information interaction and reasonably allocate spectrum resources to ensure the stability and reliability of the system. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a vehicle networking resource optimization method based on composite priority experience replay sampling in view of the shortcomings of the prior art. By modeling the vehicle networking system and using the reinforcement learning algorithm, the purpose of optimal power control is achieved. Specifically, the following steps are included:

[0006] Step 1: Problem modeling and goal definition;

[0007] Step 2: Design state space and action space;

[0008] Step 3: Design a reward function to guide the agent to obtain the maximum reward and select the optimal strategy;

[0009] Step 4: Establish a resource allocation optimization algorithm based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG) to improve the load transmission success rate of the vehicle-to-vehicle V2V communication link and maximize the total capacity of the vehicle-to-infrastructure V2I communication link;

[0010] Step 5: Establish a Composite Priority Multi-Agent Deep Deterministic Policy Gradient (Cop-MADDPG) algorithm based on composite priority experience replay sampling to improve the convergence and stability of the MADDPG algorithm and further optimize the power control strategy.

[0011] Step 1 includes: the vehicle network based on the cellular network has a vehicle-to-vehicle V2V communication link and a vehicle-to-infrastructure V2I communication link; the vehicle-to-infrastructure V2I communication link connects the vehicle to the base station through a cellular interface, and when all vehicles have only one antenna, the sets of the vehicle-to-vehicle V2V communication link and the vehicle-to-infrastructure V2I communication link are represented as K={0,…,K} and M={0,…,M}, respectively, where K and M represent the number of vehicle-to-vehicle V2V communication links and the number of vehicle-to-infrastructure V2I communication links, respectively.

[0012] Step 1 also includes: in the time period X1 (X1 is set to 30ms in the present invention), the power gain of the k-th vehicle-to-vehicle V2V communication link channel in the m-th subband is described as:

[0013] g k [m] = a k h k [m] (1)

[0014] Among them, h k [m] is the power amount of small-scale fading; a k It represents a large-scale attenuation that is independent of the frequency components.

[0015] Step 1 also includes: setting g k′,k [M] represents the interference channel from the transmitter of the inter-vehicle V2V communication link No. k′ to the inter-vehicle V2V communication link No. k using the sub-band No. m. The channel gain of the sub-band No. m from the transmitter of the inter-vehicle V2V communication link No. k to the base station is expressed as g m,B [m], the power of the transmission power signal from the transmitter of V2I link number m on subband number m is expressed as g m,k [m], let represents the signal power of the transmitting end of the V2I communication link from vehicle m to the infrastructure, and the transmission power of the transmitting end of the V2V communication link between vehicle k on subband m is σ 2 Represents the noise power, then the ratio of the signal to the interference noise that can be received on the m-vehicle-to-infrastructure V2I communication link on the m-subband, that is, the expression of the signal-to-noise ratio is:

[0016]

[0017] The expression of the signal-to-noise ratio that can be received on the k-th vehicle-to-vehicle V2V communication link on the m-th sub-band is:

[0018]

[0019] Among them, I k [m] is the magnitude of the interference power; ρ k[m] represents a spectrum selection variable, which means: if the k-th vehicle-to-vehicle V2V communication link is transmitted on the m-th subband, ρ k [m] = 1, otherwise 0;

[0020] If all inter-vehicle V2V communication links can only be connected to the same orthogonal sub-band, W is the bandwidth of each spectrum. According to Shannon's formula, the capacity of the m-vehicle to infrastructure V2I communication link on the m-sub-band is The expression is:

[0021]

[0022] Get the channel capacity of the V2V communication link between vehicles k on subband m The expression is:

[0023]

[0024] Assume B is the size of the V2V payload generated in a certain period, Δ T represents the time of the coherent channel, t represents the index of the coherent time, represents the instantaneous capacity and of all V2V links, then the expression of the transmission rate P of a data packet with a capacity of B within a specified time T is:

[0025]

[0026] The optimal resource allocation problem is expressed as follows: If for all k∈K, m∈M, the transmission power of the V2V communication link between vehicles is a continuous and variable quantity and maximize the total capacity of the vehicle-to-infrastructure V2I communication link and transfer rate.

[0027] Step 2 includes: for the state space, when the time step is t, set the environment state of agent k in advance Assume that the state space of two or more agents is S t , when the agents select their own actions, they form a joint action A t ; When all agents perform actions at the same time, the reward they receive is R t , the state of agent k at this time Probability of passing Go to the next state For a single agent k, get the state of its local environment With action And it cannot obtain the conditions of the overall channel and the action information taken by other agents; the local channel information that a single agent k can obtain includes its own channel gain gk [m], and the interference channel at the transmitting end of the V2V communication link between other vehicles is g k,k′ [m], where all m∈M, the interference channel from the sender of agent k to the base station is g k,B [m], and the interference channel g at the transmitting end of all vehicle-to-infrastructure V2I communication links m,k [m]; The state that agent k can obtain is:

[0028]

[0029] For the action space, the resource allocation problem is mapped into a continuous control problem of the transmission power of the V2V communication link between vehicles. The transmission power of the transmitting end of the V2V communication link between vehicles ranges from is the maximum value of the transmitting power of the transmitting end of the V2V communication link between vehicles, and the action strategy expression of a single agent k is:

[0030]

[0031] Step 3 includes: expressing the instantaneous capacity and the capacity of the vehicle-to-infrastructure V2I communication link as The transmission rate of the inter-vehicle V2V communication link is set to the reward value L k , after the payload is delivered, the reward is set to a constant β, when the cumulative reward L for the successful transmission of the payload of the inter-vehicle V2V communication link within time step t k (y) is expressed as:

[0032]

[0033] Reward function R t The expression is:

[0034]

[0035] Where μ is the weight, 0<μ<1;

[0036] The interference channel from the transmitter of the interference link agent to the base station is represented as g m , k [m], the negative reward factor is ε, 0<ε<1, then the expression of the reward function is:

[0037]

[0038] Step 4 includes: the resource allocation optimization algorithm based on the multi-agent deep deterministic gradient strategy MADDPG includes an Actor network and a Critic network. The Actor network continuously collects environmental states and then performs corresponding actions according to the strategy. The Critic network uses the strategy Qk (*) to evaluate the quality of the actions performed by the Actor network; let the input state of agent k be Instant Rewards The discount factor is γ, then the state and action value function Q of the DDPG algorithm k (S k , A k ) is:

[0039] Q k (S k , A k )=E[R k +γQ(S k′ ,A k′ )] (13)

[0040] First, agent k will get an optimal strategy And through continuous learning and this strategy The corresponding state and action value (State-Action) function until convergence;

[0041] The MADDPG algorithm adopts a dual network structure, including an evaluation network and a target network. The evaluation network includes an Actor evaluation network and a Critic evaluation network. The parameters of the Actor evaluation network are The parameters of the Critic evaluation network are The target network includes the Actor target network and the Critic target network. The parameters of the Actor target network are The parameters of the Critic target network are The reward for the kth agent is Then the loss function The expression is:

[0042]

[0043] Among them, Q′ k (*) is the action and state value function of the target network; E is the mathematical expectation; and are all conditional probabilities;

[0044] When the actors make corresponding actions through continuous interaction, and all agents are seeking to maximize the cumulative reward, the parameters of the actors are changed by maximizing the strategy objective function. The expression is:

[0045]

[0046] Among them, μ(*) represents a deterministic strategy π for state mapping actionk Actor evaluation network function;

[0047] Use a constant ω close to 0 to change the parameters of the target network, the expression is:

[0048]

[0049] Step 4 also includes: The resource allocation optimization algorithm based on the multi-agent deep deterministic gradient strategy MADDPG includes centralized learning and distributed execution: each group of evaluation networks and target networks are regarded as a separate agent. In the centralized training process, agent k not only needs to obtain state information from the local state and the action currently in progress You also need to get the status of other agents and action information Status Information The current action The status of other agents and action information It is stored in the experience pool of the kth agent for subsequent centralized training and evaluation of the network.

[0050] Step 5 includes: introducing a composite priority, wherein the composite priority includes a priority based on a time difference error TD-error and a priority based on a real-time report; and a sampling method of composite priority experience playback includes the following steps:

[0051] Step 5-1, use the Q value to calculate the time difference error TD-error, Q value is the state and action value function Q k (S k , A k );

[0052] Step 5-2, use formula (18) to define the priority order:

[0053]

[0054] Among them, r t Indicates the real-time return of experience, represents a constant, ω t Indicates the time difference error; Y i is the priority based on immediate return, Y f is the priority based on TD-error;

[0055] Step 5-3: Sort the experience in the experience pool according to the priority obtained in step 5-2 to obtain the sequence rank f With rank i , where ranki For high priority, rank f is a low priority, and the composite priority u of all experiences is calculated using formula (19): k :

[0056]

[0057] where Y k Indicates the degree of composite priority, α represents the probability weight, and when α = 0, it represents uniform sampling;

[0058] Step 5-4, let n be the number of experiences, then the probability P of the sth sample experience being sampled is s The expression is:

[0059]

[0060] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0061] Since the MADDPG algorithm randomly takes sample experience to train the neural network, this method leads to poor training effect, low algorithm convergence, and easy formation of local optimal solutions. The present invention improves the selection of samples in the experience pool and proposes a MADDPG algorithm (Cop-MADDDPG) based on a composite priority experience replay sampling method. The feasibility of the algorithm used and the adaptability of the algorithm in environments with different vehicle speeds are verified through experimental simulation. At the same time, the proposed algorithm is compared and analyzed with the MADDPG algorithm and the baseline algorithm. The simulation results show that the proposed improved algorithm has better performance in terms of the success rate of load transmission of the vehicle-to-vehicle V2V communication link and the improvement of the total capacity of the vehicle-to-infrastructure V2I communication link.

[0062] Beneficial effect: The Cop-MADDPG algorithm proposed in the present invention introduces the concept of composite priority in the random sampling process of the MADDPG algorithm, sets the priority of the experience tuple, and the probability of the experience tuple in the experience pool being sampled is proportional to the composite priority of the sample. The larger the composite priority, the greater the probability of being sampled. In layman's terms, more suitable samples are preferentially selected for training the neural network. The feasibility analysis and adaptability analysis of the designed simulation experiments prove that the Cop-MADDPG algorithm has better performance than the MADDPG algorithm in terms of convergence and stability, as well as in the vehicle networking environment with different vehicle speeds. The simulation results show that the proposed improved algorithm has better performance in terms of V2V link load transmission success rate and V2I link total capacity improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0064] Figure 1 This is a schematic diagram of the vehicle networking system.

[0065] Figure 2 This is the MADDPG algorithm framework diagram.

[0066] Figure 3 This is the Cop-MADDPG algorithm framework diagram.

[0067] Figure 4 It is a graph of the convergence verification results of the method of the present invention.

[0068] Figure 5a and Figure 5b This is a schematic diagram showing the algorithm's adaptability to different vehicle speeds.

[0069] Figure 6 Schematic diagram of the total capacity of each vehicle-to-infrastructure V2I communication link under different payload conditions.

[0070] Figure 7 It is a schematic diagram of the probability of successful load transmission in the V2V communication link between vehicles under different effective loads.

[0071] Figure 8a , Figure 8b , Figure 8c , Figure 8d This is a schematic diagram of the impact of different algorithms on the instantaneous power of the V2V communication link between vehicles. DETAILED DESCRIPTION

[0072] The present invention provides a vehicle network resource optimization method based on composite priority experience playback sampling, comprising the following steps:

[0073] Step 1: Problem modeling and goal definition;

[0074] like Figure 1As shown, the cellular network-based vehicle network has vehicle-to-vehicle V2V communication links and vehicle-to-infrastructure V2I communication links. These links can provide high-speed data transmission for high-speed vehicles to support vehicle-mounted applications. The vehicle-to-infrastructure V2I communication link connects the vehicle to the base station through a cellular interface to provide high-quality communication services. When all vehicles have only one antenna, the sets of vehicle-to-vehicle V2V communication links and vehicle-to-infrastructure V2I communication links are represented as K={0,…,K} and M={0,…,M}, respectively, where K represents the number of V2V links and M represents the number of V2I links. The present invention adopts a vehicle-to-vehicle V2V communication link transmission power algorithm based on continuous action to maximize the total load capacity of the vehicle-to-infrastructure V2I communication link and improve the probability of successful transmission of the vehicle-to-vehicle V2V communication link in a high-speed vehicle environment.

[0075] Orthogonal frequency division multiplexing (OFDM) is widely used in the field of Internet of Vehicles. The core idea of ​​this technology is to convert the selected channel in the frequency domain into parallel and relatively stable channels on multiple subcarriers, and concentrate these continuous subcarriers on a specific spectrum subband. When the channel fades to a similar position and these subbands are independent of each other, h k [m] is the power of small-scale fading, and it is assumed to be exponentially distributed, a k represents a large-scale attenuation that is independent of the frequency component. Therefore, in a specific time period, the channel power gain of the k-th vehicle-to-vehicle V2V communication link in the m-th subband is described as:

[0076] g k [m] = h k [m] (1)

[0077] where g k′,k [m] represents the interference channel from the transmitter of the V2V communication link between vehicles k to the V2V communication link between vehicles k using sub-band m. The channel gain of sub-band m from the transmitter of the V2V communication link between vehicles k to the base station is expressed as g m,k [m].

[0078] make represents the signal power of the transmitting end of the V2I communication link from vehicle m to the infrastructure, and the transmission power of the transmitting end of the V2V communication link between vehicle k on subband m is σ 2 Represents the noise power, then the ratio of the signal to the interference noise that can be received on the m-vehicle-to-infrastructure V2I communication link on the m-subband, that is, the expression of the signal-to-noise ratio is:

[0079]

[0080] The expression of the signal-to-noise ratio that can be received on the k-th vehicle-to-vehicle V2V communication link on the m-th sub-band is:

[0081]

[0082] Among them, I k [m] is the magnitude of the interference power; ρ k [m] represents a spectrum selection variable, which means whether V2V k is transmitted on subband m. If yes, then ρ k [m]=1, otherwise 0.

[0083] If all inter-vehicle V2V communication links can only be connected to the same orthogonal sub-band, the capacity of the m-vehicle to infrastructure V2I communication link on the m-sub-band can be expressed as follows according to the Shannon formula:

[0084]

[0085] Similarly, the expression of the channel capacity of the V2V communication link between vehicles k on sub-band m is obtained as follows:

[0086]

[0087] As mentioned above, in the Internet of Vehicles system, vehicle-to-infrastructure V2I communication links are often used to support high-speed in-vehicle services. Therefore, it is necessary to design an appropriate goal to maximize the capacity of the vehicle-to-infrastructure V2I communication link, which is crucial for the improvement of the in-vehicle service network. At the same time, the privacy data generated in real time needs to be guaranteed by the vehicle-to-vehicle V2V communication link, but the frequency period of data generated by different vehicles is not the same. Assume that B is the size of the V2V payload generated in a certain period, in bits, Δ T represents the time of the coherent channel, t represents the coherent time check, then within the specified time T, the expression of the transmission rate of the data packet with capacity B is:

[0088]

[0089] The problem of optimizing resource allocation proposed by the present invention can be expressed as follows: if for all k∈K, m∈M, the transmission power of the V2V communication link between vehicles is a continuous and variable quantity and maximize the total capacity of the vehicle-to-infrastructure V2I communication link and transmission rate. Since it is difficult for the central controller of a vehicle to accurately collect all channel states when it is traveling at high speeds, the use of distributed V2V resource allocation can more ideally solve this problem. In addition, since the complexity of the data will affect the sequential decision of multiple coherent time slots within the specified time T in the above formula, traditional optimization methods are no longer applicable to such problems. Therefore, the present invention transforms the above problem into a Markov decision process, and uses a distributed algorithm based on multi-agent reinforcement learning to optimize the transmission power control of the V2V communication link between vehicles.

[0090] During high-speed driving, the problem of how to choose different transmission powers for each vehicle's inter-vehicle V2V communication link can be transformed into a multi-agent reinforcement learning challenge. If each vehicle is regarded as an independent agent, these agents will continuously interact with the real-time environment to accumulate experience and determine the best strategy through these experience learning. When there are multiple agents in an environment, these agents need to continuously adjust their power control strategies to adapt to the ever-changing environment. In addition, by adopting a unified reward function, the interaction between multiple agents can be transformed into a collaborative mode, thereby optimizing the overall performance of the neural network.

[0091] Step 2: Design state space and action space;

[0092] For the design of state space, when the time step is t, the environment state of agent k is set in advance Select action A according to different strategies t . Assume that the state space of two or more agents is S t , when the agents select their own actions, they form a joint action A t The reward R obtained when all agents perform the action at the same time t , the state of the agent at this time Probability of passing Go to the next state For a single agent, the agent can obtain the state of its local environment With action And it cannot obtain the overall channel conditions and the action information taken by other agents. The local channel information that a single agent k can obtain mainly includes its own channel gain g k [m], and the interference channel of the transmitting end of the V2V communication link between other vehicles is g k,k′ [m], where all m∈M, the interference channel from the agent’s own transmitter to the base station is g k,B [m], and the interference channel g of all V2I transmitters m,k[m]. In summary, the states that a single agent k can obtain are:

[0093]

[0094] For the design of action space, the present invention maps the resource allocation problem into a continuous control problem of the transmission power of the V2V communication link between vehicles. The transmission power of the transmitting end of the V2V communication link between vehicles ranges from The action strategy expression of a single agent k is:

[0095]

[0096] Step 3: Design reward function;

[0097] The environment that vehicles face during high-speed driving is complex and changeable, and reinforcement learning can be used to deal with difficult-to-optimize target problems in a more complex environment. Since the reward value that can be obtained after each action made by the intelligent agent is related to the expected goal, it is particularly important to design a reasonable reward function. In view of the power control problem mentioned above, the goal of the present invention is to maximize the total capacity of the vehicle-to-infrastructure V2I communication link and to increase the probability of successful transmission of the vehicle-to-vehicle V2V communication link within a specified time. The instantaneous capacity and value of the vehicle-to-infrastructure V2I communication link are expressed as The transmission rate of the inter-vehicle V2V communication link is set to the reward value L k , after the payload is delivered, the reward is set to a constant β. As a result, the cumulative reward for successful transmission of the payload of the inter-vehicle V2V communication link within time step t is expressed as:

[0098]

[0099] Among them, before the effective load in the V2V communication link between vehicles is transported, the reward value can be set to 0, and the reward value after completion can be set to 1. However, in actual training, when the reward obtained by the agent is 0, it will cause the agent to fail to learn effective experience. Therefore, it is necessary to set a weight μ, 0<μ<1, which ensures that the final optimization result is closer to the probability of successful transmission of the load of the V2V communication link between vehicles. In order to ensure both the total capacity of V2I and the probability of successful transmission of the load of the V2V communication link between vehicles, the present invention chooses to compromise the expression of the reward function as follows:

[0100]

[0101] In the actual Internet of Vehicles environment, it is impossible for the agent to only receive positive rewards. When it interacts with the environment, it is often affected by interference channels. The interference channel from the transmitter of the interference link agent to the base station is represented as g m , k [m], the negative reward factor is ε, 0<ε<1, and the designed reward function will be closer to the actual Internet of Vehicles environment. The expression of the reward function used in the present invention is:

[0102]

[0103] Step 4: Establish a resource allocation optimization algorithm based on MADDPG;

[0104] For the policy gradient algorithm, as the number of agents continues to increase, the environment becomes more complex and changeable, which will affect the variance of the optimization method of estimating the gradient through sampling and rapidly increase.

[0105] In this invention, a deep deterministic policy gradient algorithm is used to solve the above problems. The algorithm uses a deep Q network to solve the problems of large dimensions and complex calculations of multi-agent systems. It contains two different types of networks. The Actor network continuously collects environmental states and then performs corresponding actions according to the strategy. The Critic network uses the strategy Q k (*) to evaluate the quality of the actions performed by the Actor network. Let the input state of agent k be Instant Rewards The discount factor is γ, then the expression of the state-action value function of the DDPG algorithm is:

[0106] Q k (S k , A k )=E[R k +γQ(S k′ ,A k′ )] (13)

[0107] The basic idea of ​​the algorithm is that the agent first obtains an optimal strategy And by continuously learning the State-Action function corresponding to the strategy until convergence. The Actor-Critic network of DDPG is usually a dual network structure: the evaluation network (the parameters of the Actor evaluation network are The parameters of the Critic evaluation network are ), target network (the parameters of the Actor target network are The parameters of the Critic target network are ). The parameters of the evaluation network are It is constantly changing. First, a small part of the experience needs to be randomly taken from the experience pool as a sample, and then the intelligent experience will learn this part of the sample experience. During the training process, the Critic and Actor networks can change the network parameters in real time according to the learned samples. The expression of the loss function is:

[0108]

[0109] Among them, Q′ k (*) is the action-state value function of the target network. When the Actor makes corresponding actions through continuous interaction, and all agents are seeking to maximize the cumulative reward, the parameters of the Actor are changed by maximizing the strategy objective function. The objective function expression is:

[0110]

[0111] Among them, μ(*) represents a deterministic strategy π for state mapping action k Actor evaluation network function. Because the system is a continuous action space, It will affect the direction of gradient descent. As the parameters of the Critic network change, the parameters of the target network also need to be continuously updated. Here, a constant ω close to 0 is used to change the parameters of the target network. The expression is:

[0112]

[0113] The DDPG algorithm is suitable for single-agent environments, while the MADDPG algorithm is suitable for multi-agent environments, and the agent can not only obtain local information, but also obtain information from other agents. The framework of MADDPG is as follows Figure 2 shown.

[0114] from Figure 2 As can be seen from the figure, the MADDPG algorithm can be divided into two core parts: centralized learning and distributed execution. Each group of Actor and Critic networks can be regarded as a separate agent. In the centralized training process, agent k not only needs to obtain state information from the local state, but also needs to obtain state information from the local state. and the actions currently in progress You also need to get the status of other agents and action information This information needs to be stored in the agent's experience pool to facilitate the subsequent centralized training of the Critic network, where S t represents the joint state of centralized training, A trepresents a joint action, so that there is no mutual interference between all agents in the entire model. When the agents can obtain this additional information, all agents have the ability to learn their state-action value function. In this case, if the agent is in distributed execution mode, the Actor network only needs to observe its local area, so that each agent can accurately grasp its behavior pattern without knowing the environment information of other agents.

[0115] Step 5: Establish the Cop-MADDPG algorithm

[0116] Because the experience replay mechanism adopted by the MADDPG algorithm adopts a method of mechanically randomly sampling the experience in the experience pool in the past period of time to improve the stability of the neural network. Although this mechanism can improve the efficiency of learning, the quality of the experience tuple obtained by sampling cannot be guaranteed due to the randomness of sampling, thereby affecting the efficiency of neural network training and the convergence speed of the algorithm. Therefore, the present invention introduces the concept of composite priority (Composite priority, Cop) in the process of random sampling, sets the priority of the experience tuple, and the probability of the experience tuple in the experience pool being sampled is proportional to the composite priority of the sample. The larger the composite priority, the greater the probability of being sampled. In layman's terms, it is to give priority to more suitable samples. The composite priority defined in the present invention mainly includes two priorities.

[0117] The first is the priority based on the time difference error (TD-error). In the AC algorithm, TD-error is an important indicator for evaluating the performance of Acor. The Critic network is responsible for calculating TD-error, which mainly reflects the expected difference between the reward of the current state and the reward of the next state. The difference can be used to describe the difference between the expected state and the current state. If TD-error is a positive number, it means that the actual return is higher than the expected return, and the probability of taking this behavior should increase; if it is negative, it means that the actual return is lower than the expected return, and the probability of taking this behavior should decrease. The Actor network is responsible for updating the parameters according to TD-error and the policy gradient. By continuously updating the Actor parameters, the TD-error value can be gradually reduced, thereby improving the performance of the strategy. The main reason why the present invention selects TD-error as the evaluation indicator for priority sorting is that when the absolute value of TD-error is larger, it means that the correction effect of the sample on the network is more obvious, so it is more efficient to use such experience samples to train the network. In addition, the larger the TD-error of the experience sample, the less the number of occurrences of such states, and the agent performs poorly in dealing with such states. Therefore, increasing the probability of such experience samples being selected can further improve the utilization efficiency of samples and increase the learning speed of the agent, thereby making the algorithm converge faster.

[0118] The second is the priority based on real-time rewards. Reward is an important indicator for measuring the learning of the agent. The greater the reward, the better the learning of the agent. Therefore, when considering the priority sorting problem, the present invention also selects the real-time reward of the agent as the evaluation indicator for priority sorting.

[0119] Based on the composite priority composed of the above two priorities, the MADDPG algorithm can have a greater probability of selecting experience with more information for the AC network during the sampling process, thereby further improving the algorithm performance. Figure 3 shown.

[0120] The specific process of the sampling method of composite priority experience playback is as follows:

[0121] (1) Use the Q value to calculate the time difference error.

[0122] (2) Define the priority order using formula (18), where r t Indicates the real-time return of experience, represents a constant (the purpose of setting this constant is to prevent the situation where the sample will not be selected when TD-error=0), ω t Indicates the time difference error:

[0123]

[0124] (3) Sort the experience in the experience pool according to the priority obtained in (2) to obtain the sequence rank f With rank i , where rank i For high priority, rank f is a low priority, and the priority of all experiences is calculated using formula (19):

[0125]

[0126] Wherein α represents the degree of priority, and when α=0, it represents uniform sampling.

[0127] (4) Let n be the number of experiences, then the probability P of the sth sample experience being sampled is s The expression is:

[0128]

[0129] In a specific embodiment of the present invention, the simulation experiment environment is completed on a computer equipped with a six-core Intel i7-8750H CPU and an NVIDIA GeForce RTX 2060 GPU. By using the Python programming language, wireless communication network models such as vehicles, lanes and channels are successfully modeled. According to the technical specification document 3GPPTR36.885, the link environment of urban vehicles is simulated, and the deep learning framework of TensorFlow is used to manage the programming environment using PyCharm software.

[0130] Baseline algorithm:

[0131] (1) Random optimization algorithm (Random): In the V2V communication link between vehicles, vehicles select the transmission power in a random manner to achieve uniform distribution.

[0132] (2) Agent-based Deep Reinforcement Learning Algorithm (DDPG) based on Continuous Action Space: The training process of the DDPG algorithm includes two stages: experience replay and target network update. It adopts an offline learning method, which enables the algorithm to be trained more stably and can handle high-dimensional states and actions in the continuous action space. However, the algorithm can only obtain partial channel information and corresponding rewards of a single vehicle.

[0133] (3) Multi-agent deep reinforcement learning algorithm based on discrete actions (MADQN), in which there are multiple agents, and each agent is executed according to the deep Q network algorithm. In the real-time Internet of Vehicles environment, each agent selects discrete actions from its action set. Unlike the DDPG algorithm, this algorithm only uses one neural network technology.

[0134] Simulation parameter design:

[0135] The simulation experiment of this embodiment trains a total of 3000 episodes. A constantly changing Gaussian noise is added when each agent selects an action, and each vehicle-to-infrastructure V2I communication link corresponds to a vehicle-to-vehicle V2V communication link. The networks of all agents (Actor network, Critic network) are formed by 16, 64 and 256 neurons respectively. The weights of the neural network are continuously updated by using an adaptive moment estimation optimizer, and the exploration probability is processed by a linear annealing algorithm. The main simulation parameters are listed in Table 1.

[0136] Table 1

[0137]

[0138]

[0139] By using PyCharm software for programming, the Internet of Vehicles environment was first simulated, then different algorithms were programmed separately, and the generated data was visualized.

[0140] Figure 4 It shows how the cumulative reward value of training changes with the increase of the number of training times, from which it can be compared that the convergence and stability of the algorithm proposed in the present invention are better.

[0141] from Figure 4 The convergence characteristics of the MADDPG algorithm can be observed in . With the continuous iteration of the number of training times, the cumulative reward value gradually increases, which indicates that the agent is constantly adjusting its strategy to obtain a larger cumulative reward. When the episode reaches about 1500, the cumulative reward of the algorithm begins to converge. However, due to the high mobility of vehicles during driving, the topology of the network may change rapidly, which may cause fluctuations in channel attenuation, causing the algorithm to still fluctuate after convergence. At the same time, the convergence characteristics of the Cop-MADDPG algorithm can also be observed. The algorithm begins to converge when the episode reaches about 1000, which is faster than the MADDPG algorithm. In addition, the cumulative reward value and convergence amplitude at the time of algorithm convergence are higher than those of the MADDPG algorithm. By Figure 4The data comparison and analysis in can verify that the proposed algorithm has better robustness and stability than the MADDPG algorithm.

[0142] In order to verify the adaptability of the Cop-MADDPG algorithm to different environments, this embodiment changes the vehicle speed to test the performance of the algorithm under different vehicle speeds. Figure 5a Figure 5b As shown in the figure, when the number of vehicles is set to 4 and the load size is 2×1060Bytes, the change in vehicle speed has little effect on the success rate of the V2V communication link transmission between vehicles, indicating that the algorithm is more adaptable to environmental changes when the load size is low. When the load size is 4×1060Bytes and 6×Bytes, the change in vehicle speed has a greater impact on the success rate. The main reason is that with the rapid increase in vehicle speed, the environment faced by the Internet of Vehicles is more complex, and its network topology will change rapidly. With the increase in payload, this has higher requirements for channel selection and power allocation of the V2V communication link between vehicles, but from Figure 5a , Figure 5b It can be seen that the variation range of the transmission success rate of the V2V communication link between vehicles under different vehicle speeds is small, while the MADDPG algorithm has weak adaptability to vehicle speed changes. It can be seen that the improved algorithm proposed in the present invention has strong adaptability to the environment.

[0143] like Figure 6 As shown in the figure, the vehicle speed is set to 10m / s. At this time, as the V2V payload continues to increase, the interference to the vehicle-to-infrastructure V2I communication link will increase over a longer period of time, thereby affecting its capacity performance. That is, the larger the load of the inter-vehicle V2V communication link, the longer the data transmission time of the inter-vehicle V2V communication link will be, and the transmission power will be greater, thereby enhancing the interference ability to V2I and reducing the channel capacity. However, by comparison, it can be seen that the difference in the optimization problem of the total capacity of V2I using the reinforcement learning algorithm is obvious, and the optimization effect of the random algorithm on this problem is not significant. The optimization effect of the MADDPG algorithm is the most significant. When the load conditions are the same, the total capacity of the vehicle-to-infrastructure V2I communication link optimized by this algorithm is always higher than that of the other three algorithms, and as the load increases, it shows a spiral downward trend, while the other three algorithms have been showing a downward trend. In comparison, the robustness of this algorithm is better. The Cop-MADDPG algorithm proposed in the present invention is more accurate for the sampled data than the MADDPG algorithm. It can be seen from the figure that the optimization effect of the algorithm is basically consistent with that of the MADDPG algorithm, but because more reasonable data is selected in the sampling process, the training effect is improved.

[0144] like Figure 7As shown, the vehicle speed is set to 10m / s, and the influence of the load size of the inter-vehicle V2V communication link on the probability of successful transmission of the inter-vehicle V2V communication link load is observed. It can be observed that the probability of successful transmission of the five algorithms is decreasing. The main reason is that when the transmission rate is consistent with the transmission time slot, the amount of load that can be transmitted in the channel will not change significantly. Moreover, when the total load required for the inter-vehicle V2V communication link increases, the possibility of successful transmission of the load of the inter-vehicle V2V communication link will decrease accordingly. In addition, it can be clearly observed that under different load sizes, the inter-vehicle V2V communication link load transmission success rate of the MADDPG algorithm and the Cop-MADDPG algorithm is significantly higher than that of the MADQN algorithm and the DDPG algorithm, which proves that the method of the present invention has a significant effect in optimizing the Internet of Vehicles system.

[0145] In order to better understand the reasons why the Cop-MADDPG algorithm has a significant optimization effect, the algorithm will be analyzed separately. The Cop-MADDPG, MADDPG, MADQN and random algorithms are analyzed and compared through the four links of an episode (because the DDPG algorithm is only for a single agent and MADDPG is derived from this algorithm, this algorithm is not selected here).

[0146] like Figure 8a , Figure 8b , Figure 8c , Figure 8d As shown in the figure, when the time step is 30ms and the load of the V2V communication link between vehicles is set to 1060Bytes, the impact of the four algorithms on the instantaneous power of the V2V communication link between vehicles.

[0147] exist Figure 8a In the random algorithm, each link competes for spectrum resources, which will lead to increased signal interference between different links, thereby affecting the probability of successful load transmission, and thus greatly prolonging the time required to complete the load transmission.

[0148] exist Figure 8b In the paper, similar problems also exist when using the MADQN algorithm for optimization. However, compared with the random algorithm, the MADQN algorithm can reduce the time step and thus improve the efficiency of data transmission. Moreover, the peak value of the transmission rate is also significantly higher than that of the random algorithm, indicating that the algorithm can increase the total amount of data that can be transmitted within a period of time.

[0149] exist Figure 8cIn the figure, by using the MADDPG algorithm for optimization, the competition for spectrum resources among the links has been significantly improved. It can be seen that within 0-15ms, the transmission rates of the four links are all kept at a high level, which indicates that no strong interference signals are generated between different links. The probability of successful load transmission will be greatly enhanced, and the time required for successful load transmission will be greatly reduced, indicating that the algorithm is significantly better than the other two algorithms in optimizing the power control problem of the Internet of Vehicles.

[0150] exist Figure 8d It can still be seen that the optimization effects of the MADDPG algorithm and the Cop-MADDPG algorithm are similar, but because Cop-MADDPG uses a composite priority method based on the MADDPG algorithm to sample higher quality experience from the experience pool, the algorithm training efficiency and stability are improved. Therefore, on the basis of ensuring a high transmission rate, the impact of interference signals on the system is reduced, and the time to transmit the load is also reduced.

[0151] This embodiment uses Python programming to simulate the Internet of Vehicles environment, and uses the MADDPG algorithm to optimize the power control problem of the intelligent agent in the continuous action space. The optimization effects of various reinforcement learning algorithms are verified through experimental simulation, among which the Cop-MADDPG algorithm and the MADDPG algorithm are particularly effective in optimizing such problems. The probability of successful load transmission of all vehicle-to-vehicle V2V communication links and the total capacity of the vehicle-to-infrastructure V2I communication links are significantly improved. However, compared with the MADDPG algorithm, the Cop-MADDPG algorithm proposed in the present invention has better convergence and robustness, showing better system performance.

[0152] The present invention provides a method for optimizing vehicle network resources based on composite priority experience playback sampling. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A method for optimizing Internet of Vehicles resources based on composite priority experience replay sampling, characterized in that: The following steps are involved: Step 1: Problem modeling and goal definition; Step 2: Design state space and action space; Step 3: Design a reward function to guide the agent to obtain the maximum reward and select the optimal strategy; Step 4: Establish a resource allocation optimization algorithm based on the multi-agent deep deterministic gradient strategy MADDPG to improve the load transmission success rate of the vehicle-to-vehicle V2V communication link and maximize the total capacity of the vehicle-to-infrastructure V2I communication link; Step 5: Establish a multi-agent deep deterministic gradient strategy Cop-MADDPG algorithm based on composite priority experience replay sampling to improve the convergence and stability of the MADDPG algorithm and further optimize the power control strategy; Step 1 includes: the vehicle network based on the cellular network has a vehicle-to-vehicle V2V communication link and a vehicle-to-infrastructure V2I communication link; the vehicle-to-infrastructure V2I communication link connects the vehicle to the base station through a cellular interface, and when all vehicles have only one antenna, the sets of the vehicle-to-vehicle V2V communication link and the vehicle-to-infrastructure V2I communication link are respectively represented as K={0,…,K} and M={0,…,M}, where K and M represent the number of vehicle-to-vehicle V2V communication links and the number of vehicle-to-infrastructure V2I communication links, respectively; Step 1 also includes: in the time period X1, the power gain of the k-th vehicle-to-vehicle V2V communication link channel in the m-th subband is described as: g k [m]=a h h k [m] (1) Among them, h k [m] is the power amount of small-scale fading; a k It represents a large-scale attenuation that is independent of the frequency components; Step 1 also includes: setting g k′,k [m] represents the interference channel from the transmitter of the inter-vehicle V2V communication link No. k′ to the inter-vehicle V2V communication link No. k using the sub-band No. m. The channel gain of the sub-band No. m from the transmitter of the inter-vehicle V2V communication link No. k to the base station is expressed as g m,B [m], the power of the transmission power signal from the transmitter of V2I link number m on subband number m is expressed as g m,k [m], let represents the signal power of the transmitting end of the V2I communication link from vehicle m to the infrastructure, and the transmission power of the transmitting end of the V2V communication link between vehicle k on subband m is σ 2 Represents the noise power, then the ratio of the signal to the interference noise that can be received on the m-vehicle-to-infrastructure V2I communication link on the m-subband, that is, the expression of the signal-to-noise ratio is: The expression of the signal-to-noise ratio that can be received on the k-vehicle inter-vehicle V2V communication link on the m-subband is: Among them, I k [m] is the magnitude of the interference power; ρ k [m] represents a spectrum selection variable, which means: if the k-th vehicle-to-vehicle V2V communication link is transmitted on the m-th subband, ρ k [m] = 1, otherwise 0; If all inter-vehicle V2V communication links can only be connected to the same orthogonal sub-band, W is the bandwidth of each spectrum. According to Shannon's formula, the capacity of the m-vehicle to infrastructure V2I communication link on the m-sub-band is The expression is: Get the channel capacity of the V2V communication link between vehicles k on subband m The expression is: Assume B is the size of the V2V payload generated in a certain period, Δ T represents the time of the coherent channel, t represents the index of the coherent time, represents the instantaneous capacity and of all V2V links, then the expression of the transmission rate P of a data packet with a capacity of B within a specified time T is: The optimal resource allocation problem is expressed as follows: If for all k∈K, m∈M, the transmission power of the V2V communication link between vehicles is a continuous and variable quantity and maximize the total capacity of the vehicle-to-infrastructure V2I communication link and transmission rate; Step 2 includes: for the state space, when the time step is t, set the environment state of agent k in advance Assume that the state space of two or more agents is S t , when the agents select their own actions, they form a joint action A t ; When all agents perform actions at the same time, the reward they receive is R t , the state of agent k at this time Probability of passing Go to the next state For a single agent k, get the state of its local environment With action And it cannot obtain the conditions of the overall channel and the action information taken by other agents; the local channel information that a single agent k can obtain includes its own channel gain g k [m], and the interference channel at the transmitting end of the V2V communication link between other vehicles is g k,k′ [m], where all m∈M, the interference channel from the sender of agent k to the base station is g k,B [m], and the interference channel g at the transmitting end of all vehicle-to-infrastructure V2I communication links m,k [m]; the state that agent k can obtain is: For the action space, the resource allocation problem is mapped into a continuous control problem of the transmission power of the V2V communication link between vehicles. The transmission power of the transmitting end of the V2V communication link between vehicles ranges from is the maximum value of the transmitting power of the transmitting end of the V2V communication link between vehicles, and the action strategy expression of a single agent k is: Step 3 includes: expressing the instantaneous capacity and the capacity of the vehicle-to-infrastructure V2I communication link as The transmission rate of the V2V communication link between vehicles is set to the reward value L k , after the payload is delivered, the reward is set to a constant β, when the cumulative reward L for the successful transmission of the payload of the inter-vehicle V2V communication link within time step t k (t) is expressed as: Reward function R t The expression is: Where μ is the weight, 0<μ<1; The interference channel from the transmitter of the interference link agent to the base station is represented as g m , k [m], the negative reward factor is ε, 0<ε<1, then the expression of the reward function is: Step 4 includes: the resource allocation optimization algorithm based on the multi-agent deep deterministic gradient strategy MADDPG includes an Actor network and a Critic network. The Actor network continuously collects environmental states and then performs corresponding actions according to the strategy. The Critic network uses the strategy Q k (*) to evaluate the quality of the actions performed by the Actor network; let the input state of agent k be Instant Rewards The discount factor is γ, then the state and action value function Q of the DDPG algorithm k (S k , A k ) is: Q k (S k ,A k )=E[R k +γQ(S k′ ,A k′ )] (13) First, agent k will get an optimal strategy And through continuous learning and this strategy The corresponding state and action value functions until convergence; The MADDPG algorithm adopts a dual network structure, including an evaluation network and a target network. The evaluation network includes an Actor evaluation network and a Critic evaluation network. The parameters of the Actor evaluation network are The parameters of the Critic evaluation network are The target network includes the Actor target network and the Critic target network. The parameters of the Actor target network are The parameters of the Critic target network are The reward for the kth agent is Then the loss function The expression is: Among them, Q′ k (*) is the action and state value function of the target network; E is the mathematical expectation; and are all conditional probabilities; When the actors make corresponding actions through continuous interaction, and all agents are seeking to maximize the cumulative reward, the parameters of the actors are changed by maximizing the strategy objective function. The expression is: Among them, μ(*) represents a deterministic strategy π for state mapping action k Actor evaluation network function; Using a constant close to 0 ω To change the parameters of the target network, the expression is:

2. The method according to claim 1, characterized in that Step 4 also includes: The resource allocation optimization algorithm based on the multi-agent deep deterministic gradient strategy MADDPG includes centralized learning and distributed execution: each group of evaluation networks and target networks are regarded as a separate agent. In the centralized training process, agent k not only needs to obtain state information from the local state and the actions currently in progress You also need to get the status of other agents and action information Status Information The current action The status of other agents and action information It is stored in the experience pool of the kth agent for subsequent centralized training and evaluation of the network.

3. The method according to claim 2, characterized in that Step 5 includes: introducing a composite priority, wherein the composite priority includes a priority based on a time difference error TD-error and a priority based on a real-time report; and a sampling method of composite priority experience playback includes the following steps: Step 5-1, use the Q value to calculate the time difference error TD-error, Q value is the state and action value function Q k (S k , A k ); Step 5-2, use formula (18) to define the priority order: Among them, r t Indicates the real-time return of experience, represents a constant, ω t Indicates the time difference error; Y i is the priority based on immediate return, Y f is the priority based on TD-error; Step 5-3: Sort the experience in the experience pool according to the priority obtained in step 5-2 to obtain the sequence rank f With rank i , where rank i For high priority, rank f is a low priority, and the composite priority u of all experiences is calculated using formula (19): k : where Y k Indicates the degree of composite priority, α represents the probability weight, and when α = 0, it represents uniform sampling; Step 5-4, let n be the number of experiences, then the probability P of the sth sample experience being sampled is s The expression is:

4. An electronic device, comprising a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 3.

Images