Vehicle network resource allocation method, storage medium and device based on reinforcement learning

Abstract

The present application discloses a method for allocating resources in an Internet of Vehicles based on reinforcement learning, including: an Internet of Vehicles environment module collects current environmental data transmissions and stores them in an initial experience replay module; a cosine similarity module filters the data stored in the initial experience replay module and sends them to a spectrum allocation module and a transmission power allocation module for training a Target network; a power allocation module based on a DQN algorithm and a transmission power allocation module based on a DDPG algorithm output allocation actions, and the actions are compounded by a final strategy module and acted on the Internet of Vehicles environment module. The present invention improves the reliability of Internet of Vehicles resource allocation by taking environmental change variables as parameter calculations; reduces the deviation of Internet of Vehicles resource allocation results by compounding spectrum and power conditions; and makes the entire system converge faster by adding cosine similarity filtering to experience replay.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a vehicle network resource allocation method, storage medium and device based on reinforcement learning. Background Art

[0002] In the study of resource allocation in the Internet of Vehicles, how to ensure that the vehicle network can provide satisfactory quality of service (QoS) is the main challenge. When vehicles are driving at high speed, a suitable resource allocation scheme can avoid a series of factors that affect QoS, such as inappropriate channels (congested or wrong) and congested spectrum. This is closely related to the development and utilization of the Internet of Vehicles and has broad application prospects.

[0003] Reinforcement learning (RL) can effectively solve sequential decision-making problems. Current research findings also fully demonstrate that RL has great potential in solving the resource allocation problem in the Internet of Vehicles. For example, in the article "Deep Reinforcement Learning Resource Allocation in Wireless Sensor Networks With Energy Harvesting and Relay", a resource allocation strategy developed by deep reinforcement learning (DRL) in the model of building a Markov decision process is used to solve the communication interference problem between sensor nodes in wireless sensor networks caused by channel fading, insufficient and random energy arrival, and improper deployment of sensors. In the article "Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning", resource sharing is modeled as a multi-agent reinforcement learning (RL) problem to solve the spectrum allocation problem occupied by vehicle-to-infrastructure (V2I) links reused by multiple vehicle-to-vehicle (V2V) links.

[0004] In the existing research on RL-based IoV resource allocation methods, the most common approach is to assume that the power is constant and then allocate the spectrum, which results in experimental results under the condition of certain power. In actual applications, when the vehicle is driving at high speed, the power of each link in the IoV should change in real time, which makes it difficult for this type of algorithm to obtain experimental results that match the operation of the IoV when the vehicle is driving at high speed in actual situations. This is extremely unfavorable for our research on IoV resource allocation. In addition, this research method not only ignores the variables brought about by power conditions and environmental changes, but also fails to comprehensively observe and analyze the conditions of spectrum and power, and even causes a large deviation between the experimental results and the actual process production. Summary of the invention

[0005] In order to overcome the above technical defects, the present invention provides a vehicle network resource allocation method, storage medium and device based on reinforcement learning, which can improve the reliability of vehicle network resource allocation.

[0006] In order to solve the above problems, the present invention is implemented according to the following technical solutions:

[0007] In a first aspect, a method for allocating resources of an Internet of Vehicles based on reinforcement learning comprises the following steps:

[0008] S100: The Internet of Vehicles environment module transmits the current state S, the current action A, the current reward R, and the next state S+ generated after the current action acts on the current reward to the initial experience playback module;

[0009] S200: Experience replay setting optimization step, the initial data placed in the initial experience replay area is filtered by the cosine similarity module, the filtered samples include the current state S, the current action A, the current reward R, and the next state S+ generated after the current action acts on the current reward, the filtered samples are stored in the final experience replay area, and the data are transmitted to the spectrum allocation module and the transmission power allocation module respectively to train the corresponding Target network;

[0010] S300: The spectrum allocation module outputs a spectrum sub-band allocation action based on the DQN algorithm and transmits it to the final action strategy module. The transmission power allocation module outputs a power allocation action based on the DDPG algorithm and transmits it to the final action strategy module. The final action strategy module combines the actions from the two modules, outputs the final action, acts on the Internet of Vehicles environment module, and goes to step S100.

[0011] Furthermore, the reward R is expressed as:

[0012] in:

[0013] represents the channel capacity of the nth V2I link on the mth sub-band in time slot t;

[0014] P k is a binary number, if P k =1, the kth V2V link uses the mth sub-band, otherwise P k =0;

[0015] represents the capacity of the kth V2V link on the mth sub-band in time slot t;

[0016] Bk represents the effective load of the kth V2V link on the mth sub-band; the hyperparameter is set to β.

[0017] Furthermore, the spectrum allocation module adopts an ε-greedy strategy, and the spectrum allocation module trains the Target network including the following steps:

[0018] S210: Before the start, the Internet of Vehicles environment outputs S to the spectrum allocation module, and the spectrum allocation module will randomly select an action and then transmit the action to the Internet of Vehicles environment. The Internet of Vehicles environment generates a new environmental condition S+ for the corresponding action. When the action is selected for the second time, a judgment will be made to select the action with the highest corresponding Q value with a probability of 1-ε, or a random action will be selected with a probability of ε and then stored in the initial experience playback area module;

[0019] S220: The Q network obtains the Internet of Vehicles environment state S from the Internet of Vehicles environment module, and obtains an action after calculation. After the action is stored in the initial experience playback area module, it will act on the Internet of Vehicles environment module to generate a new state for the Q network to obtain. This step is repeated until the initial experience playback area module is filled with preset data, and the spectrum allocation module will start to update the Q network.

[0020] Further, the updating of the Q network comprises the steps of:

[0021] S221: The current reward R and the next state S+ are obtained by sampling in the final experience replay area and output to the Target network;

[0022] S222: The Target network calculates the next Q value y using R and S+, and sends y to the Q network. The Q network calculates the loss value based on y and starts updating the Q network based on the loss value.

[0023] S223: When the number of Q network updates reaches the set first threshold, the Q network will copy all parameters to the Target network and use them to update the Target network. After the update, it will continue to obtain the corresponding status from the environment and go to step S210.

[0024] Furthermore, the transmission power allocation module adopts the Uhlenbeck-Ornstein random process, adds random noise when selecting actions, obtains corresponding actions through sampling, and then stores the sampled actions in the initial experience playback area module.

[0025] Furthermore, the transmission power allocation module includes an actor module and a Critic module. The state, reward and next state obtained by environmental sampling will be input into the Actor module. The Actor module converts the input state into state, action, reward and next state, and stores them in the initial experience playback area module for updating the data sets of the Actor module and the Critic module.

[0026] Furthermore, the updating of the Critic module comprises the steps of:

[0027] According to the loss function and label value y i , use back-propagation to get the current gradient of the Q network, where the loss function is defined as the mean square error, and the label value y i It is calculated jointly by the Target strategy network and the Target Q network;

[0028] Start updating the Q network using the Adam optimizer method based on the gradient;

[0029] When the number of Q network updates reaches the second threshold, the Target Q network is updated in a weighted mode, where the parameter weight of the Q network is τ, and the parameter weight of the current Target Q network is 1-τ.

[0030] Furthermore, the updating of the Actor module comprises the steps of:

[0031] The function J(θ) is used to measure the quality of the policy network, which is expressed as follows: J(θ) = ∫ S ρ(s)Qθ(s,θ(s))ds, where θ is the parameter of the policy network, ρ(s) is the distribution function of the state S that follows the normal distribution after Uhlenbeck-Ornstein random noise, and Q θ (s,θ(s)) is the Q value that can be generated after selecting an action according to the policy network;

[0032] The gradient of the J(θ) function is obtained using the Monte-Carlo method, and the Adam optimizer method is used to update the policy network parameters θ.

[0033] When the number of policy network updates reaches the third threshold, the Target policy network is updated in a weighted mode, where the parameter weight of the policy network is τ, and the parameter weight of the current Target policy network is 1-τ.

[0034] In a second aspect, the present invention provides a computer-readable storage medium, wherein at least one instruction, at least one program, code set or instruction set is stored in the computer-readable storage medium, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the reinforcement learning-based vehicle network resource allocation method as described in the first aspect.

[0035] In a third aspect, the present invention provides a device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the reinforcement learning-based vehicle network resource allocation method as described in the first aspect.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention improves the reliability of vehicle network resource allocation by taking environmental change variables as parameter calculations; reduces the deviation of vehicle network resource allocation results by composite analysis of spectrum and power conditions; and makes the entire system converge faster by adding cosine similarity filtering to experience playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings, wherein:

[0039] Figure 1 This is an application scenario diagram of the vehicle networking resource allocation method based on reinforcement learning in this embodiment 1;

[0040] Figure 2 This is a flow chart of the vehicle networking resource allocation method based on reinforcement learning in this embodiment 1;

[0041] Figure 3 This is a flowchart of vehicle networking resource allocation based on reinforcement learning in this embodiment 1;

[0042] Figure 4 This is a structural block diagram of the vehicle networking resource allocation method based on reinforcement learning in this embodiment 1;

[0043] Figure 5 This is a flowchart of step S200 of the vehicle networking resource allocation method based on reinforcement learning in this embodiment 1;

[0044] Figure 6 This is a diagram showing the internal structure of the spectrum allocation module in Embodiment 1;

[0045] Figure 7 This is a flowchart of step S220 of the vehicle networking resource allocation method based on reinforcement learning in this embodiment 1;

[0046] Figure 8 This is a diagram of the internal structure of the transmission power allocation module of this embodiment 1.

[0047] Description of reference numerals:

[0048] 1. Internet of Vehicles environment module; 2. Spectrum allocation module; 3. Transmitting power allocation module; 4. Final action strategy module; 5. Initial experience playback area module; 6. Cosine similarity filtering module; 7. Final experience playback area module; 21. First current state acquisition; 22. First random action selection; 23. First Q network; 24. First Target Q network; 25. Strategy calculation; 31. Actor module; 32. Critic module; 33. Second current state acquisition; 34. Second random action selection; 311. Target strategy network; 312. Strategy network; 313. First optimization module; 321. Second Target Q network; 322. Second Q network; 323. Second optimization module; 8. V2V link; 9. V2I link; 10. Internet of Vehicles user; 11. Base station; 12. Vehicle driving direction. DETAILED DESCRIPTION

[0049] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0050] It should be noted that the serial numbers mentioned in this article, such as S1, S2, etc., are only used to distinguish between steps and do not mean that the steps must be strictly executed in the order of the serial numbers.

[0051] Example 1

[0052] The vehicle network resource allocation method based on reinforcement learning provided in this embodiment is applied to Figure 1 In the Internet of Vehicles environment shown in the figure. Figure 1 As shown in the figure, for the Internet of Vehicles in a high-speed vehicle environment, the V2X network can be divided into a V2I link and a V2V link, where the V2I link provides a high-speed information flow to ensure the user's entertainment needs, and the V2I provides an ultra-high stability connection to transmit the user's safety information. Because the link resources available for the Internet of Vehicles are limited within a certain space, and the high-speed driving of the vehicle will cause part of the limited resources to be unavailable in certain specific situations.

[0053] In order to solve the above problems, Figure 2 , Figure 3 As shown, this embodiment provides a method for allocating Internet of Vehicles resources based on multi-agent reinforcement learning, comprising the steps of:

[0054] S100: The Internet of Vehicles environment module transmits the current state S, the current action A, the current reward R, and the next state S+ generated after the current action acts on the current reward to the initial experience playback module;

[0055] S200: Experience replay setting optimization step, the initial data placed in the initial experience replay area is filtered by the cosine similarity module, the filtered samples include the current state S, the current action A, the current reward R, and the next state S+ generated after the current action acts on the current reward, the filtered samples are stored in the final experience replay area, and the data are transmitted to the spectrum allocation module and the transmission power allocation module respectively to train the corresponding Target network;

[0056] S300: The spectrum allocation module outputs a spectrum sub-band allocation action based on the DQN algorithm and transmits it to the final action strategy module. The transmission power allocation module outputs a power allocation action based on the DDPG algorithm and transmits it to the final action strategy module. The final action strategy module combines the actions from the two modules, outputs the final action, acts on the Internet of Vehicles environment module, and goes to step S100 to enter the next cycle.

[0057] In this embodiment, if Figure 4 As shown, the Internet of Vehicles resource allocation method based on multi-agent reinforcement learning includes an Internet of Vehicles environment module, a spectrum allocation module, a transmission power allocation module, a final action strategy module, an initial experience replay area module, a cosine similarity filtering module and a final experience replay area module.

[0058] The vehicle networking environment module consists of V2V links, vehicle networking users, base stations, driving directions, and V2I links. In this module, we regard each vehicle networking user as a single agent and other vehicle networking users as the environment, which makes each agent face a non-stationary environment when taking action, so the sampling link needs to be optimized. Multiple agents communicate with each other using V2V links, and each agent is connected to the base station using V2I links.

[0059] When working, the Internet of Vehicles environment module first obtains the current state S, the current action A, the current reward R, and the next state S+ generated after the current action acts on the current reward, and transmits the above acquired data to the initial experience playback module;

[0060] The experience replay mechanism will optimize the data stored in the initial experience replay module by transferring the data currently stored in the initial experience replay area to the cosine similarity filtering module, and the cosine similarity filtering module will perform cosine similarity filtering on the data samples. After the cosine similarity filtering, the data samples will be output to the final experience replay area module for storage. The data in the final experience replay area module will be output to the spectrum allocation module and the transmission power allocation module, and will be used to train the corresponding Target network. In the Internet of Vehicles environment, in order to solve the problem of multi-agent RL, each agent must face a non-stationary environment, which makes the final result difficult to converge. By optimizing the traditional experience replay, adding cosine similarity filtering, and filtering the experience replay samples with corresponding weight conditions, the entire system can converge faster.

[0061] Finally, the spectrum allocation module and the power allocation module will train the Target network according to the data input by the final experience playback area module, and respectively assign a final allocation action to the final action strategy module. The final action strategy module will compound the allocation action from the spectrum allocation spectrum subband and the power allocation action from the transmission power allocation module, output the final action (i.e., the spectrum allocation action and the transmission power control action of the V2V link), apply the final action to the environment, and repeat step S210 to start the next cycle. Among them, the spectrum allocation module optimizes the Target network based on the DQN algorithm, and the transmission power allocation module optimizes the Target network based on the DDPG algorithm. While allocating the spectrum through the DQN algorithm, the link transmission power is allocated through the DDPG algorithm, which effectively avoids the adverse effects brought by the traditional algorithm. In the process of close to the high-speed driving of the real vehicle, the composite environment simulation of the Internet of Vehicles can allocate limited resources more reasonably to each link, so that better Qos can be enjoyed during the high-speed driving of the vehicle.

[0062] The basic parameters in reinforcement learning RL are defined as follows:

[0063] State St: The state obtained by the agent from the environment consists of three parts: interference between V2V links, interference of V2V links on V2I links, and interference of V2I links on V2V links.

[0064] Action A t:RL action consists of two parts, namely the output A1 of the power allocation module and the output A2 of the transmission power allocation module. These two parts are combined into an action to act on the environment, thereby generating a new state. The spectrum allocation module is optimized based on the DQN algorithm, so the output spectrum subband selection action A1 of this module is a discrete action. The transmission power allocation module is optimized based on the DDPG algorithm, so the output transmission power selection output A2 of this module is a continuous action, which is determined by a strategy, that is, it can be expressed as A2=π(S t +N), where N represents random noise. Therefore, in the final strategy action module, At = {A1, A2}, and At is the composite action output by the final strategy action module.

[0065] Reward R: In RL, the setting of reward is very important, which is related to the direction in which the agent should be optimized. In this invention, the reward setting is as follows:

[0066]

[0067] in represents the channel capacity of the nth V2I link on the mth sub-band in time slot t;

[0068] P k is a binary number, if P k =1, the kth V2V link uses the mth sub-band, otherwise P k =0;

[0069] represents the capacity of the kth V2V link on the mth sub-band in time slot t;

[0070] Bk represents the effective load of the kth V2V link on the mth sub-band.

[0071] At the same time, a hyperparameter β is set to improve the packet delivery rate of the V2V link.

[0072] In one embodiment, the spectrum allocation module adopts an ε-greedy strategy, such as Figure 5 , Figure 6 As shown, the spectrum allocation module trains the Target network including the following steps:

[0073] S210: Before the start, the Internet of Vehicles environment outputs S to the spectrum allocation module, and the spectrum allocation module will randomly select an action and then transmit the action to the Internet of Vehicles environment. The Internet of Vehicles environment generates a new environmental condition S+ for the corresponding action. When the action is selected for the second time, a judgment will be made to select the action with the highest corresponding Q value with a probability of 1-ε, or a random action will be selected with a probability of ε and then stored in the initial experience playback area module;

[0074] S220: The Q network obtains the Internet of Vehicles environment state S from the Internet of Vehicles environment module, obtains an action after calculation, and after the action is stored in the initial experience playback area module, it will act on the Internet of Vehicles environment module to generate a new state for the Q network to obtain, and repeat this step until the initial experience playback area module is full of preset data, and the spectrum allocation module will start to update the Q network;

[0075] Among them, the spectrum allocation module includes the first current state acquisition, the first random action selection, the first Q network, the first Target Q network, and the strategy calculation. The Q network obtains the Internet of Vehicles environment state S from the Internet of Vehicles environment module, and obtains an action after calculation. After the action is stored in the initial experience playback area module, it will act on the Internet of Vehicles environment module, so that the Internet of Vehicles environment module generates a new state according to the action. This new state will be used as channel information for the Q network to obtain. Through the above process cycle, until the initial experience playback area module is full of preset data, the spectrum allocation module will start to update the Q network. The Q network in DQN is the main network, and its parameters will determine the estimation of the predicted Q value by the entire network.

[0076] Specifically, Figure 7 As shown, updating the Q network includes the steps:

[0077] S221: The current reward R and the next state S+ are obtained by sampling in the final experience replay area and output to the Target network;

[0078] S222: The Target network uses R and S+ to calculate the next Q value y, and sends y to the Q network. The Q network calculates the loss value through y and starts updating the Q network according to the loss value. The loss value is the optimization target, and the ultimate goal of the entire DQN algorithm is to minimize the loss value.

[0079] S223: When the number of Q network updates reaches the set first threshold, the Q network will copy all parameters to the Target network and use them to update the Target network. After the update, it will continue to obtain the corresponding status from the Internet of Vehicles environment module and go to step S210.

[0080] In one embodiment, if Figure 8As shown, the transmit power allocation module consists of a second current state acquisition, a second random action selection, an Actor module, a Target strategy network, a strategy network, a first optimization module, a Critic module, a second TargetQ network, a second Q network, and a second optimization module.

[0081] The transmit power allocation module uses the Uhlenbeck-Ornstein random process, adds random noise when selecting actions, obtains corresponding actions through sampling, and then stores them in the initial experience playback area module. However, the Uhlenbeck-Ornstein random process will not be added in the RL test phase.

[0082] Specifically, the transmission power allocation module includes an actor module and a critic module. The state, reward, and next state obtained by environmental sampling are input to the actor module. The actor module converts the input state into state, action, reward, and next state, and stores them in the initial experience playback area module to update the data sets of the actor module and the critic module. The ultimate goal of the actor module is to obtain a strategy that maximizes the J function, while the ultimate goal of the critic module is to obtain an action that minimizes the loss.

[0083] Furthermore, for updating the dataset of the Critic module, the parameters of the second Q network need to be updated first, and the update of the parameters needs to refer to the gradient of the loss function. As in the spectrum allocation module, the loss function here is defined as the mean square error, where y is used as the label value i The target strategy network and the second target Q network jointly calculate the loss function and label value y i , the current gradient of the second Q network can be obtained by back-propagation. After obtaining the gradient, the second Q network can be updated by the Adam optimizer method. In this way, after the second Q network is updated 10 times, the second Target Q network will be updated. Different from the spectrum allocation module, the update here adopts a soft update method. It does not simply directly cover the second Target Q network with the parameters of the second Q network, but uses a weighted mode. Among them, the parameter weight of the second Q network is τ, the parameter weight of the current Target Q network is 1-τ, and the parameter τ is 0.001.

[0084] Further, for updating the data set of the Actor module: First, a function is used to measure the quality of our policy network. This function is generally named J(θ), where θ is the parameter of the policy network. In this embodiment, the function is expressed as: J(θ) = ∫ S ρ(s)Q θ(s,θ(s))ds, where ρ(s) is the distribution function of the state S that follows the normal distribution after Uhlenbeck-Ornstein random noise, Q θ (s,θ(s)) is the Q value that can be generated if the action is selected according to the policy network. The ultimate goal of the Actor module is to adjust θ to maximize the J(θ) function. Here, the gradient of the J(θ) function needs to be calculated first. This gradient can be obtained using the Monte-Carlo method. After obtaining the gradient of J(θ), the optimization module uses the Adam optimizer to update θ. After the policy network is updated 10 times, the Target policy network will begin to be updated, and the update method is soft update. Among them, the parameter weight of the policy network is τ, the parameter weight of the current Target policy network is 1-τ, and the parameter τ is 0.001.

[0085] Example 2

[0086] An embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the vehicle network resource allocation method based on reinforcement learning provided in Example 1 of the present invention.

[0087] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0088] As is known to those of ordinary skill in the art, the term computer-readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0089] Exemplarily, the computer-readable storage medium may be an internal storage unit of the network management device described in the foregoing embodiment, such as a hard disk or memory of the network management device. The computer-readable storage medium may also be an external storage device of the network management device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the network management device.

[0090] Example 3

[0091] An embodiment of the present invention provides a device, including a processor and a memory, wherein the memory is used to store a computer program; the processor is used to execute the computer program and implement the covariate-based Kubernetes resource scheduling optimization method provided in Embodiment 1 of the present invention when executing the computer program.

[0092] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0093] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A vehicle network resource allocation method based on reinforcement learning, characterized in that: Includes steps: S100: The Internet of Vehicles environment module transmits the current state S, the current action A, the current reward R, and the next state S+ generated after the current action acts on the current reward to the initial experience playback module; S200: Experience replay setting optimization step, the initial data placed in the initial experience replay area is filtered by the cosine similarity module, the filtered samples include the current state S, the current action A, the current reward R, and the next state S+ generated after the current action acts on the current reward, the filtered samples are stored in the final experience replay area, and the data are transmitted to the spectrum allocation module and the transmission power allocation module respectively to train the corresponding Target network; S300: The spectrum allocation module outputs a spectrum subband allocation action based on the DQN algorithm and transmits it to the final action strategy module. The transmit power allocation module outputs a power allocation action based on the DDPG algorithm and transmits it to the final action strategy module. The final action strategy module combines the actions from the two modules, outputs the final action, acts on the Internet of Vehicles environment module, and goes to step S100; The spectrum allocation module adopts the ε-greedy strategy, and the spectrum allocation module trains the Target network including the following steps: S210: Before the start, the Internet of Vehicles environment outputs S to the spectrum allocation module, and the spectrum allocation module will randomly select an action and then transmit the action to the Internet of Vehicles environment. The Internet of Vehicles environment generates a new environmental condition S+ for the corresponding action. When the action is selected for the second time, a judgment will be made to select the action with the highest corresponding Q value with a probability of 1-ε, or a random action will be selected with a probability of ε and then stored in the initial experience playback area module; S220: The Q network obtains the Internet of Vehicles environment state S from the Internet of Vehicles environment module, obtains an action after calculation, and after the action is stored in the initial experience playback area module, it will act on the Internet of Vehicles environment module to generate a new state for the Q network to obtain, and repeat this step until the initial experience playback area module is full of preset data, and the spectrum allocation module will start to update the Q network; The updating of the Q network comprises the steps of: S221: The current reward R and the next state S+ are obtained by sampling in the final experience replay area and output to the Target network; S222: The Target network calculates the next Q value y using R and S+, and sends y to the Q network. The Q network calculates the loss value based on y and starts updating the Q network based on the loss value. S223: When the number of Q network updates reaches the set first threshold, the Q network will copy all parameters to the Target network and use them to update the Target network. After the update, it will continue to obtain the corresponding status from the environment and go to step S210.

2. The method for allocating Internet of Vehicles resources based on reinforcement learning according to claim 1, characterized in that: The reward R is expressed as: in: represents the channel capacity of the nth V2I link on the mth sub-band in time slot t; P k is a binary number, if P k =1, the kth V2V link uses the mth sub-band, otherwise P k =0; represents the capacity of the kth V2V link on the mth sub-band in time slot t; B k represents the effective load of the kth V2V link on the mth sub-band; the hyperparameter is set to β.

3. The method for allocating Internet of Vehicles resources based on reinforcement learning according to claim 1, characterized in that: The transmission power allocation module adopts the Uhlenbeck-Ornstein random process, adds random noise when selecting actions, obtains corresponding actions through sampling, and then stores the sampled actions in the initial experience playback area module.

4. The method for allocating Internet of Vehicles resources based on reinforcement learning according to claim 3, characterized in that: The transmission power allocation module includes an actor module and a critic module. The state, reward and next state obtained by environmental sampling are input into the actor module. The actor module converts the input state into state, action, reward and next state, and stores them in the initial experience playback area module for updating the data sets of the actor module and the critic module.

5. The method for allocating Internet of Vehicles resources based on reinforcement learning according to claim 4, characterized in that: The updating of the Critic module comprises the following steps: According to the loss function and label value y i , use back-propagation to get the current gradient of the Q network, where the loss function is defined as the mean square error, and the label value y i It is calculated jointly by the Target strategy network and the Target Q network; Start updating the Q network using the Adam optimizer method based on the gradient; When the number of Q network updates reaches the second threshold, the Target Q network is updated in a weighted mode, where the parameter weight of the Q network is τ, and the parameter weight of the current Target Q network is 1-τ.

6. The method for allocating Internet of Vehicles resources based on reinforcement learning according to claim 4, characterized in that: The updating of the Actor module includes the following steps: The function J(θ) is used to measure the quality of the policy network, which is expressed as follows: J(θ) = ∫ S ρ(s)Q θ (s,θ(s))ds, where θ is the parameter of the policy network, ρ(s) is the distribution function of the state S that follows the normal distribution after Uhlenbeck-Ornstein random noise, and Q θ (s,θ(s)) is the Q value that can be generated after selecting an action according to the policy network; The gradient of the J(θ) function is obtained using the Monte-Carlo method, and the Adam optimizer method is used to update the policy network parameters θ. When the number of policy network updates reaches the third threshold, the Target policy network is updated in a weighted mode, where the parameter weight of the policy network is τ, and the parameter weight of the current Target policy network is 1-τ.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the reinforcement learning-based vehicle network resource allocation method as described in any one of claims 1 to 6.

8. A device, characterized in that The device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the reinforcement learning-based vehicle network resource allocation method as described in any one of claims 1 to 6.

Images