Vehicle cooperation caching relay method based on deep reinforcement learning in internet of vehicles

Abstract

The application discloses a vehicle cooperation cache relay method based on deep reinforcement learning in Internet of Vehicles, divides popular contents of the Internet of Vehicles, predicts a moving track of each vehicle in the Internet of Vehicles whenever a cooperation cache update period arrives, calculates a predicted weight adjacency matrix according to link stability between any two vehicles obtained by the predicted moving track, solves a cache vehicle and a corresponding service vehicle set of a next cache period based on the predicted weight adjacency matrix, and each cache vehicle respectively carries out Markov decision to obtain a content cache strategy of the cache vehicle. The application solves an optimal cache strategy of the cache vehicle of the next cache period, realizes relay cooperation transmission of the cache vehicle to a request vehicle, and determines system performance.

Description

Technical Field

[0001] This invention belongs to the field of mobile vehicle edge network technology, and more specifically, relates to a vehicle cooperative caching relay method based on deep reinforcement learning in the Internet of Vehicles. Background Technology

[0002] With the rapid development of cloud computing, the Internet of Things (IoT), and the Internet of Vehicles (IoV), the demand for content from mobile vehicles is also increasing rapidly. Intelligent driving vehicles need to constantly learn their surroundings and make decisions in real time. Therefore, vehicles can be considered mobile devices used to collect and process environmental data and support various information services. Traditional mobile cloud computing, due to its high latency and network power consumption, is unsuitable for latency-sensitive applications, impacting the user experience. Mobile edge computing (MEC) is a technology that moves computing and storage resources to the network edge, improving data transmission and computing efficiency while reducing data center load and latency. As an important technology in edge computing, edge caching can cache popular data and applications on edge devices, thereby accelerating data access and application response, and reducing network bandwidth and latency. It utilizes edge nodes as caching nodes, storing content near the user, thus reducing the burden on the core network and improving user experience. In MEC, edge nodes can serve a large number of content requests, reducing service latency and content delivery distance. Therefore, mobile edge caching, as a promising technology, is widely used in mobile edge computing frameworks to reduce latency.

[0003] In existing mobile edge caching technologies, vehicle-to-everything (V2X) content is typically cached in a subset of vehicles, known as cache vehicles. Other vehicles requesting content (referred to as requesting vehicles) can retrieve the content from the cache vehicles. However, due to the mobility of vehicles, requesting vehicles may move out of the coverage area before the required content is fully cached within the communication range of the cache vehicles, causing caching interruptions. In this case, the requesting vehicles would need to re-request the content, resulting in wasted resources. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a vehicle cooperative caching relay method based on deep reinforcement learning in the Internet of Vehicles. By solving the optimal caching strategy for caching vehicles in the next caching cycle, the system achieves relay cooperative transmission of cached vehicles to requesting vehicles, thereby determining the system performance.

[0005] To achieve the above-mentioned objectives, the vehicle cooperative caching relay method based on deep reinforcement learning in the Internet of Vehicles (IoV) of this invention includes the following steps:

[0006] S1: Let the popular content in the Internet of Vehicles be F = {f1, f2, ..., f W}, where f wLet f represent the w-th popular content chunk, where w = 1, 2, ..., W, and W represents the number of popular content chunks. w The size is L w Let V be the number of intelligent vehicles in the entire vehicle network area. If vehicle i requests popular content within the current cache period t, then vehicle i is the requesting vehicle; otherwise, vehicle i is not the requesting vehicle. i = 1, 2, ..., V.

[0007] S2: Whenever the collaborative cache update cycle arrives, predict the movement trajectory of each vehicle in the vehicle network to obtain the predicted trajectory of the vehicle in the next cache cycle t+1.

[0008] S3: For any two vehicles i and j, i,j=1,2,…,V, calculate the distance between vehicles i and j in the next buffer period t+1 based on the predicted trajectories of these two vehicles in the next buffer period t+1. Then, the normalized communication distance tolerance between vehicles i and j in the next buffer period t+1 is calculated using the following formula.

[0009]

[0010] Among them, R veh d represents the communication radius of the vehicle. min This indicates the minimum distance between two vehicles under the safe distance limit;

[0011] Based on the predicted trajectories of the two vehicles in the next buffer period t+1, the duration of mutual coverage between vehicles i and j in the next buffer period t+1 is calculated as the link duration between vehicles i and j.

[0012]

[0013] Then, the link stability between vehicles i and j within the next buffer period t+1 is calculated using the following formula.

[0014]

[0015] Where α∈[0,1] represents the preset weighting factor;

[0016] Link stability As elements, a prediction weight adjacency matrix of size N×N can be constructed.

[0017] S4: Treat vehicles as nodes and predict the adjacency matrix W. t+1The graph G is constructed using the adjacency matrix. The minimum dominating set U of graph G is obtained by using the minimum dominating set algorithm. The vehicle nodes contained in U are used as cached vehicles. The specific steps of the minimum dominating set algorithm are as follows:

[0018] S4.1: Initialize the state of each vehicle to be an undetermined vehicle, that is, set the flag bit of the vehicle to 0 in the next cache cycle t+1;

[0019] S4.2: Determine if there are any vehicles with undetermined status. If yes, proceed to step S4.3; otherwise, end the cached vehicle filtering process.

[0020] S4.3: For each vehicle in an undetermined state, calculate its connectivity Z in graph G, which connects to the nodes in the undetermined state. i , i∈Y, where Y represents the set of vehicles whose current state is undetermined;

[0021] S4.4: Select cached vehicles from the current undetermined vehicle set Y using the following method:

[0022] If the connectivity Z exists i For vehicle nodes with a value of 0, the vehicle will be designated as a cached vehicle, and its flag will be set to 2.

[0023] If there is no connectivity Z i For vehicle nodes with a degree of 0, determine if they have a connectivity Z. i For vehicle nodes with a value of 1, if they exist, the adjacency matrix W is used to predict the link weights. t+1 Select the vehicle node with the largest average neighbor links, set its flag to 1, and then use the undetermined vehicles among its neighbor vehicle nodes as cached vehicles, and set the flag of the cached vehicles to 2.

[0024] If there is no connectivity Z i =0 and connectivity Z i For vehicle nodes with a connectivity of 1, select the connectivity Z. i The largest vehicle node, based on the predicted link weight adjacency matrix W t+1 Select the vehicle node with the highest average neighbor predicted link weight as the cached vehicle, and set the flag of the cached vehicle to 2;

[0025] S4.5: Let m be the cached vehicle obtained from step S4.4, and obtain the set ψ of undetermined vehicle nodes among its neighboring nodes. m If set ψ m The number of vehicles in the system is less than or equal to N. max Then the set ψ m The set of service neighbor vehicles β of cached vehicle m m Otherwise from set ψ m N in the middlemax Each vehicle constitutes a service neighbor vehicle set β m ; The obtained set of service neighbor vehicles β m Each vehicle marker in the sequence is set to position 1; return to step S4.2;

[0026] S5: For each cached vehicle m selected in step S4, a Markov decision is performed to obtain the content caching strategy for that cached vehicle, where m∈U. The system state, system action, and reward function in the Markov decision are set as follows:

[0027] System status s t =(X t B t ,σ t ),in, This represents the cache state vector of the content cached on vehicle m within cache period t, where This represents the popular content chunks f in vehicle m cached within cache period t. w The cache status, if the cached vehicle caches popular content chunks f w Then let Otherwise

[0028] This represents the set of neighboring vehicles β up to cache period t. m The request state vector of vehicle h for popular content, h∈β m ,in This indicates that vehicle h represents the segmentation of popular content f. w The request status, if vehicle h requests popular content chunk f w Then let Otherwise

[0029] This represents the set of neighboring vehicles β up to cache period t. m The cache state vector of vehicle h for popular content, where This indicates that vehicle h represents the segmentation of popular content f. w The cache status, if vehicle h caches popular content chunks f w Then let Otherwise

[0030] The system action is to cache the cache state vector of the contents of vehicle m in the next cache cycle t+1. in This represents the popular content chunk f in vehicle m cached within the next cache period t+1. w The cache state, if the cached vehicle m needs to cache popular content chunks fw Then let Otherwise And cached state vector satisfy τ m This indicates the maximum cache capacity for the vehicle m.

[0031] The reward function R is calculated as follows:

[0032] For the service neighbor vehicle set β m For each vehicle h, according to its Cache state vector And the cached state vector of the cached vehicle in the next cache period t+1 Obtain the set F of cacheable popular content for vehicle h within the next cache period t+1. h The set of cacheable popular content F is calculated. h Size L of all content in h Get the transmission rate (rate) between vehicle h and cached vehicle m. m,h The maximum amount of data g that vehicle h can acquire in the next buffer period t+1 is calculated. h =rate m,h ×Δt, and then use the following formula to calculate the transmission cost c_tr of the content transmitted from cached vehicle m to vehicle h. m,h :

[0033]

[0034] Where ρ1 represents the unit transmission cost;

[0035] Compare cache state vectors and cache state vector We obtain the set of deleted cached content χ1 and the set of added cached content χ2, and then use the following formula to calculate the adjustment cost of cached vehicle m:

[0036]

[0037] Where ρ2 represents the unit cost of deleting cached content, and ρ3 represents the unit cost of adding cached content;

[0038] The reward function R is calculated using the following formula:

[0039]

[0040] This invention presents a vehicle collaborative caching relay method based on deep reinforcement learning in the Internet of Vehicles (IoV). The popular content of the IoV is divided into blocks. Whenever the collaborative caching update cycle arrives, the movement trajectory of each vehicle in the IoV is predicted. Based on the predicted movement trajectory, the link stability between any two vehicles is calculated to obtain the predicted weight adjacency matrix. Based on the predicted weight adjacency matrix, the cached vehicles and the corresponding service vehicle set for the next caching cycle are obtained. Each cached vehicle performs Markov decision to obtain the content caching strategy of that cached vehicle.

[0041] The present invention has the following beneficial effects:

[0042] 1) This invention determines the predicted weight adjacency matrix by calculating link stability, and then solves for the cached vehicles accordingly, making the obtained cached vehicles more reasonable and improving the effectiveness of cooperative caching relay.

[0043] 2) This invention models the collaborative caching relay of caching vehicles as a Markov decision process, with the goal of minimizing system cost, thereby saving system resources while realizing collaborative caching relay. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a specific implementation of the vehicle collaborative caching relay method based on deep reinforcement learning according to the present invention.

[0045] Figure 2 This is a flowchart of the vehicle trajectory prediction method based on GNN in this embodiment;

[0046] Figure 3 This is a structural diagram of the vehicle trajectory prediction network in this embodiment;

[0047] Figure 4 This is a flowchart of the cached vehicle selection process based on load constraints in this invention. Detailed Implementation

[0048] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0049] Example

[0050] Figure 1 This is a flowchart illustrating a specific implementation of the vehicle collaborative caching relay method based on deep reinforcement learning, as described in this invention. Figure 1 As shown, the specific steps of the deep reinforcement learning-based vehicle collaborative caching relay method of the present invention include:

[0051] S101: Obtain vehicle network parameters:

[0052] In the context of vehicle networking, the popular content is F = {f1, f2, ..., f...} W}, where f w Let f represent the w-th popular content chunk, where w = 1, 2, ..., W, and W represents the number of popular content chunks. w The size is L w Let V be the number of intelligent vehicles in the entire vehicle-to-everything (V2X) area. If vehicle i requests popular content within the current cache period t, then vehicle i is the requesting vehicle; otherwise, vehicle i is not the requesting vehicle. i = 1, 2, ..., V.

[0053] S102: Vehicle movement trajectory prediction:

[0054] In the Internet of Vehicles (IoV), the high-speed mobility of vehicles causes the distance between two vehicles to constantly change, and the requested content is often not hit within the coverage area of ​​the roadside unit where the vehicle is located. Therefore, the design of the caching strategy must take into account the mobility of vehicles. In this invention, whenever the collaborative cache update cycle arrives, the movement trajectory of each vehicle in the IoV is predicted to obtain the predicted trajectory of that vehicle in the next cache cycle t+1.

[0055] The specific method for predicting vehicle trajectory can be selected according to actual needs. In this embodiment, a vehicle trajectory prediction method based on GNN is adopted. Figure 2 This is a flowchart of the vehicle trajectory prediction method based on GNN in this embodiment. Figure 2 As shown, the specific steps of the vehicle trajectory prediction method based on GNN in this embodiment include:

[0056] S201: Obtain training samples for vehicle movement trajectory:

[0057] The vehicle's historical trajectory data is sampled to obtain several consecutive position sequences of length P+Q. For the first P positions, the vehicle's coordinates (x, y, q) at each position are obtained. p ,y p Vehicle type c p p = 1, 2, ..., P, if the vehicle is located at position coordinates (x p ,y p If the vehicle is a buffer vehicle, then let c p =0, if it is a vehicle request, then let c = 0. p =1, and then form the position state vector (x p ,y p ,c p ), and construct a position state sequence from the position state vectors of the first P positions. For the last Q positions, based on each position coordinate (x... q ,yq ) constitutes the trajectory vector (x) 1 ,y 2 ,…,x q ,y q ,…,x Q ,y Q ).

[0058] S202: Constructing a vehicle trajectory prediction network:

[0059] In this embodiment, the vehicle trajectory prediction network is constructed using an LSTM (Long Short-Term Memory) network. Figure 3 This is a structural diagram of the vehicle trajectory prediction network in this embodiment. For example... Figure 3 As shown, the vehicle trajectory prediction network in this embodiment includes a first LSTM network, a feature fusion module, and a second LSTM network, wherein:

[0060] The first LSTM network is used to process the input location state sequence to obtain an interactive event recognition result sequence with the same length as the location state sequence, and then input it into the feature fusion module.

[0061] The feature fusion module is used to superimpose the location state sequence and the interaction event recognition result sequence to obtain the fused feature sequence, which is then input into the second LSTM network.

[0062] The second LSTM network is used to process the fused feature sequence to obtain the predicted trajectory vector.

[0063] S203: Training the vehicle trajectory prediction network:

[0064] Using the position state sequence in step S201 as input and the corresponding trajectory vector as the expected output, the vehicle trajectory prediction network is trained to obtain the trained vehicle trajectory prediction network.

[0065] S204: Vehicle movement trajectory prediction:

[0066] The system obtains the P nearest location coordinates and vehicle type of the vehicle at the current moment, forming a position state sequence. This sequence is then input into a trained vehicle trajectory prediction network to obtain a predicted trajectory vector. The coordinates of each predicted location are then extracted from this vector to form the predicted trajectory.

[0067] S103: Construct the predicted weight adjacency matrix:

[0068] To obtain the connectivity between vehicles at the next time step t+1, this invention calculates the link stability between vehicles based on the vehicle movement trajectories predicted in step S102, thereby obtaining the predicted weighted adjacency matrix. The specific method is as follows:

[0069] For any two vehicles i and j, i,j=1,2,…,V, the distance between vehicles i and j in the next buffer period t+1 is calculated based on the predicted trajectories of these two vehicles in the next buffer period t+1. Then, the normalized communication distance tolerance between vehicles i and j in the next buffer period t+1 is calculated using the following formula.

[0070]

[0071] Among them, R veh This represents the communication radius of the vehicles. When the distance between the vehicles is greater than or equal to the communication radius, the communication distance tolerance between the two vehicles is considered to be 0, meaning communication is impossible. min This represents the minimum distance between two vehicles under the safe distance limit, when d min A larger value indicates that the two vehicles are closer together, the channel quality is better under the same obstruction conditions, and the link duration is longer when the speeds of the two vehicles remain constant.

[0072] Based on the predicted trajectories of the two vehicles in the next buffer period t+1, the duration of mutual coverage between vehicles i and j in the next buffer period t+1 is calculated as the link duration between vehicles i and j. A longer link duration indicates a more stable topology between the two vehicles. Assuming a buffer period interval of Δt, since the link duration is updated in each period, when its value exceeds the buffer period interval, the upper limit of the link duration is set to Δt and normalized. The normalized link duration can then be calculated using the following formula.

[0073]

[0074] Then, the link stability between vehicles i and j within the next buffer period t+1 is calculated using the following formula.

[0075]

[0076] Where α∈[0,1] represents the preset weighting factor.

[0077] Link stability As the predicted link weights, an N×N predicted link weight adjacency matrix is ​​constructed.

[0078] S104: Filtering cached vehicles based on the minimum dominating set algorithm:

[0079] To reduce the overhead of managing cache nodes and minimize channel contention among them, the number of cache nodes selected should be minimized to achieve optimal performance. Therefore, this invention uses vehicles as nodes and predicts the link weight adjacency matrix W. t+1 The minimum dominating set of graph G is obtained by using the adjacency matrix to form the graph G.

[0080] In this invention, since the service capacity of each cache vehicle is limited, the maximum number N of request vehicles that a cache vehicle can respond to must also be considered when solving for the minimum dominance set. max This is used as a load constraint to filter cached vehicles. Figure 4 This is a flowchart of the cached vehicle selection process based on load constraints in this invention. Figure 4 As shown, the specific steps of the load-constrained cache vehicle screening in this invention include:

[0081] S401: Initialize vehicle status:

[0082] Initialize the state of each vehicle as an undetermined vehicle, that is, set the flag bit of the vehicle to 0 in the next cache cycle t+1.

[0083] S402: Determine if there are any vehicles with undetermined status. If yes, proceed to step S403; otherwise, end the cached vehicle filtering process.

[0084] S403: Calculate the connectivity degree of vehicle nodes with undetermined connection states:

[0085] For each vehicle whose state is currently undetermined, calculate its connectivity Z in graph G, which connects to the nodes of vehicles with undetermined states. i , i∈Y, where Y represents the set of vehicles whose current state is undetermined.

[0086] S404: Select cache node:

[0087] The following method is used to select cached vehicles from the current undetermined vehicle set Y:

[0088] If the connectivity Z exists i For vehicle nodes with a value of 0, the vehicle will be designated as a cached vehicle, and its flag will be set to 2.

[0089] If there is no connectivity Z i For vehicle nodes with a degree of 0, determine if they have a connectivity Z. i For vehicle nodes with a value of 1, if they exist, the adjacency matrix W is used to predict the link weights. t+1 Select the vehicle node with the largest average neighbor links, set its flag to 1, and then use the undetermined vehicles among its neighbor vehicle nodes as cached vehicles, setting the flag of the cached vehicles to 2.

[0090] If there is no connectivity Z i =0 and connectivity Z i For vehicle nodes with a connectivity of 1, select the connectivity Z. i The largest vehicle node, based on the predicted link weight adjacency matrix W t+1 The vehicle node with the highest average neighbor predicted link weight is selected as the cached vehicle, and the flag of the cached vehicle is set to 2.

[0091] S405: Obtain the set of service neighbor vehicles:

[0092] Let m be the cached vehicle obtained from step S404, and obtain the set ψ of undetermined vehicle nodes among its neighbor nodes. m If set ψ m The number of vehicles in the system is less than or equal to N. max Then the set ψ m The set of service neighbor vehicles β of cached vehicle m m Otherwise from set ψ m N in the middle max Each vehicle constitutes a service neighbor vehicle set β m The obtained service neighbor vehicle set β m Each vehicle marker is positioned at 1. Return to step S402.

[0093] To make the service neighbor vehicle set more reasonable, this embodiment starts from set ψ m The following method is used to filter vehicles: Prioritize the set ψ according to the following order: connectivity of undetermined vehicle nodes from smallest to largest; the ranking of the predicted link weight of cached vehicle m with its neighboring vehicle nodes among all predicted link weights of its neighboring vehicle nodes from highest to lowest; and the predicted link weights from largest to smallest. m The neighboring vehicle nodes are sorted as follows: First, they are sorted by connectivity score from smallest to largest. If neighboring vehicle nodes have the same connectivity score, their predicted link weights are sorted from largest to smallest. This yields the ranking of the cached vehicle m relative to its neighboring vehicle node's predicted link weight. These rankings are then sorted from highest to lowest. If neighboring vehicle nodes have the same ranking, they are sorted from largest to smallest predicted link weight. The top N neighboring vehicle nodes are selected from the resulting sequence. max The vehicle nodes constitute the service neighbor vehicle set β m .

[0094] As can be seen from the above process, in the load-constrained cache vehicle selection of the present invention, under the premise of ensuring that each ordinary node is covered by only one cache node, each cache node selects the neighbor vehicle node with the largest predicted link weight within its coverage area as the service neighbor node, thereby obtaining the best cache service performance.

[0095] S105: Determine the collaborative caching strategy:

[0096] To determine the collaborative caching strategy, this invention employs a deep reinforcement learning algorithm to solve the content update problem for each cache vehicle. Specifically, each cache vehicle m selected in step S104 undergoes a Markov decision process to obtain its content caching strategy, where m∈U. The following sections describe the system state, system actions, and reward function of the Markov Decision Process (MDP) in this invention:

[0097] System State: In this invention, system state describes the cache state and request state at different times. Within the cache period t, state s t The cache state X containing the content of cached vehicle m. t Request the vehicle to state each content f. Content availability status σ t The caching decision depends on the content already cached in the cache vehicle and the content of the current request. In other words, the system state s in this invention... t It can be represented as:

[0098] s t =(X t B t ,σ t )

[0099] in, This represents the cache state vector of the content cached on vehicle m within cache period t, where This represents the popular content chunks f in vehicle m cached within cache period t. w The cache status, if the cached vehicle caches popular content chunks f w Then let Otherwise

[0100] This represents the set of neighboring vehicles β up to cache period t. m The request state vector of vehicle h for popular content, h∈β m ,in This indicates that vehicle h represents the segmentation of popular content f. w The request status, if vehicle h requests popular content chunk f w Then let Otherwise

[0101] This represents the set of neighboring vehicles β up to cache period t. m The cache state vector of vehicle h for popular content, where This indicates that vehicle h represents the segmentation of popular content f. w The cache status, if vehicle h caches popular content chunks f w Then let Otherwise

[0102] System Action: In this invention, the system action is the cache state vector of the contents of vehicle m within the next cache period t+1. in This represents the popular content chunk f in vehicle m cached within the next cache period t+1. w The cache state, if the cached vehicle m needs to cache popular content chunks f w Then let Otherwise And cached state vector satisfy τ m This indicates the maximum cache capacity for the cached vehicle m.

[0103] Reward Function: In this invention, the reward function is constructed based on the cost incurred by the cache vehicle in providing popular content transmission to vehicles in the service neighbor vehicle set. The cost in this invention is divided into two types: one is the transmission cost incurred by the cache vehicle in transmitting popular content chunks to vehicles in the service neighbor vehicle set; the other is the adjustment cost incurred by the cache vehicle in adjusting its own cached content to suit the needs of vehicles in the service neighbor vehicle set.

[0104] In terms of transmission cost, for the set of neighboring vehicles β m For each vehicle h, according to its Cache state vector And the cached state vector of the cached vehicle in the next cache period t+1 Obtain the set F of cacheable popular content for vehicle h within the next cache period t+1. h The set of cacheable popular content F is calculated. h Size L of all content in h Get the transmission rate (rate) between vehicle h and cached vehicle m. m,h The maximum amount of data g that vehicle h can acquire in the next buffer period t+1 is calculated. h =rate m,h ×Δt, and then use the following formula to calculate the transmission cost c_tr of the content transmitted from cached vehicle m to vehicle h. m,h:

[0105]

[0106] Where ρ1 represents the unit transmission cost.

[0107] In terms of adjustment costs, compared to cached state vectors and cache state vector We obtain the set of deleted cached content χ1 and the set of added cached content χ2, and then use the following formula to calculate the adjustment cost of cached vehicle m:

[0108]

[0109] Where ρ2 represents the unit cost of deleting cached content, and ρ3 represents the unit cost of adding cached content.

[0110] Therefore, the formula for calculating the reward function R can be expressed as follows:

[0111]

[0112] To improve efficiency and accuracy, this embodiment employs a Recursive Deep Reinforcement Learning (RDRL-CR) algorithm to solve for the content caching policy of the cached vehicle. The RDRL-CR method mainly consists of two neural networks: an Actor network to approximate the policy and a Critic network to approximate the value function. At each time step, the RDRL-CR algorithm uses the Actor network to select an action for the current state and the Critic network to estimate the value function of the current state. RDRL-CR uses the Bellman equation to update the Critic network and the Actor network to execute the policy gradient algorithm to maximize long-term cumulative reward.

[0113] Actor Network: The output of the Actor Network is the caching strategy π = {π1, π2, ..., π} for the action a corresponding to the current state s. M}, π k This describes the strategy for caching vehicle k, where k = 1, 2, ..., M, and M represents the number of vehicles to be cached. Cached vehicle k selects an action a. t , depends on its state s t and strategy

[0114]

[0115] Critic Network: During the training phase, the action value function V(S) is used to approximate the Critic network, caching the actions performed by vehicles in the environment. and the current status information s tand the rewards R obtained from the environment t The information is sent to the Critic network, which then sends back the state information s for the next time step. t+1 and rewards. For each cached vehicle, the Q function is defined as follows: θ k This represents the network parameters corresponding to vehicle k in the cache, which solves the problem caused by non-stationary environments. The state transition function is:

[0116]

[0117] Compile the TD objective using the current policy network and the objective value function network:

[0118]

[0119] Where γ is the discount factor for future cumulative rewards, 0≤γ<1.

[0120] Individuals optimize the policy directly, setting the policy parameters to... In this case, the goal is to maximize the reward. Therefore, the objective is to maximize the cumulative reward function.

[0121]

[0122] Historical information from the Actor and Critic networks is respectively used as follows: and This indicates that the individual agent is based on the previous state. Select an action, that is Then the Q function becomes in Loss function in Critic network yes:

[0123]

[0124]

[0125] The objective function is expressed as:

[0126] The caching vehicle receives user requests and retrieves their characteristics. It then provides the current request and cache state to the network to initiate caching operations. After executing the action generated by the policy, each vehicle receives a reward and its next state. Information received from the environment is stored as a historical record using LSTM.

[0127]

[0128]

[0129] The Actor and Critic networks store historical data in a replay sample pool. To train the Actor and Critic networks, a sample set S is randomly selected from the replay sample pool, and the target Critic network is set up as follows:

[0130]

[0131] Where j∈S.

[0132] And update its parameter θ by reducing the loss function:

[0133]

[0134] The actor calculates the policy gradient using a loss function and updates the parameters using mini-batch gradients.

[0135]

[0136] ,

[0137] Update the Actor and Critic parameters of the target network:

[0138]

[0139]

[0140] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A vehicle collaborative caching relay method based on deep reinforcement learning in the Internet of Vehicles, characterized in that, Includes the following steps: S1: A record of popular content in the Internet of Vehicles (IoV) ,in Indicates the first Popular content segments , This represents the number of chunks in popular content, and is used to record the chunks. The size is The number of intelligent vehicles in the entire vehicle-to-everything (V2X) area is denoted as . If the vehicle In the current cache cycle If popular content is requested, then the vehicle... Request a vehicle, otherwise vehicle It is not a request for a vehicle. ; S2: Whenever the collaborative cache update cycle arrives, predict the movement trajectory of each vehicle in the vehicle network to obtain the vehicle's position in the next cache cycle. Predicted trajectory within; S3: For any two vehicles and , According to the next cache cycle The predicted trajectories of these two vehicles are used to calculate the next buffer cycle. Inner vehicle and Distance between Then, the next cache cycle is calculated using the following formula. Inner vehicle and Normalized communication distance tolerance between : ； in, Indicates the vehicle's communication radius. This indicates the minimum distance between two vehicles under the safe distance limit; According to the next cache cycle The predicted trajectories of these two vehicles are used to calculate the next buffer cycle. Inner vehicle and The duration of mutual coverage as a vehicle and Link duration between : ； in, Indicates the cache cycle interval; The next cache cycle is then calculated using the following formula. Inner vehicle and Link stability : ； in, This represents the preset weighting factor; Link stability As an element, it can construct a structure of size Predicted weight adjacency matrix ; S4: Treat vehicles as nodes and predict the weighted adjacency matrix. As an adjacency matrix, it forms a graph. The graph is obtained by using the minimum dominator set algorithm. Minimal dominating set The specific steps of the minimum dominator set algorithm are as follows: The vehicle nodes contained within are used as cached vehicles. S4.1: Initialize the state of each vehicle as undetermined, and let the vehicle be in the next cache cycle. The internal flag is 0; S4.2: Determine if there are any vehicles with undetermined status. If yes, proceed to step S4.3; otherwise, end the cached vehicle filtering process. S4.3: For each vehicle in an undetermined state, calculate its position in the graph. Connectivity of vehicle nodes with undetermined connection status , , This represents the set of vehicles whose current state is undetermined. S4.4: Use the following method to select from the current set of undetermined vehicles. Select cached vehicles: If connectivity exists If the vehicle node is a cached vehicle, then the vehicle will be used as a cached vehicle and the flag will be set to 2; If there is no connectivity For vehicle nodes, determine whether connectivity exists. If a vehicle node exists, it is determined based on the predicted link weight adjacency matrix. Select the vehicle node with the largest average neighbor links, set its flag to 1, and then use the undetermined vehicles among its neighbor vehicle nodes as cached vehicles, and set the flag of the cached vehicles to 2. If there is no connectivity and connectivity For vehicle nodes, select connectivity. The largest vehicle node, based on the predicted link weight adjacency matrix. Select the vehicle node with the highest average neighbor predicted link weight as the cached vehicle, and set the flag of the cached vehicle to 2; S4.5: Let the cached vehicles obtained from step S4.4 be denoted as Get the set of undetermined vehicle nodes among its neighboring nodes. If set The number of vehicles in the country is less than or equal to Then the set As a cache vehicle Service Neighbor Vehicle Collection Otherwise from the set Screening Each vehicle constitutes a service neighbor vehicle set ; collect the service neighbor vehicles obtained Each vehicle marker in the sequence is set to position 1; return to step S4.2; S5: Each cached vehicle obtained from step S4 Markov decisions are performed separately to obtain the content caching strategy for the cached vehicle. In Markov decision making, the system state, system action, and reward function are set as follows: System status ,in, Indicates cache duration Internal cache vehicles The cached state vector of the above content, where Indicates cache duration Internal cache vehicles Popular content segmentation The cache status, if the cached vehicle caches popular content chunks Then let Otherwise ; , Indicates the period up to the cache period. Service Neighbor Vehicle Collection China Vehicle The request state vector for popular content. ,in Indicates vehicle Segmenting popular content The request status, if the vehicle Requested popular content chunks Then let Otherwise ; , Indicates the period up to the cache period. Service Neighbor Vehicle Collection China Vehicle The cached state vector for popular content, where Indicates vehicle Segmenting popular content The cache state, if the vehicle Popular content chunks are cached Then let Otherwise ; The system action is for the next cache cycle. Internal cache vehicles Cache state vector of the above content ,in Indicates the next cache cycle Internal cache vehicles Popular content segmentation The cache state, if the cached vehicle Popular content needs to be cached in chunks Then let Otherwise And cache state vector satisfy , Indicates cached vehicles The maximum cache capacity; reward function The calculation method is as follows: Service neighbor vehicle collection Each vehicle According to its Cache state vector And cached vehicles in the next cache cycle Internal cache state vector Get the next cache cycle Inner vehicle A collection of cacheable popular content The set of cacheable popular content is calculated. Size of all content in Obtain vehicle and cached vehicles Transmission rate between Calculations are performed to determine the cache cycle in the next cache period. Inner vehicle Maximum achievable data transfer volume Then, the cached vehicles are calculated using the following formula. To the vehicle Transmission cost of transmitted content : ； in, Indicates the unit transmission cost; Compare cache state vectors and cache state vector Get the set of deleted cached content and increase the cache content set Then, the cached vehicles are obtained using the following calculation formula. Adjustment costs: ； in, This represents the unit cost of deleting cached content. This represents the unit cost of adding cached content. The reward function is calculated using the following formula. : 。 2. The vehicle collaborative buffer relay method according to claim 1, characterized in that, The specific steps for predicting the vehicle movement trajectory in step S2 include: S2.1: Sample the historical movement trajectory data of the vehicle to obtain several data points with a length of... A sequence of consecutive positions; for the first Get the vehicle's coordinates at each location. Vehicle type , If the vehicle is located at the coordinates If the vehicle is in a buffer state, then... If it is a request for a vehicle, then order Then, the position state vector is constructed. and will the former The position state vectors of each position constitute a position state sequence; for the later... Each location, based on its coordinates. Constructing a trajectory vector ; S2.2: Construct a vehicle trajectory prediction network, including a first LSTM network, a feature fusion module, and a second LSTM network, wherein: The first LSTM network is used to process the input location state sequence to obtain an interactive event recognition result sequence with the same length as the location state sequence, and then input it into the feature fusion module. The feature fusion module is used to superimpose the location state sequence and the interaction event recognition result sequence to obtain a fused feature sequence, which is then input into the second LSTM network. The second LSTM network is used to process the fused feature sequence to obtain the predicted trajectory vector; S2.3: Using the position state sequence in step S2.1 as input and the corresponding trajectory vector as the expected output, train the vehicle trajectory prediction network to obtain the trained vehicle trajectory prediction network. S2.4: Obtain the vehicle's most recent... The location coordinates and vehicle type constitute a position state sequence, which is then input into a trained vehicle trajectory prediction network to obtain a predicted trajectory vector. The coordinates of each predicted position are then extracted from the vector to form the predicted trajectory.

3. The vehicle collaborative buffer relay method according to claim 1, characterized in that, In step S4.5, from the set The specific method for filtering vehicles is as follows: Based on the connectivity of vehicle nodes with undetermined connection states from smallest to largest, cached vehicles are selected. The priority set of predicted link weights of the neighboring vehicle nodes, ranked from high to low and from large to small, is used to determine the order of priority of predicted link weights among all predicted link weights of the neighboring vehicle nodes. Sort the neighboring vehicle nodes in the sequence, and select the top-ranked neighboring vehicle nodes from the resulting sequence. Each vehicle node constitutes a set of serving neighbor vehicles.

4. The vehicle collaborative buffer relay method according to claim 1, characterized in that, The content caching strategy in step S5 is solved using a recursive deep reinforcement learning algorithm.

Images