A federated reinforcement learning-based edge content caching decision method for internet of vehicles
By employing federated reinforcement learning in vehicle-to-everything (V2X) networks, vehicle agents interact with roadside units to acquire information, independently decide on the caching strategy of edge node clusters, and leverage the privacy protection advantages of federated learning to address the shortcomings of centralized network architectures, achieving efficient edge content caching and privacy protection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2022-12-29
- Publication Date
- 2026-07-21
Smart Images
Figure CN116156455B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of wireless communication technology, and in particular to a method for edge content caching decision-making in vehicle-to-everything (V2X) networks based on federated reinforcement learning. Background Technology
[0002] In recent years, driven by sixth-generation mobile communication technology, emerging fields such as the Internet of Vehicles (IoV), big data, and artificial intelligence have deeply integrated with new-generation information technologies, providing efficient and reliable communication services. However, with the increasing number of vehicles, the growing real-time communication services place higher demands on the ultra-low latency and ultra-high reliability of IoV. To address these challenges, edge caching technology deploys cache resources at edge nodes to complete local content distribution, avoiding reliance on centralized delivery in the cloud, thereby reducing content delivery latency (Zhang Y, Zhao J, Cao G. Roadcast: a popularity aware content sharing scheme in vanets[J]. ACMSIGMOBILE Mobile Computing and Communications Review, 2010, 13(4): 1-14.). However, due to the limited storage capacity of edge nodes, the rapidly increasing data requests for IoV applications cannot be fully cached at edge nodes; secondly, given the high mobility, frequently changing content requirements, and harsh communication environment of IoV, it is necessary to design efficient edge content caching methods for IoV scenarios.
[0003] Considering that the intelligent edge content caching problem in the Internet of Vehicles is essentially a model-free discrete sequence decision problem, a multi-agent reinforcement learning method that can complete local decision-making by sharing training information can be used to solve it. Compared with traditional optimization algorithms, deep reinforcement learning can learn from the interaction of agents with uncertain environments to solve dynamic decision problems. Even if the dynamic environmental changes cannot be predicted in advance, the agent can still learn how to take action or how to map the acquired information to actions to maximize the system reward. In recent years, researchers at home and abroad have focused on studying intelligent edge content caching decision-making to efficiently utilize edge caching resources in the dynamic wireless transmission environment of the Internet of Vehicles. For example, Qiao et al. used a deep deterministic policy gradient algorithm to learn the changing patterns of the wireless environment of the Internet of Vehicles by utilizing the local observation information of vehicle users, and proposed a cooperative edge caching method to minimize the long-term trade-off between system content transmission latency and caching overhead (Qiao G, Leng S, Maharjan S, et al. Deep reinforcement learning for cooperativecontent caching in vehicular edge computing and networks[J]. IEEE Internet of Things Journal, 2019, 7(1): 247-257.). However, most existing vehicle-to-everything (V2X) edge content caching solutions are built on centralized network architectures, failing to fully utilize the dense, heterogeneous deployment characteristics of edge nodes. Furthermore, considering the open nature of V2X communication methods, the privacy protection of individual vehicle user data is also a crucial concern. Therefore, in an environment with dense, heterogeneous edge node deployments, how to achieve high capacity, low caching overhead, and seamless in-vehicle communication coverage using a decentralized network architecture while maintaining privacy protection still requires further research. Summary of the Invention
[0004] To overcome the problems existing in related technologies, this invention discloses an embodiment of a method for edge content caching in vehicle-to-everything (V2X) networks based on federated reinforcement learning. The technical method is as follows:
[0005] Step 1: Input the vehicle network environment, each vehicle intelligent agent initializes the parameters of its own actor network and critic network, and models the optimization problem;
[0006] Step 2: In the current time slot, each vehicle agent interacts with the roadside units within the observation range to obtain observation information such as the distance between itself and the roadside units, the buffer status of the roadside units, and the remaining buffer capacity of the roadside units.
[0007] Step 3: Based on local observation information, each vehicle agent can independently decide on the roadside units associated with the edge node cluster, and decide within the cluster whether to cache the request content of the current time slot.
[0008] Step 4: After executing the action decision, each vehicle intelligent agent obtains the reward for the total system content delivery latency and edge caching overhead from the feedback of the vehicle network environment, and at the same time, all sample data is cached to the experience reuse pool.
[0009] Step 5: Determine if the sample size is sufficient. If yes, proceed to Step 6; otherwise, proceed to Step 7.
[0010] Step 6: When the number of samples is sufficient, each vehicle agent updates its Actor network and Critic network parameters according to the flexible actor-critic algorithm;
[0011] Step 7: The aggregation center collects the Actor network weight parameters of each vehicle agent and performs federated aggregation. The aggregated parameters are broadcast to the vehicle users in one training round for local training.
[0012] Step 8: Determine if the current training round has ended. If not, return to step 2 to start the next training round. If yes, proceed to step 9.
[0013] Step 9: Determine if convergence has occurred. If not, reset the vehicle network environment and return to Step 1; if yes, training ends, completing the vehicle network edge content caching decision.
[0014] Compared with the prior art, the significant advantages of this invention are: (1) In response to the link congestion load caused by the centralized network architecture in the vehicle network caching scenario, this invention utilizes the deployment characteristics of dense edge nodes to design a user-centric edge node cluster to achieve high capacity, low cache overhead and seamless coverage of vehicle communication; (2) In response to the problem of a large amount of information interaction and privacy leakage caused by centralized training of intelligent algorithms, this invention utilizes the privacy protection advantages of federated learning to achieve collaborative decision-making by vehicle users through sharing the weights of the local model neural network, so as to reduce the long-term transmission latency of system content and edge cache overhead.
[0015] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description
[0016] Figure 1 This is a flowchart of the edge content caching decision method for vehicle-to-everything (V2X) networks based on a federated reinforcement learning framework, as described in this invention.
[0017] Figure 2 This is a graph showing the variation of average trade-off convergence performance per vehicle user over a period of time in an embodiment of the present invention.
[0018] Figure 3 This is a graph showing the variation of average transmission delay convergence performance per vehicle user with the period in an embodiment of the present invention.
[0019] Figure 4 This is a graph showing the change in average cache overhead convergence performance per vehicle user over a period of time in an embodiment of the present invention.
[0020] Figure 5 This is a graph showing how the average trade-off convergence performance per vehicle user varies with the maximum number of associations in an embodiment of the present invention. Detailed Implementation
[0021] This invention proposes a federated reinforcement learning-based edge content caching decision-making method for vehicle-to-everything (V2X) networks. Specifically, within a unit time slot, each vehicle user is treated as an agent. By acquiring observational information such as the distance between the vehicle and roadside units (BSUs), the caching status of the BSUs, and the remaining caching capacity of the BSUs within the observation range, a neural network is used to train an action strategy to minimize the trade-off between the total system content delivery latency and edge caching overhead. Figures 1-2 This includes the following steps:
[0022] Step 1: Input the vehicle network environment, each vehicle intelligent agent initializes the parameters of its own actor network and critic network, and models the optimization problem;
[0023] Step 2: In the current time slot, each vehicle agent interacts with the roadside units within the observation range to obtain observation information such as the distance between itself and the roadside units, the buffer status of the roadside units, and the remaining buffer capacity of the roadside units.
[0024] Step 3: Based on local observation information, each vehicle agent can independently decide on the roadside units associated with the edge node cluster, and decide within the cluster whether to cache the request content of the current time slot.
[0025] Step 4: After executing the action decision, each vehicle intelligent agent obtains the reward for the total system content delivery latency and edge caching overhead from the feedback of the vehicle network environment, and at the same time, all sample data is cached to the experience reuse pool.
[0026] Step 5: Determine if the sample size is sufficient. If yes, proceed to Step 6; otherwise, proceed to Step 7.
[0027] Step 6: When the number of samples is sufficient, each vehicle agent updates its Actor network and Critic network parameters according to the flexible actor-critic algorithm;
[0028] Step 7: The aggregation center collects the Actor network weight parameters of each vehicle agent and performs federated aggregation. The aggregated parameters are broadcast to the vehicle users in one training round for local training.
[0029] Step 8: Determine if the current training round has ended. If not, return to step 2 to start the next training round. If yes, proceed to step 9.
[0030] Step 9: Determine if convergence has occurred. If not, reset the vehicle network environment and return to Step 1; if yes, training ends, completing the vehicle network edge content caching decision.
[0031] As one specific implementation method, the input of the vehicle networking environment in step 1 specifically includes:
[0032] (1) Time-slot model: The continuous training time is discretized into multiple time slots, represented as follows: The duration of each time slot is The channel state information and system parameters remain unchanged over the duration of a single time slot, but may vary randomly between different time slots;
[0033] (2) Network Model: The vehicle-to-everything (V2X) network model is established as a Manhattan grid model, in which roadside units that can provide communication services are evenly distributed on both sides of the road; the roadside units, as edge nodes, have dedicated communication resources and limited local storage resources, and are connected to the edge server through a high-speed wired link; the edge server is controlled by a software-defined centralized controller, which can perform edge association and cache resource allocation, etc.; let the set of roadside units be . The vehicle group is All vehicle users can travel in four directions along the road, with multiple lanes in each direction to ensure vehicle passage.
[0034] (3) Vehicle movement model: The speed changes of the vehicle all follow the following Gaussian-Markov stochastic process. Specifically, when the vehicle user With initial velocity During travel, it is in a time slot speed at This can be represented as a time slot. speed at The sum of the asymptotic rate and a random variable:
[0035]
[0036] in, and Vehicle users The corresponding asymptotic mean and standard deviation of the velocity; parameters It represents the memory depth of the previous time slot speed and determines the vehicle user's... Temporal correlation of motion; This represents a standard normal distribution with zero mean and unit variance, indicating no correlation. It is worth noting that... As it approaches 1, vehicle users The current time slot velocity becomes more dependent on the velocity of the previous time slot;
[0037] (4) Content Request Model: Let This represents the collection of all vehicle request content. This indicates the total size of all requested content. These are characteristic values representing not all request content, used to distinguish different request contents; assuming a vehicle user... In the time slot Only one request content can be generated. , represented as
[0038]
[0039] in, Indicates vehicle user In the time slot Request content ,on the contrary Considering that vehicle users may prefer files similar to those requested in previous time slots, in addition to requesting content based on global popularity, we assume that vehicle users request content based on... The probability is based on global popularity. The probability is based on local personal preference, where global popularity and local personal preference are defined as follows:
[0040] ① Global popularity: Let This represents the global popularity of user requests for files for each vehicle, which follows a Mandelbrot-Zipf distribution, i.e., satisfies...
[0041]
[0042] in, Content Sort in descending order of global popularity; and These represent the platform factor and the skewness factor, respectively.
[0043] ② Local Personal Preferences: In this case, each vehicle user requests files based on the similarity of their previously requested content; for example, vehicle users in different time slots... Request content Then in the time slot Request and content The content with the highest similarity; this section uses cosine similarity to represent the content. and Similarity:
[0044]
[0045] (5) User-Centric Edge Caching Model: To improve the transmission rate of requested content by vehicle users, a user-centric network architecture is adopted to design the vehicle-to-everything (V2X) edge content caching framework. Specifically, each vehicle user observes the information of roadside units within its perception range, thereby selecting one or more nearby roadside units to construct user-centric edge node clusters; enabling vehicle users to... In the time slot The maximum observable number of roadside units is The maximum number of roadside units that can be associated is expressed as ;make Indicates time slot Time service vehicle users The edge node cluster, then This indicates the number of roadside units in the cluster at this time; it allows vehicle users to... In the time slot The edge association decision is represented as:
[0046]
[0047] in, Indicates roadside unit In the time slot Belongs to vehicle users edge node cluster ,on the contrary Furthermore, a user-centric edge node cluster can be represented as...
[0048]
[0049] Assuming both roadside units and the cloud center have cache capacity, with the cloud center's cache capacity sufficient to handle cache requests from all vehicle users, and each roadside unit having a limited cache capacity... In a user-centric edge content caching framework, vehicle users may request that some of their requested files be cached in their associated edge node cluster to provide ultra-high-speed cached content services. Therefore, vehicle users need to further decide whether to cache the request content for the current time slot in each roadside unit within the edge node cluster; thus, the time slot... Vehicle users The edge caching decision variables are represented as
[0050]
[0051] in, Indicates vehicle user time slot The requested content is cached in the roadside unit. ,on the contrary ;
[0052] (6) Wireless transmission model: It is assumed that mutual interference between communication links has been eliminated by allocating orthogonal resource blocks, and that both roadside units and vehicle users are equipped with single antennas; considering that channel power gain includes both fast fading and slow fading, the main cause of fast fading is multipath effect, i.e., Rayleigh fading; the slow fading consists of path loss and shadowing fading; let Indicates in time slot Vehicle users With roadside units The channel gain between them is expressed as:
[0053]
[0054] in, This corresponds to the fast fading portion, which follows a complex Gaussian distribution with zero mean and unit variance, i.e., satisfies... Vehicle users With roadside units The slow fading part is represented as
[0055]
[0056] in, This is a constant in path loss; It is a shadow fading with standard deviation The log-normal components; Indicates time slot vehicle users With roadside units Distance between; The attenuation coefficient representing the path loss component; Indicates in time slot vehicle users With roadside units Path loss of the channel between them;
[0057] Time slot Edge node cluster Central side unit The transmission power is Then vehicle user The achievable downlink transmission rate is expressed as
[0058]
[0059] in, For channel bandwidth, The power of additive white Gaussian noise;
[0060] (7) Latency Model: Let the tolerable latency of each vehicle user's request content be... In addition, to fully utilize edge caching resources, the edge server periodically... Its storage space is automatically refreshed, therefore the edge server experiences brief service downtime, occurring every certain period. Replace the cached content; therefore, if the vehicle user In the time slot When cached content needs to be downloaded from an edge server, its delivery must be within tolerable latency. And it will be completed within the next cache replacement and refresh cycle, that is
[0061]
[0062] In the formula, This indicates the number of time slots remaining in the corresponding cache refresh cycle. ;
[0063] Considering that the content delivery latency for vehicle users depends on whether the requested content has been pre-cached on the edge server, time slots are defined. Roadside Unit The cache status is
[0064]
[0065] in, Content In the time slot Cached to roadside unit In the middle, or vice versa Therefore, the delivery of requested content to vehicle users includes both edge caching and cloud downloading:
[0066] ① Edge caching: When an edge node cluster... The requested content has been cached at any edge node. At that time, vehicle users The requested content can be obtained directly from the edge, and its latency is expressed as
[0067]
[0068] ② Cloud download: When the edge node cluster None of the edge nodes in the middle cached the requested content. At times, edge servers need to download from the cloud center, resulting in additional fixed latency. ;
[0069] Based on the above model, vehicle users Get the request content The total delay can be expressed as
[0070]
[0071] in, This indicates that the value is set to 1 if the constraint is met, and 0 otherwise.
[0072] As a specific implementation method, the modeling of the optimization problem in step 1 specifically includes:
[0073] Considering minimizing the long-term trade-off between content delivery latency and edge caching overhead as the optimization objective, a user-centric intelligent edge caching scheme is designed. To this end, the normalized content delivery latency utility and edge caching overhead utility are first designed, expressed as follows:
[0074]
[0075]
[0076] Therefore, the optimization problem is formulated as follows:
[0077]
[0078]
[0079]
[0080]
[0081]
[0082] in, and These represent the weights of content delivery latency utility and caching overhead utility, respectively; This indicates that the size of the edge node cluster for each vehicle user in any time slot does not exceed [a certain value]. ;Mode This indicates that each time slot can serve at most a single vehicle user; This indicates that the size of the cached content of each roadside unit cannot exceed its local cache capacity; This means that the delivery delay of the content requested by each vehicle user in any time slot should not exceed the maximum allowable delay.
[0083] In one specific implementation, step 2 involves each vehicle agent interacting with roadside units within its observation range to obtain observation information such as the distance between itself and the roadside units, the buffer status of the roadside units, and the remaining buffer capacity of the roadside units. Specifically:
[0084] In the time slot Each vehicle user acts as an intelligent agent, obtaining their own observed state through interaction with the environment. In the time slot The observation state is represented as
[0085]
[0086] in, Indicates time slot vehicle users With roadside units The distance between them; Indicates time slot Roadside Unit Has the requested content been cached? The flag bit; Indicates time slot Roadside Unit Remaining cache space size; Indicates time slot Roadside Unit Number of time slots remaining until the cache space is refreshed.
[0087] As one specific implementation, step 3, which involves each vehicle agent independently deciding on the associated roadside units in the edge node cluster based on local observation information, and deciding within the cluster whether to cache the request content of the current time slot, specifically involves:
[0088] Determine the actions taken by each vehicle agent in its interaction with the environment, including edge-related decision variables and edge-cached decision variables. Then the vehicle... In the time slot The action is represented as:
[0089]
[0090] As a specific implementation method, after the action decision is executed in step 4, the reward for each vehicle intelligent agent to obtain the total system content delivery latency and edge caching overhead feedback from the vehicle network environment is as follows:
[0091] After all vehicle agents have completed their actions, the environment will provide a global reward. The average trade-off for each user is defined as the global reward, expressed as:
[0092]
[0093] in, This represents the normalized content delivery delay effect. This represents the utility of normalized edge cache overhead; This indicates that the constraint is not satisfied. The penalty items at that time.
[0094] As one specific implementation, step 6 involves each vehicle agent updating its actor and critic network parameters according to the federated discrete flexible actor-critic algorithm, specifically as follows:
[0095] (1) Flexible value function: The optimization objective of the flexible actor-critic algorithm, in addition to maximizing the cumulative reward value returned by training, also needs to maximize entropy, that is...
[0096]
[0097] in, Representation strategy The trajectory distribution; This represents the entropy coefficient; it is used to ensure that the agent can continuously explore. Introducing entropy randomizes the policy, as shown below. Therefore, when the algorithm makes decisions, the probability of each action can be distributed as widely as possible, thereby improving the vehicle agent's ability to learn from the environment. This allows it to adaptively adjust its strategy in the constantly changing channel conditions of the vehicle-to-everything (V2X) environment, and thus make more reasonable decisions. Based on this, the soft-state action value function is defined as follows:
[0098]
[0099] in, This represents the cumulative reward for the discount; further, the soft-state value function can be expressed as...
[0100]
[0101] Among them, actions Depends on probability distribution .
[0102] (2) Critic Network: Under the Actor-Critic network framework of the algorithm, the entire training process alternates between policy evaluation and policy improvement, thereby maximizing the long-term trade-off of the vehicle network system environment. To evaluate the merits of action policies, two Critic networks are constructed, denoted as online value networks 1 and 2, and target value networks are constructed for each. The inputs of these neural networks are local observation information collected by vehicle agents in the environment, and the outputs are the values corresponding to each action in the action space. Given that the algorithm framework generally uses neural networks to evaluate soft-value functions, the weight parameters of online value networks 1 and 2 and the corresponding target networks are defined as follows: , as well as , ;
[0103] To improve training stability and avoid divergence in reinforcement learning algorithms, an experience reuse pool is used to mitigate the correlation between samples. During training, each vehicle agent retrieves data from the reuse pool. Randomly select sequence pairs of a batch size from the data, i.e. The flexible state value function is redefined as follows:
[0104]
[0105] In the formula Represented as Action probability distribution under certain conditions Transpose of;
[0106] In the Critic network, the updates to value networks 1 and 2 are approximated by establishing a Bellman mean squared error loss function for each Q-network, expressed as:
[0107]
[0108] in, Let 1 and 2 represent value networks respectively. To achieve the goal of maximizing entropy, entropy needs to be incorporated into the reward. The functional expression for judging the value of a behavioral strategy, derived from the Bellman general derivation formula, is as follows:
[0109]
[0110] In addition, in order to obtain The optimal approximation is obtained, and the parameters are updated in the gradient direction. Minimize the loss function, which is expressed as:
[0111]
[0112] in, This represents the learning rate of the Critic network. This represents the calculation of the gradient of the loss function;
[0113] Since the target value network does not actively participate in the learning process and cannot be updated independently, a soft update method is adopted. It periodically copies the latest parameters of the online value network for a small-scale update, represented as:
[0114]
[0115] in, Indicates the degree of soft update.
[0116] (3) Actor Network: The task of the Actor Network is to seek policy improvement based on the estimates generated by the neural network. Its input is the local observation information collected by each vehicle agent in the environment, and its output is the action probability of the action dimension, expressed as... Since the update process of an Actor network can be represented as exploring a policy by maximizing the system reward, the decision-making of actions is guided by a soft-state action-value function. Therefore, the loss function of the Actor network is defined as follows:
[0117]
[0118] Similarly, the Actor network updates parameters along the gradient direction. Minimize the loss function, which is expressed as:
[0119]
[0120] in, This represents the learning rate of the Actor network. This represents the calculation of the gradient of the loss function;
[0121] (4) Self-regulating entropy coefficient: Entropy coefficient As a weight, the randomness of the action strategy is controlled by changing the value. The larger the value, the more dispersed the probability of the algorithm's output action when making decisions, leading to a more thorough exploration of the environment by the vehicle's intelligent agent, and thus, the more actions it attempts. Considering that the reward value fed back by the system is constantly changing during the training process in a connected vehicle environment, relying on a fixed prior value... This can lead to training instability, thus affecting the system's convergence performance. Therefore, this section considers an adaptive entropy coefficient method, which increases the entropy coefficient at the beginning of training when the vehicle agent explores the new environment. To allow the agent to explore as much as possible; and as the number of training rounds increases, the optimal action is basically determined, thus reducing... To optimize the long-term reward tradeoff of the system as much as possible, the loss function for the entropy coefficient is defined as:
[0122]
[0123] in, Indicates the current state Below, the threshold of entropy.
[0124] In one specific implementation, step 7 involves the aggregation center collecting the Actor network weight parameters of each vehicle agent and performing federated aggregation. The aggregated parameters are then broadcast to the vehicle users during a training round for local training. Specifically:
[0125] First, each vehicle agent trains its DRL model in a distributed manner based on its local observation information. Second, the Actor network weight parameters of the trained local DRL models are uploaded to the cloud center for policy sharing. Finally, the globally aggregated model parameters are broadcast to the local agents for training in the next iteration. This communication mode only requires uploading neural network parameters and downloading policies, greatly reducing the communication load. Furthermore, since user local information cannot be directly obtained from the neural network parameters, the system's privacy is protected. In each training round, the update formula for the Actor network weight parameters of each vehicle agent is:
[0126]
[0127] in, , These are the weighting coefficients.
[0128] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0129] Example
[0130] This embodiment provides a method for edge content caching decision-making in vehicle-to-everything (V2X) networks based on federated reinforcement learning, which is described in detail below:
[0131] 1. Establish a vehicle-to-everything (V2X) system model:
[0132] The simulation environment was set up according to the Manhattan grid model in the 3GPP TR 36.885 standard. The city map was set to 500. The distance is 500 m, the number of vehicles is 4, and the number of roadside units is 36. The maximum observation range for vehicle users is 200 m, and the maximum number of associated roadside units is 4. Vehicle users can request 20 pieces of content, with the content data size set to [20, 50] Mbit. The platform factor for the global popularity of requested content is also considered. The skewness factor is -0.88. The initial value is 0.35. At the start of training, vehicle positions are initialized. When a vehicle user reaches an intersection, they will choose the subsequent direction of travel with equal probability, meaning each direction has a probability of 0.25. The speed of each vehicle follows a Gaussian-Markov movement model, where the asymptotic mean of the initial speed is... The standard deviation is ,parameter Set as The path loss model is set using... Furthermore, setting the unit time slot duration to 0.1 s sets the cloud download latency to 1 s, thus constraining the latency. Set to 1.5 s.
[0133] 2. Establish a federated reinforcement learning algorithm framework:
[0134] The federated reinforcement algorithm combines the federated averaging algorithm and the flexible actor-critic algorithm. Within the flexible actor-critic algorithm framework, the Critic network is fitted using a fully connected neural network with two hidden layers, each containing 64 neurons, and the activation function is a linear rectified function. Similar to the Critic network, the Actor network uses the same fully connected layer neural network for fitting. A virtual cloud server is constructed as an aggregation center to perform federated aggregation of the weight parameters of the local Actor network.
[0135] 3. Algorithm training phase:
[0136] First, each vehicle agent is trained using a flexible actor-critic algorithm to implement its edge caching strategy. For the vehicle agents... Enter local status information Defined as , i.e., slot The data size of the request content from the vehicle intelligent agent, the global popularity of the file, and its relationship with roadside units. Distance between, roadside units Cache request content The flag bit, the remaining cache space size, and the remaining time slots for cache space refresh are specified. Secondly, the vehicle's intelligent agent makes decisions based on local observations, represented as... These are edge-related decision variables and edge-cached decision variables.
[0137] After the algorithm inputs the state, based on the actions predicted by the Actor network, and interacts with the vehicle-to-everything (V2X) environment, the vehicle's intelligent agent receives a global reward from the system. And transition to the next state. When sufficient sample data is available, the vehicle agents update the Actor network and Critic network respectively using gradient descent. At the end of each round, each vehicle agent uploads the weight parameters of its local Actor network to the cloud center for federated averaging. The aggregated global parameters are then broadcast to the vehicle agents for local training in the next round. A training set of 3000 data points and a test set of 100 data points are used. During training, the learning rate of the Critic network is set... for Actor network learning rate for Discount factor The value is 0.9, the soft update degree is 0.01, the experience reuse pool size is 5000, and the number of samples for training is set to 100.
[0138] The present invention compares the proposed solution with the following benchmark solutions:
[0139] (1) Edge caching scheme based on Independent Soft Actor-Critic (ISAC) algorithm: Each vehicle user is an agent and uses the SAC algorithm to train its own strategy. That is, each agent trains an independent Critic and Actor network and makes decisions in a distributed manner based on its own local observation information.
[0140] (2) Edge caching scheme based on Independent Deep Q Network (IDQN) algorithm: Each vehicle user acts as an agent and uses the DQN algorithm to train its own policy. That is, each agent trains an independent Q network and makes decisions in a distributed manner based on its own local observation information.
[0141] (3) Edge caching scheme based on Federated DQN: Each vehicle user acts as an agent and uses the DQN algorithm with shared local Q network weights to train its own strategy. The Q network weights are uploaded to the cloud center for federated averaging. Then each vehicle user downloads the aggregated global parameters and makes decisions in a distributed manner based on its own local observation information.
[0142] like Figure 2 As shown, comparing the proposed scheme with the baseline scheme, it is evident that the ISAC-based scheme converges the fastest, followed by the proposed scheme, while the FedDQN-based and IDQN-based schemes converge the slowest. The underlying reason for this phenomenon is that the proposed scheme and the ISAC-based scheme attempt to maximize the entropy objective, enhancing the algorithm's exploration capability and thus promoting model convergence, resulting in better convergence performance. In contrast, the IDQN-based and FedDQN-based schemes utilize stochastic strategies for exploration, leading to difficulties in model convergence and coordination issues in utilizing sample exploration in the high-dimensional action space of the vehicle-to-everything (V2X) environment. Furthermore, compared to the ISAC-based scheme, the proposed scheme achieves a better system average tradeoff while protecting privacy. This is due to two factors: first, the proposed scheme shares the weight parameters of the Actor network in the local DRL model, allowing agents to share their experience interacting with the environment, thereby promoting better policy decisions; second, the cloud center only collects weight parameters and cannot obtain local observation information from training, which reduces communication overhead and protects the privacy of vehicle users.
[0143] like Figure 3As shown, the average transmission latency per vehicle user decreases with increasing training rounds, indicating the effectiveness of each scheme. Compared to the baseline scheme, the proposed scheme significantly outperforms the IDQN-based and FedDQN-based schemes in optimizing average transmission latency, and slightly outperforms the ISAC-based scheme. This phenomenon arises from two aspects: firstly, all four schemes effectively optimize the average transmission latency metric, but after normalization of the utility function, the performance differences become difficult to distinguish; secondly, the complex and dynamic vehicular network environment leads to oscillating convergence results, making it difficult to demonstrate a clear performance advantage. Figure 4 As shown, the edge caching overhead of each scheme decreases with the increase of the number of training rounds. The proposed scheme has the lowest edge caching overhead, followed by the ISAC-based scheme, while the schemes based on IDQN and FedDQN have the worst performance. (Combined) Figure 3 and Figure 4 It can be seen that the proposed solution can significantly reduce edge caching overhead while also effectively optimizing the transmission latency for each vehicle user.
[0144] Figure 5 This study demonstrates the impact of different maximum associated number parameters on the average performance trade-off per vehicle user in building an edge node cluster for vehicle-to-everything (V2X) networks. Specifically, as the number of associated roadside units increases, the convergence rate of the proposed scheme also increases. This phenomenon arises because user-centric edge node clusters can seamlessly adapt to dynamic fluctuations in network topology, and as the number of roadside units in the cluster increases, their joint transmission technology can significantly improve data transmission rates. However, when the number of associated roadside units reaches a certain point, the gain of the edge node cluster on transmission rate reaches saturation, i.e., when... Compared to the average convergence result per vehicle user The difference is not significant.
[0145] In summary, this invention addresses the unknown, highly dynamic topology and channel state characteristics of vehicular networks (V2X) by proposing a federated flexible actor-critic algorithm-based edge caching scheme within a user-centric framework, leveraging the privacy advantages of federated learning. The proposed scheme achieves collaborative training decisions in a multi-agent environment without disclosing local training data of vehicle users. This ensures high privacy while jointly optimizing transmission latency and caching overhead, outperforming benchmark schemes in terms of average tradeoffs per vehicle user and convergence performance.
Claims
1. A method for edge content caching decision-making in vehicle-to-everything (V2X) networks based on federated reinforcement learning, characterized in that, The specific steps include: Step 1: Input the vehicle network environment, each vehicle intelligent agent initializes the parameters of its own actor network and critic network, and constructs an optimization problem with the goal of minimizing the long-term trade-off between content delivery latency and edge caching overhead. Step 2: In the current time slot, each vehicle agent interacts with the roadside units within the observation range to obtain observation information such as the distance between itself and the roadside units, the buffer status of the roadside units, and the remaining buffer capacity of the roadside units. Step 3: Based on local observation information, each vehicle agent can independently decide on the roadside units associated with the edge node cluster, and decide within the cluster whether to cache the request content of the current time slot. Step 4: After executing the action decision, each vehicle intelligent agent obtains the reward for the total system content delivery latency and edge caching overhead from the feedback of the vehicle network environment, and at the same time, all sample data is cached to the experience reuse pool. Step 5: Determine if the sample size is sufficient. If yes, proceed to Step 6; otherwise, proceed to Step 7. Step 6: When the number of samples is sufficient, each vehicle agent updates its Actor network and Critic network parameters according to the flexible actor-critic algorithm; Step 7: The aggregation center collects the Actor network weight parameters of each vehicle agent and performs federated aggregation. The aggregated parameters are broadcast to the vehicle users in one training round for local training. Step 8: Determine if the current training round has ended. If not, return to step 2 to start the next training round. If yes, proceed to step 9. Step 9: Determine if convergence has occurred. If not, reset the vehicle network environment and return to Step 1; if yes, training ends, completing the vehicle network edge content caching decision.
2. The method for edge content caching decision-making in vehicle-to-everything (V2X) networks based on federated reinforcement learning according to claim 1, characterized in that, The input of the vehicle networking environment mentioned in step 1 specifically includes: (1) Time-slot model: The continuous training time is discretized into multiple time slots, represented as follows: The duration of each time slot is The channel state information and system parameters remain unchanged over the duration of a single time slot, but may vary randomly between different time slots; (2) Network Model: The vehicle-to-everything (V2X) network model is established as a Manhattan grid model, in which roadside units that can provide communication services are evenly distributed on both sides of the road; the roadside units, as edge nodes, have dedicated communication resources and limited local storage resources, and are connected to the edge server through a high-speed wired link; the edge server is controlled by a software-defined centralized controller, which can perform edge association and cache resource allocation, etc.; let the set of roadside units be . The vehicle group is All vehicle users can travel in four directions along the road, with multiple lanes in each direction to ensure vehicle passage. (3) Vehicle movement model: The speed changes of the vehicle all follow the following Gaussian-Markov stochastic process. Specifically, when the vehicle user With initial velocity During travel, it is in a time slot speed at This can be represented as a time slot. speed at The sum of the asymptotic rate and a random variable: in, and Vehicle users The corresponding asymptotic mean and standard deviation of the velocity; parameters It represents the memory depth of the previous time slot speed and determines the vehicle user's... Temporal correlation of motion; This represents a standard normal distribution with zero mean and unit variance, indicating no correlation. It is worth noting that... As it approaches 1, vehicle users The current time slot velocity becomes more dependent on the velocity of the previous time slot; (4) Content Request Model: Let This represents the collection of all vehicle request content. This indicates the total size of all requested content. This represents the characteristic value of all request content, used to distinguish different request contents; assuming a vehicle user... In the time slot Only one request content can be generated. , represented as in, Indicates vehicle user In the time slot Request content ,on the contrary Considering that vehicle users may prefer files similar to those requested in previous time slots, in addition to requesting content based on global popularity, we assume that vehicle users request content based on... The probability is based on global popularity. The probability is based on local personal preference, where global popularity and local personal preference are defined as follows: ① Global popularity: Let This represents the global popularity of user requests for files for each vehicle, which follows a Mandelbrot-Zipf distribution, i.e., satisfies... in, Content Sort in descending order of global popularity; and These represent the platform factor and the skewness factor, respectively. ② Local Personal Preferences: In this case, each vehicle user requests files based on the similarity of their previously requested content; for example, vehicle users in different time slots... Request content Then in the time slot Request and content The content with the highest similarity; this section uses cosine similarity to represent the content. and Similarity: (5) User-Centric Edge Caching Model: To improve the transmission rate of requested content by vehicle users, a user-centric network architecture is adopted to design the vehicle-to-everything (V2X) edge content caching framework. Specifically, each vehicle user observes the information of roadside units within its perception range, thereby selecting one or more nearby roadside units to construct user-centric edge node clusters; enabling vehicle users to... In the time slot The maximum observable number of roadside units is The maximum number of roadside units that can be associated is expressed as ;make Indicates time slot Time service vehicle users The edge node cluster, then This indicates the number of roadside units in the cluster at this time; it allows vehicle users to... In the time slot The edge association decision is represented as: in, Indicates roadside unit In the time slot Belongs to vehicle users edge node cluster ,on the contrary Furthermore, a user-centric edge node cluster can be represented as... Assuming both roadside units and the cloud center have cache capacity, with the cloud center's cache capacity sufficient to handle cache requests from all vehicle users, and each roadside unit having a limited cache capacity... In a user-centric edge content caching framework, vehicle users may request that some of their requested files be cached in their associated edge node cluster to provide ultra-high-speed cached content services. Therefore, vehicle users need to further decide whether to cache the request content for the current time slot in each roadside unit within the edge node cluster; thus, the time slot... Vehicle users The edge caching decision variables are represented as in, Indicates vehicle user time slot The requested content is cached in the roadside unit. ,on the contrary ; (6) Wireless transmission model: It is assumed that mutual interference between communication links has been eliminated by allocating orthogonal resource blocks, and that both roadside units and vehicle users are equipped with single antennas; considering that channel power gain includes both fast fading and slow fading, the main cause of fast fading is multipath effect, i.e., Rayleigh fading; the slow fading consists of path loss and shadowing fading; let Indicates in time slot Vehicle users With roadside units The channel gain between them is expressed as: in, This corresponds to the fast fading portion, which follows a complex Gaussian distribution with zero mean and unit variance, i.e., satisfies... Vehicle users With roadside units The slow fading part is represented as in, This is a constant in path loss; It is a shadow fading with standard deviation The log-normal components; Indicates time slot vehicle users With roadside units Distance between; The attenuation coefficient representing the path loss component; Indicates in time slot vehicle users With roadside units Path loss of the channel between them; Time slot Edge node cluster Central side unit The transmission power is Then vehicle user The achievable downlink transmission rate is expressed as in, For channel bandwidth, The power of additive white Gaussian noise; (7) Latency Model: Let the tolerable latency of each vehicle user's request content be... In addition, to fully utilize edge caching resources, the edge server periodically... Its storage space is automatically refreshed, therefore the edge server experiences brief service downtime, occurring every certain period. Replace the cached content; therefore, if the vehicle user In the time slot When cached content needs to be downloaded from an edge server, its delivery must be within tolerable latency. And it will be completed within the next cache replacement and refresh cycle, that is In the formula, This indicates the number of time slots remaining in the corresponding cache refresh cycle. ; Considering that the content delivery latency for vehicle users depends on whether the requested content has been pre-cached on the edge server, time slots are defined. Roadside Unit The cache status is in, Content In the time slot Cached to roadside unit In the middle, or vice versa Therefore, the delivery of requested content to vehicle users includes both edge caching and cloud downloading: ① Edge caching: When an edge node cluster... The requested content has been cached at any edge node. At that time, vehicle users The requested content can be obtained directly from the edge, and its latency is expressed as ② Cloud download: When the edge node cluster None of the edge nodes in the middle cached the requested content. At times, edge servers need to download from the cloud center, resulting in additional fixed latency. ; Based on the above model, vehicle users Get the request content The total delay can be expressed as in, This indicates that the value is set to 1 if the constraint is met, and 0 otherwise.
3. The method for edge content caching decision-making in vehicle-to-everything (V2X) networks based on federated reinforcement learning according to claim 1, characterized in that, The optimization problem described in step 1, which aims to minimize the long-term trade-off between content delivery latency and edge caching overhead, is as follows: First, the normalized content delivery latency utility and edge caching overhead utility are designed and expressed as follows: Therefore, the optimization problem is formulated as follows: in, and These represent the weights of content delivery latency utility and caching overhead utility, respectively; This indicates that the size of the edge node cluster for each vehicle user in any time slot does not exceed [a certain value]. ;Mode This indicates that each roadside unit can serve at most a single vehicle user in any time slot; This indicates that the size of the cached content of each roadside unit cannot exceed its local cache capacity; This means that the delivery delay of the content requested by each vehicle user in any time slot should not exceed the maximum allowable delay.
4. The edge content caching decision method for vehicle-to-everything (V2X) networks based on federated reinforcement learning according to claim 1, characterized in that, Step 2 involves each vehicle agent interacting with roadside units within its observation range to obtain observation information such as the distance between itself and the roadside units, the buffer status of the roadside units, and the remaining buffer capacity of the roadside units. Specifically: In the time slot Each vehicle user acts as an intelligent agent, obtaining their own observed state through interaction with the environment. In the time slot The observation state is represented as in, Indicates time slot vehicle users With roadside units The distance between them; Indicates time slot Roadside Unit Has the requested content been cached? The flag bit; Indicates time slot Roadside Unit Remaining cache space size; Indicates time slot Roadside Unit Number of time slots remaining until the cache space is refreshed.
5. The edge content caching decision method for vehicle-to-everything (V2X) networks based on federated reinforcement learning according to claim 1, characterized in that, Step 3 describes how, based on local observation information, each vehicle agent can independently decide which roadside units are associated with in the edge node cluster, and within the cluster, decide whether to cache the request content of the current time slot. Specifically: Determine the actions taken by each vehicle agent in its interaction with the environment, including edge-related decision variables and edge-cached decision variables; then the vehicle... In the time slot The action is represented as: 。 6. The edge content caching decision method for vehicle-to-everything (V2X) networks based on federated reinforcement learning according to claim 1, characterized in that, After the action decision is executed as described in step 4, the reward for each vehicle agent to obtain the total system content delivery latency and edge caching overhead from the vehicle network environment feedback is as follows: After all vehicle agents have completed their actions, the environment will provide a global reward, defined as the average trade-off for each user, expressed as: in, This represents the normalized content delivery delay effect. This represents the utility of normalized edge cache overhead; This indicates that the constraint is not satisfied. The penalty items at that time.
7. The method for edge content caching decision-making in vehicle-to-everything (V2X) networks based on federated reinforcement learning according to claim 1, characterized in that, Step 6 describes how each vehicle agent updates its Actor and Critic network parameters according to the flexible actor-critic algorithm, specifically as follows: (1) Flexible value function: The optimization objective of the flexible actor-critic algorithm, in addition to maximizing the cumulative reward value returned by training, also needs to maximize entropy, that is... in, Representation strategy The trajectory distribution; This represents the entropy coefficient; it is used to ensure that the agent can continuously explore. Introducing entropy randomizes the policy, as shown below. Therefore, when the algorithm makes decisions, the probability of each action can be distributed as widely as possible, thereby improving the vehicle agent's ability to learn from the environment. This allows it to adaptively adjust its strategy in the constantly changing channel conditions of the vehicle-to-everything (V2X) environment, and thus make more reasonable decisions. Based on this, the soft-state action value function is defined as follows: in, This represents the cumulative reward for the discount; further, the soft-state value function can be expressed as... Among them, actions Depends on probability distribution ; (2) Critic Network: Under the Actor-Critic network framework of the algorithm, the entire training process alternates between policy evaluation and policy improvement, thereby maximizing the long-term trade-off of the vehicle network system environment. To evaluate the merits of action policies, two Critic networks are constructed, denoted as online value networks 1 and 2, and target value networks are constructed for each. The inputs of these neural networks are local observation information collected by the vehicle agents in the environment, and the outputs are the values corresponding to each action in the action space. Given that the algorithm framework generally uses neural networks to evaluate soft-value functions, the weight parameters of online value networks 1 and 2 and the corresponding target value networks are defined as follows: , as well as , ; To improve training stability and avoid divergence in reinforcement learning algorithms, an experience reuse pool is used to mitigate the correlation between samples. During training, each vehicle agent retrieves data from the reuse pool. Randomly select sequence pairs of a batch size from the data, i.e. The flexible state value function is redefined as follows: In the formula Represented as Action probability distribution under certain conditions Transpose of; In the Critic network, the updates to online value networks 1 and 2 are approximated by establishing a Bellman mean squared error loss function for each Q-network, expressed as: in, Let represent online value networks 1 and 2, respectively. To achieve the goal of maximizing entropy, entropy needs to be incorporated into the reward. The functional expression for judging the value of a behavioral strategy, derived from the Bellman general derivation formula, is as follows: In addition, in order to obtain The optimal approximation is obtained, and the parameters are updated in the gradient direction. Minimize the loss function, which is expressed as: in, This represents the learning rate of the Critic network. This represents the calculation of the gradient of the loss function; The target value network does not actively participate in the learning process and cannot be updated independently; therefore, a soft update method is adopted. It periodically copies the latest parameters of the online value network for small-scale updates, represented as: in, Indicates the degree of soft update; (3) Actor Network: The task of the Actor Network is to seek policy improvement based on the estimates generated by the neural network. Its input is the local observation information collected by each vehicle agent in the environment, and its output is the action probability of the action dimension, expressed as... Since the update process of an Actor network can be represented as exploring a policy by maximizing the system reward, the decision-making of actions is guided by a soft-state action-value function. Therefore, the loss function of the Actor network is defined as follows: Similarly, the Actor network updates parameters along the gradient direction. Minimize the loss function, which is expressed as: in, This represents the learning rate of the Actor network. This represents the calculation of the gradient of the loss function; (4) Self-regulating entropy coefficient: Entropy coefficient As a weight, the randomness of the action strategy is controlled by changing the value; The larger the value, the more dispersed the probability of the algorithm's output action when making decisions, leading to a more thorough exploration of the environment by the vehicle's intelligent agent, and thus attempting more actions; considering that the reward value fed back by the system is constantly changing during the training process in the vehicle-to-everything (V2X) environment, relying on prior fixed values... This can lead to training instability, thus affecting the system's convergence performance. Therefore, this section considers an adaptive entropy coefficient method, which increases the coefficient at the beginning of training when the vehicle agent explores the new environment. To allow the agent to explore as much as possible; and as the number of training rounds increases, the optimal action is basically determined, thus reducing... To optimize the long-term reward tradeoff of the system as much as possible, the loss function for the entropy coefficient is defined as: in, Indicates the current state The threshold of downentropy.
8. The method for edge content caching decision-making in vehicle-to-everything (V2X) networks based on federated reinforcement learning according to claim 1, characterized in that, Step 7 describes the aggregation center collecting the Actor network weight parameters of each vehicle agent and performing federated aggregation. The aggregated parameters are broadcast to the vehicle users in one training round for local training. Specifically: First, each vehicle agent trains its DRL model in a distributed manner based on its local observation information; second, the Actor network weight parameters of the trained local DRL model are uploaded to the cloud center for policy sharing; finally, the global model parameters of the federated aggregation are broadcast to the local agents for training in the next iteration. This communication mode only requires uploading neural network parameters and downloading strategies, greatly reducing the communication load. Furthermore, since user local information cannot be directly obtained from the neural network parameters, the system's privacy is protected. In each training round, the update formula for the network weight parameters of each vehicle agent (Actor) is as follows: in, , These are the weighting coefficients.