Unmanned aerial vehicle assisted cooperative caching method based on double-agent reinforcement learning

Abstract

The application discloses a kind of unmanned vehicle auxiliary based on double intelligent agent reinforcement learning's collaborative caching method, belong to mobile communication technical field, for solving unmanned vehicle auxiliary car networking in cache resource allocation and unmanned vehicle trajectory difficult to collaborative optimization problem, to reduce content distribution delay, reduce system energy consumption.This method includes: constructing the collaborative caching model consisting of macro base station, unmanned vehicle, roadside unit and vehicle;Realize dynamic clustering based on vehicle position and interest;Predict content popularity by hierarchical asynchronous federated learning;Establish the optimization problem with minimizing average delay and energy consumption as target, and decouple into two sub-problems of cache decision and unmanned vehicle trajectory optimization;Adopt competitive double deep Q network algorithm and deep deterministic policy gradient algorithm to constitute double intelligent agent reinforcement learning framework, collaboratively solve the above sub-problems, output optimal caching strategy and unmanned vehicle flight trajectory.The application effectively improves the overall performance of edge caching system in dynamic vehicular networking environment.

Description

Technical Field

[0001] This invention belongs to the field of mobile communication technology, specifically relating to a collaborative caching method based on dual-agent reinforcement learning with the assistance of unmanned aerial vehicles (UAVs). Background Technology

[0002] Against the backdrop of the deep integration of intelligent transportation and communication technologies, the Internet of Vehicles (IoV) has evolved into a core infrastructure supporting the intelligent upgrading of transportation. With the widespread adoption of real-time navigation, in-vehicle entertainment, and vehicle-road cooperative services, the surge in data traffic has placed stringent demands on the low latency and high reliability of edge services. Traditional roadside units, due to their fixed deployment, struggle to adapt to the topology changes and service blind spots caused by high-speed vehicle movement. Drones, with their advantages of flexible deployment and rapid response, have become an important supplement to IoV edge services, providing innovative solutions for collaborative caching and resource scheduling.

[0003] Collaborative caching effectively reduces backhaul load and service latency by forwarding frequently requested content to edge nodes. However, in connected vehicle scenarios, this mechanism faces significant challenges: limited node caching resources, dynamic network topology changes, and regional heterogeneity of requested content make content-node matching difficult. Traditional static caching strategies are prone to redundancy of popular content and lack of personalized content, resulting in insufficient cache hit rate and increased backhaul pressure. On the other hand, as a dynamic service carrier, the trajectory optimization of drones directly affects whether cache resources can accurately reach users. Existing research mostly focuses on a single coverage target without coordinating with cache distribution and real-time request hotspots, making it difficult to balance coverage accuracy and energy consumption. More importantly, the disconnect between collaborative caching and trajectory optimization not only limits the release of cache resource utility but also weakens the maneuverability advantage of drones, becoming a core bottleneck restricting system performance.

[0004] In existing technologies, single caching strategies or static drone deployments cannot effectively address the aforementioned complex optimization problems. Therefore, focusing on typical application scenarios of drone-assisted vehicle-to-everything (V2X) requires systematic research on the collaborative adaptation mechanism of cooperative caching and trajectory optimization. This aims to overcome the aforementioned technical bottlenecks and provide theoretical support and methodological innovation for improving edge service performance. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a collaborative caching method based on dual-agent reinforcement learning with UAV assistance. Combining vehicle dynamic clustering and hierarchical asynchronous federated prediction models, a competitive dual-deep Q-network algorithm is employed to obtain the optimal caching decision strategy. Furthermore, a depth-determined policy gradient algorithm is used to optimize the UAV trajectory. Finally, the optimal caching placement strategy is obtained through a dual-agent collaborative optimization framework, effectively reducing service latency and network energy consumption, and demonstrating good robustness and practicality.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a collaborative caching method based on dual-agent reinforcement learning under UAV assistance, specifically including the following steps: Step 1, in a UAV-assisted vehicle network, constructing an edge collaborative caching model composed of macro base stations, UAVs, roadside units, and vehicle terminal users; Step 2, utilizing the location and preference information of vehicle users to achieve dynamic vehicle clustering, dividing vehicles into multiple interest clusters in dense areas and independent vehicles in sparse areas; Step 3, using a hierarchical asynchronous federated learning prediction model to calculate the popularity of requested content; the hierarchical asynchronous federated learning prediction model includes: constructing a cluster-local model based on the request history of each interest cluster in the dense area, and constructing a cluster-local model based on the request history of independent vehicles in the sparse area. Step 4: Construct a local vehicle model based on the vehicle's request history, and an asynchronous aggregation cluster local model and a local vehicle model to generate a global popularity model; Step 5: Construct a cache placement optimization problem with the joint goal of minimizing the system's average latency and average energy consumption, and decouple this optimization problem into a cache decision subproblem and a UAV trajectory optimization subproblem; Step 6: Use a competitive dual-deep Q-network algorithm to solve the cache decision subproblem to obtain the optimal cache placement strategy; Use a depth-determined policy gradient algorithm to solve the UAV trajectory optimization subproblem to obtain the optimal UAV flight trajectory; Through a dual-agent collaborative optimization framework, the cache decision agent and the trajectory optimization agent interact and provide feedback, ultimately obtaining the system's optimal collaborative caching strategy.

[0007] Preferably, in step 1, the vehicle-to-everything (V2X) edge collaborative caching model includes: Macro base stations serve as content origination centers; roadside unit clusters Fixed deployment on both sides of the road, equipped with buffering and communication functions; drone ensemble As an airborne mobile buffer and communication node; vehicle collection It moves on city roads and has local caching capabilities; the system operates within a discrete time slot set. The network topology and channel state remain unchanged within each time slot; edge buffer nodes are connected to vehicle users and vehicle users to each other via wireless communication links.

[0008] Preferably, in step 2, the process of implementing vehicle dynamic clustering includes: Step 2.1, Region Division and Candidate Vehicle Screening: Based on the Euclidean distance between vehicles and the connectivity threshold, select vehicles with a neighborhood number of at least [number missing]. And the total number of vehicles in the connected component is not less than The vehicles constitute the candidate vehicle set. The vehicles in this set have stable communication capabilities, and based on the vehicle location distribution, the area is initially divided into densely populated and sparsely populated areas. Set up vehicles The neighborhood set is ,in This represents the Euclidean distance between the two vehicles. This represents the connectivity threshold between vehicles; if a vehicle is at least connected to... All other vehicles remain connected and the total number of vehicles in the connected component is not less than [number missing]. Then the vehicle Included in the candidate vehicle set ,in This represents the minimum number of connected vehicles. This represents the minimum number of clusterable vehicles; based on vehicle location distribution, dense and sparse regions are divided. Step 2.2: Extraction and quantification of interest features; Based on the vehicle's historical content request records, extract content type vectors and calculate the time-weighted interest vectors of the vehicle for each type of content. ; Let the content set be First, the content type information is extracted; second, all occurrences of content type are counted, resulting in a total of... Different types; each content type is represented as a string of length. One-hot encoded vector The first of each content A value of 1 for a type element indicates that the content belongs to the [number]th [element]. If it is a specific type, then it is 0; otherwise, it is 0. Then calculate the weighted score for each vehicle user for each type; assuming the vehicle For content The rating is Then the vehicle In the time slot Previous request history is ,vehicle Interest vector The calculation formula is as follows: (1); in, It is a length of The time-weighted interest vector represents the degree of interest that vehicles have in each type of content. It is the time-degradation factor; yes The time-deterioration factor is superimposed within the time slot; A content type vector; Step 2.3: Dimensionality reduction of interest features; Principal component analysis is used to reduce the time-weighted interest vector. Dimensional reduction is obtained ; All vehicle user interest vectors are combined into a matrix ,in It's the number of users. It is the real number field; therefore, calculation Find the covariance matrix of the given matrix and determine its eigenvalues ​​and eigenvectors; select the first... The eigenvectors corresponding to the largest eigenvalues ​​are used as principal components to form the projection matrix. Finally, the reduced-dimensional interest vector Represented as; (2); Each user is randomly represented as a... 3D interest vector It can comprehensively reflect users' preferences for different types of movies; Step 2.4: Clustering based on interest preferences; use the K-means algorithm to cluster low-dimensional interest vectors, dividing vehicles in dense areas into clusters. Interest clusters And select the vehicle with the largest cache capacity as the cluster head for each cluster; Step 2.5: Dynamic update of vehicle clusters; periodically update the position of the vehicle based on its movement, update its interest vector based on the new request record, and re-execute steps S2.1 to S2.4 to achieve dynamic update of vehicle clusters. When updating the position, the vehicle moves at a constant speed on the road, and the formula for the change in vehicle position is: (3); in, Indicates the vehicle's speed; Indicates the direction of vehicle movement; Indicates unit conversion; This is the time when the vehicle's location was updated; It is the vehicle in the time slot Location; This is the vehicle's updated location; It is the unit component of the vehicle's direction of movement on the x-axis; It is the unit component of the vehicle's direction of movement on the y-axis; When updating interests, vehicle interests evolve with the request history, and the interest vector update formula is: (4); in, Indicates to retain the most recent Each time slot contains a request record, and each record also contains content. Request time slot and rating , represented as ; This represents the updated interest vector; Each interval Each training round re-executes clustering based on the updated vehicle positions and interests.

[0009] Preferably, in step 3, the process of hierarchical asynchronous federated learning predicting content popularity includes: Step 3.1: Building the local model; Clusters of interest in dense areas Based on the request history of all vehicles within the cluster, the content is calculated using a time-weighted request frequency. Popularity score Build a cluster-local model; For independent vehicles in sparse areas Based on its own request history, the same time-weighted method is used to calculate the content. Popularity score Build a local vehicle model; Specifically, the steps are as follows; Step 3.1.1, Local Model for Dense Areas: In dense areas, vehicles within a cluster form stable cooperative units through vehicle-to-vehicle (V2V) communication, and their content requests are similar. The cluster-local model is built on a cluster-by-cluster basis, based on the request history of all vehicles within the cluster. Record of every content request for the vehicle for; (5); in, For content; This represents the cumulative number of requests for this content. The time slot in which this content was most recently requested; When cluster In-vehicles in time slots Initiate content When making a request, request history The update rule formula is: (6); in, This is the set of content that has been requested. For each subsequent request, the cumulative count is incremented by 1, and the most recent time slot request history is updated. Let the time-weighted request frequency be cluster. Local model for content Popularity score The calculation formula is: (7); in, This is the sum of time decay factors between the most recent request time slots; For the current time slot; For content The most recently requested time slot; For content The number of content requests; For content The number of content requests is used to reduce the weight of historical requests and highlight the impact of recent requests; thus, the cluster-local model is saved as... ; Step 3.1.2, Local Model for Sparse Regions: Vehicles in sparse regions are widely distributed and their request patterns are highly independent. Therefore, a local model is directly built based on the request history of individual vehicles to capture individual preferences. The local model for vehicles in sparse regions is based on content... The formula for calculating the popularity score is consistent with the cluster-local model, except that the statistical object is changed from all vehicles within the cluster to vehicles in a single sparse area, and thus the vehicle-local model is saved as... ; Step 3.2, Layered asynchronous aggregation; When the cumulative number of local model updates reaches a preset threshold When triggered, global aggregation is performed; by content type. The prediction results of all cluster-local models and sparse region vehicle-local models are aggregated separately, and the content is obtained through weighted calculation. Global popularity score This forms a global popularity model; global popularity model The calculation formula is: (8); in, For content type For type Cluster local model for content Popularity score; For type Sparse vehicle local model for content The average popularity score; The weighting coefficients are used to balance the contributions.

[0010] Preferably, the specific process of step 4 is as follows: Step 4.1: Construct the system's cache model and UAV trajectory model; Step 4.2: Construct an average latency model for the system's content request service process; Step 4.3: Construct an average energy consumption model during the system content request service process; Step 4.4: Define the optimization problem, with the ultimate optimization objective being to jointly minimize the system's average latency and average energy consumption. Step 4.5: Decouple the optimization problem into two sub-problems: cache decision and UAV trajectory optimization.

[0011] Preferably, the specific process of step 4.1 is as follows; Step 4.1.1: Construct the system's caching model; the caching scenario mainly targets vehicle content requests, setting... For a collection of cache nodes, Indicates the first A cluster head, Indicates the first One roadside unit, Indicates the first One drone; define that within the same time slot, a vehicle cannot simultaneously request content from multiple nodes; assume the cache capacity of the cache node is... ,content The content size is The cache decision variable is ,when When it is 1, it represents the content. In the time slot Cached on cache nodes If the cached value is not specified, it must be cached; otherwise, it must not be cached. At the same time, cache capacity constraints must be met. (9); Step 4.1.2: Construct the UAV trajectory model for the system; time slot The location of the drone at that time ,in The coordinates are the x-coordinate, y-coordinate, and altitude of the drone's position, and the drone's flight speed. Calculated as; (10); in, This is the time interval for movement; For time slots The location of the drone at that time; The motion constraint formula for a drone is expressed as: (11); The speed of the drone is limited to the maximum and minimum value And its location is within the permitted flight zone. Inside; in the two-dimensional flight area The internal dynamic adjustment of the position aims to cover the cluster heads of clusters in densely populated areas that require coverage, thereby optimizing content distribution efficiency.

[0012] Preferably, the specific process of step 4.2 is as follows; Step 4.2.1, Components of System Delay; System delay includes transmission delay, processing delay, queuing delay, and source latency; Transmission delay is determined by data size and link rate; processing delay is inversely proportional to node processing rate and modeled according to exponential distribution; queuing delay is introduced under multi-user resource contention, and the average queuing delay is approximated by the M / M / 1 queuing model; when neither the drone nor the roadside unit hits the cache, the content is transmitted back to the requesting vehicle by the macro base station, resulting in back-to-origin delay. Step 4.2.2, delay calculation in sparse areas: Vehicles prioritize sending requests to the roadside unit with the closest geographical distance. The path follows a two-layer routing logic: if the roadside unit's cache is hit, transmission is performed directly; if the roadside unit's cache is not hit, transmission is performed back to the source macro base station. Queuing delays for roadside units in sparse areas are negligible; delays when hitting roadside units are negligible. Represented as: (12); in, For the transmission delay of the roadside unit; This is for the processing delay of the roadside unit; Miss delay Represented as: (13); in, This refers to the transmission delay from the macro base station to the roadside unit; For origin retrieval delay; Single request latency Represented as: (14); in, This is an indicator of whether the roadside unit has been hit; Step 4.2.3, delay calculation in dense areas; the path follows a three-level routing strategy: priority to cluster head hit, second hit by UAV, and base station backhaul when neither hit, making full use of the complementarity between the mobility of UAV and the close-range coverage of ground cluster head; set up Assuming the signaling interaction delay between the UAV and the cluster head; For the transmission delay from the macro base station to the drone; Hit cluster head delay Represented as: (15); in, , , These are the cluster head's transmission delay, processing delay, and queuing delay, respectively. Hitting the drone delay Represented as: (16); in, , , These are the transmission delay, processing delay, and queuing delay of the drone, respectively. All misses delay Represented as: (17); Single request latency Represented as: (18); in, , These are indicators that represent whether the cluster head and the drone have been hit; Step 4.2.4: Calculation of system average delay; Any vehicle in the global map In the time slot Request content latency Represented as: (19); in, For vehicles In the time slot Related nodes Indicator; , , vehicles In the time slot Related nodes Request content Transmission delay, processing delay, and queuing delay; Global average latency Represented as: (20); in, This represents the total time slots.

[0013] Preferably, the specific process of step 4.3 is as follows: Step 4.3.1, Calculation of system energy consumption; The system energy consumption consists of the energy consumption of the UAV, the energy consumption of the roadside unit, the energy consumption of the cluster head vehicle, and the backhaul energy consumption of the macro base station. The energy consumption of drones includes energy consumption during drone movement, energy consumption related to communication, hovering energy consumption when drones hover to provide content services to vehicles, and processing energy consumption. Among them, the movement energy consumption is modeled based on the power model of rotary-wing drones, the hovering energy consumption is the basic power consumption to maintain the stability of the aircraft, and the communication and processing energy consumption is related to the service duration and the number of requests. The energy consumption of roadside units and cluster head vehicles mainly comes from the communication and processing energy consumption of service vehicles; the energy consumption of macro base stations is only generated when the cache is not hit, including the communication and processing energy consumption of backhaul content. Step 4.3.2: Energy consumption calculation for sparse and dense regions; similar to the latency model, the energy consumption of a single request under different paths in the sparse region. Represented as: (twenty one); in, The communication power consumption of the roadside unit; The communication energy consumption from the macro base station to the roadside unit; This represents the total energy consumption of the roadside unit. For macro base station back-to-source power consumption; In densely populated areas, the energy consumption of a single request under different paths for; (twenty two); in, This indicates the total energy consumption of the leading vehicle. Communication power consumption for cluster-head vehicles; Indicates the mobile energy consumption of the drone; Indicates the hovering energy consumption of the drone; This indicates the communication and processing energy consumption of the drone; This indicates the energy consumption for transmission from the cluster head to the drone; This indicates the energy consumption for transmission from the drone to the macro base station; Step 4.3.3: Calculation of system average energy consumption; global calculation for any vehicle. In the time slot Request content energy consumption for; (twenty three); in, , For vehicles In the time slot Related nodes Request content Communication and processing energy consumption; Global average energy consumption for; (twenty four); in, For drones in time slots The sum of the energy consumption during movement and the energy consumption during hovering.

[0014] Preferably, in step 4.4, weights are introduced with the goal of minimizing the weighted sum of latency and energy consumption. A linear normalization method is used to map all targets to the [0,1] interval, normalizing the average delay and average energy consumption. for: (25); (26); in, , These represent the estimated maximum latency and maximum energy consumption, respectively. The optimization problem is represented as: (27); Satisfy constraints: (28); in, , The joint decision space represents the packet caching decision and the UAV trajectory variable. C1 indicates that the total amount of content stored in the edge node cache must not exceed its cache capacity; C2 indicates that the cached variable is a binary variable; C3 and C5 indicate that the vehicle speed must be within the range allowed by physical performance, and the speed calculation method is specified; C4 indicates that the UAV must move within the preset feasible flight range. In step 4.5, the original optimization problem is decoupled into two sub-problems based on discrete and continuous variables: the buffer decision sub-problem and the UAV trajectory optimization sub-problem. The caching decision subproblem is represented as: (29); Satisfy constraints C1 and C2; The drone trajectory optimization subproblem is represented as: (30); The constraints C3, C4, and C5 are satisfied.

[0015] Preferably, step 5 specifically involves the following process: Step 5.1: Solve the caching decision problem using a competitive dual-depth Q-network algorithm; the specific process includes: The caching decision problem is modeled as a Markov decision process, and a distributed caching decision optimization algorithm based on D3QN is proposed. This algorithm provides prior knowledge to D3QN through global-local popularity collaborative prediction driven by federated learning. Then, it utilizes the deep representation capability of D3QN and the bias removal mechanism of Double Q-Learning to generate low-latency and low-energy caching strategies in real time under a three-layer heterogeneous caching architecture. The state and action are defined for the optimization algorithm. Status: Time Slot The global state is represented as ;in As a region indicator, it uses one-hot encoding to quantify the region type of the requesting vehicle. The encoding rule is: if the vehicle is in a dense region formed by clusters, the corresponding encoding dimension is set to 1; if the vehicle is in a sparse region without cluster coverage, the encoding dimension is set to 0. For cluster attributes, first define the distinguishing regions, set the cluster ID of the sparse region to -1, and the cluster ID of the dense region to a valid non-negative value, and then normalize the valid cluster ID of the dense region, that is, map it to the interval [0,1]. Content characteristics, including normalized content size. and the one-heat type vector obtained after PCA dimensionality reduction ; This refers to the instantaneous utilization of resources, including RSU, UAV, and cluster cache. Action: Time Slot The action space is represented as Among them, action 0 means caching to the nearest roadside unit, action 1 means caching to the cluster head, action 2 means caching to the drone, and action 3 means not caching but returning to the source from the base station. A region-aware mask is applied to the actions of vehicles in sparse areas, so that vehicles in sparse areas can only choose action 0 or 3. Step 5.2: Use the depth-deterministic gradient algorithm to solve the UAV trajectory optimization problem; The UAV trajectory control problem is modeled as a Markov decision process in a continuous action space. A state and action representation adapted to content distribution tasks is designed, and a continuous trajectory control algorithm for a trajectory agent is proposed to optimize the flight path of the UAV in real time in a dynamic environment to minimize the weighted sum of system latency and energy consumption. The state and action are defined for the control algorithm. Status; Time Slot The global state is represented as ;in The current location information of the drone is converted into a two-dimensional trajectory optimization with its fixed flight altitude. For drone speed; Let the content request distribution density of the UAV service area be set to , Let be the normalized vehicle density of the k-th cluster in time slot t; Action; time slot The action space is represented as ;in Indicates time slot The displacement increment along the x-axis; Indicates time slot The displacement increment along the y-axis; where the velocity constraint determines the displacement constraint. ; Range constraints To define the flight area; Step 5.3: Collaborative optimization based on dual-agent reinforcement learning algorithm; The algorithm employs a dual-agent reinforcement learning approach for collaborative optimization, based on a three-layer progressive architecture. The bottom layer is the vehicle-to-everything (V2X) environment layer, providing status and feedback; the middle layer is the dual-agent decision-making layer, enabling differentiated optimization; and the top layer is the system-level optimization objective layer, unifying evaluation standards. Through the design concepts of functional division of labor, temporal collaboration, and information linkage, the decoupling and optimization of the two types of sub-problems are achieved. The global reward function is: (31); in, This is the global reward function; These represent the reward weight and normalized metric for cache hits, respectively. These represent the reward weight and normalized index for vehicles requesting drone coverage, respectively. These represent the reward weight and normalized index for the collaboration between the two agents, respectively.

[0016] The specific process of the dual-agent reinforcement learning collaborative optimization algorithm is as follows; Single agent independent training: The cache agent (D3QN) uses a dual network architecture to sample experience and update parameters; the trajectory agent (DDPG) generates continuous actions, updates parameters using the policy gradient method, and introduces exploration noise to ensure action diversity; Dual-agent collaborative feedback: The two agents interact and feedback through the environment and reward function, influencing each other's decision-making direction; the system performance is evaluated in fixed rounds, and the reward weights and hyperparameters are dynamically adjusted to ensure collaborative convergence to the global optimum; Training Termination and Deployment: Training is terminated when the number of iterations reaches the upper limit, or when the reward value is stable and the performance indicators meet the target. The optimized parameters are then deployed to achieve dynamic decision-making and adaptation.

[0017] The optimization results of the caching strategy subproblem provide key constraints such as content request distribution and cache node location for trajectory optimization. The output of trajectory optimization serves as the environmental state input for caching strategy updates. Through the joint reward mechanism of the dual-agent collaborative framework, the optimization results of the two subproblems are globally integrated, ultimately ensuring that the overall optimization performance of the original problem is not compromised by decoupling.

[0018] The beneficial technical effects of this invention are as follows: Addressing the communication service requirements of low latency, low energy consumption, and high resource utilization in highly dynamic vehicle-to-everything (V2X) scenarios, this invention proposes a dual-agent reinforcement learning collaborative optimization framework. By precisely decoupling the strong coupling between cache placement and UAV trajectory optimization through a cache agent (D3QN) and a trajectory agent (DDPG), it overcomes the drawbacks of fragmented optimization in traditional methods. Simultaneously, by combining a vehicle dynamic clustering mechanism and a hierarchical asynchronous federated learning prediction model, it achieves accurate perception and prediction of dynamic changes in network topology, vehicle interests, and content popularity, endowing the cache strategy and UAV trajectory with real-time adaptability. This collaborative optimization framework effectively improves cache hit rate, significantly reduces average vehicle service latency and average system energy consumption, and, being data-driven and not reliant on precise mathematical models, exhibits good robustness to complex dynamic environments. Its clear architecture demonstrates strong potential for engineering implementation. Attached Figure Description

[0019] Figure 1 This is an overall flowchart of the method of the present invention.

[0020] Figure 2 This is a schematic diagram of the drone-assisted vehicle-to-everything (V2X) collaborative edge caching system architecture constructed according to the present invention.

[0021] Figure 3 This is a schematic diagram of the vehicle dynamic clustering process based on location and interest in this invention.

[0022] Figure 4 This is a schematic diagram of the hierarchical asynchronous federated learning content popularity prediction model in this invention.

[0023] Figure 5 This is a flowchart of the collaborative optimization algorithm based on dual-agent reinforcement learning in this invention.

[0024] Figure 6 This is a comparison chart of the average latency of the dual-agent reinforcement learning collaborative optimization algorithm of this invention.

[0025] Figure 7 This is a comparative graph showing the average energy consumption of the dual-agent reinforcement learning collaborative optimization algorithm of this invention.

[0026] Figure 8 This is a comparison chart of cache hit rates for the dual-agent reinforcement learning collaborative optimization algorithm of this invention.

[0027] Figure 9 This is a comparison chart of the drone coverage experiment using the dual-agent reinforcement learning collaborative optimization algorithm of this invention.

[0028] Figure 10 This is a comparative experimental chart showing the average reward of the dual-agent reinforcement learning collaborative optimization algorithm of this invention.

[0029] Figure 11 This is a comparison chart of the average reward experiment between the vehicle dynamic clustering and asynchronous federated prediction module ablation experiments of the present invention. Detailed Implementation

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments; like Figure 1 As shown, this invention proposes a collaborative caching method based on dual-agent reinforcement learning under UAV assistance. After introducing vehicle dynamic clustering and hierarchical asynchronous federated prediction model, it adopts a dual-agent reinforcement learning joint optimization algorithm based on the fusion of D3QN and DDPG for iterative learning to complete the reasonable placement of cache resources; specifically including the following steps; Step 1: In UAV-assisted vehicle-to-everything (V2X) computing, consider three different types of edge caching nodes in the vehicle edge computing network, including roadside units, unmanned and vehicle nodes. The constructed UAV-assisted V2X model is as follows: Figure 2 As shown, a macro base station is represented as... M roadside units (RSUs) are represented as follows: And L unmanned aerial vehicles (UAVs) are represented as Let N be the number of vehicles moving on the road. Provides content distribution services. Macro base stations are deployed in city centers or at high altitudes, with coordinates as follows: It is tightly connected to the core network via fiber optic cable, serving as the origin hub to cover all content. Roadside units are fixedly installed on both sides of the road, providing content distribution services to vehicles moving along the road; their coordinates are... Drones also possess caching and communication capabilities, serving as mobile aerial nodes and providing services in real time. The coordinates are Set flight altitude The problem is constant, thus transforming it into a two-dimensional plane problem. Vehicles move on city roads and can cache requested content locally; their time... The coordinates are The system in discrete time slots Running on this platform, the network topology and channel conditions remain approximately unchanged within a single time slot. Edge nodes are connected to vehicles, and vehicles are connected to each other via wireless communication links.

[0031] Furthermore, a region-aware hierarchical request forwarding strategy is proposed to ensure low-complexity access in sparse areas and high-concurrency resilience in dense areas. In sparse areas, where vehicles are dispersed, requests are preferentially searched in roadside unit caches; if a hit is found, a direct response is provided; otherwise, content is retrieved from the origin via a base station. In dense areas, where request frequency is high, responses are preferentially provided by the cluster head cache of the vehicle's cluster; if a hit is not found, a drone cache serves as a second layer of protection, leveraging its mobility for rapid coverage; if both levels fail, the macro base station ultimately retrieves the content from the origin. This strategy uses dynamic clustering based on vehicle location and interest, and hierarchical asynchronous federated learning to predict content popularity. Edge nodes are pre-cached to improve cache hit rate, and dual-agent reinforcement learning is used to collaboratively optimize cache decisions and drone trajectories. Through iteration, a stable caching strategy is obtained, effectively reducing average service latency and system energy consumption.

[0032] like Figure 3 As shown, different patterns represent vehicles with different preferences. In intelligent transportation systems, vehicle distribution exhibits significant dynamism and heterogeneity. For example, urban commercial districts often experience dense vehicle traffic, frequent and diverse content requests, and an urgent need for low-latency, high-bandwidth services; while suburban roads have sparse vehicle distribution and relatively stable service demands. This difference requires the network to have dynamic adaptability, achieving precise resource scheduling through reasonable regional division and vehicle clustering.

[0033] Step 2: Propose a location-interest-based vehicle dynamic clustering method, which uses vehicle communication reachability and content interest similarity as the core division criteria to achieve clustering of vehicles that are geographically close and have similar interests, thus providing support for differentiated services; the specific process is as follows: Preferably, in step 2, the process of implementing vehicle dynamic clustering includes: Step 2.1, Spatial candidate for region division; Since vehicles move at high speed and have a reachable communication range in the Internet of Vehicles, it is necessary to first screen the vehicles and retain the candidate objects with stable communication capabilities to determine the effective vehicles participating in clustering. Step 2.2, Quantitative representation of interest features: Vehicles in dense areas all have stable communication capabilities. Then, through fine division of interest features, vehicles that are geographically close but have different interests are further separated to achieve sub-clustering of vehicles with similar preferences. First, extract the content type information; second, count all occurrences of content type, resulting in a total of... Different types; then calculate the weighted score for each vehicle user for each type; assuming the vehicle For content The rating is Then the vehicle In the time slot Previous request history is ,vehicle Interest vector The calculation formula is as follows: (1); in, It is a length of The time-weighted interest vector represents the degree of interest that vehicles have in each type of content. It is the time-degradation factor; yes The time-deterioration factor is superimposed within the time slot; A content type vector; Step 2.3, Dimensionality Reduction and Optimization of High-Dimensional Features: To address the dimensionality redundancy problem in the original interest vector (such as the high correlation between real-time navigation and traffic warning preferences), Principal Component Analysis (PCA) is used for unsupervised dimensionality reduction. Through linear transformation, the data is mapped to a low-dimensional space, extracting core features while retaining the main variance information. All vehicle user interest vectors are combined into a matrix ,in It's the number of users. It is the real number field; therefore, calculation Find the covariance matrix of the given matrix and determine its eigenvalues ​​and eigenvectors; select the first... The eigenvectors corresponding to the largest eigenvalues ​​are used as principal components to form the projection matrix. Finally, the reduced-dimensional interest vector Represented as; (2); Each user is randomly represented as a... 3D interest vector It can comprehensively reflect users' preferences for different types of movies; Step 2.4: Clustering based on interest preferences; use the K-means algorithm to cluster low-dimensional interest vectors, dividing vehicles in dense areas into clusters. Interest clusters And select the vehicle with the largest cache capacity as the cluster head for each cluster; Step 2.5, Dynamic update of vehicle clusters; In the Internet of Vehicles, vehicle movement and the evolution of content interests can cause the initial clusters to fail. It is necessary to introduce a timed dynamic update mechanism, which adapts to the dynamic nature of the system through three steps: location update, interest update, and re-clustering. During position updates, the vehicle moves at a constant speed on the road, and the formula for the change in vehicle position is: (3); in, Indicates the vehicle's speed; Indicates the direction of vehicle movement; Indicates unit conversion; This is the time when the vehicle's location was updated; It is the vehicle in the time slot Location; This is the vehicle's updated location; It is the unit component of the vehicle's direction of movement on the x-axis; It is the unit component of the vehicle's direction of movement on the y-axis; When updating interests, vehicle interests evolve with the request history. The formula for updating the interest vector is: (4); in, Indicates to retain the most recent A request record for each time slot; This represents the updated interest vector; Each interval Each training round re-executes clustering based on the updated vehicle positions and interests.

[0034] like Figure 4 As shown, in the vehicle-to-everything (V2X) caching system, accurate content popularity prediction is a prerequisite for reducing the average system latency and energy consumption. A hierarchical asynchronous federated content popularity prediction mechanism is proposed. Combining the vehicle dynamic clustering results, a three-level architecture of local update, hierarchical aggregation, and time-weighted optimization is used to improve the timeliness and accuracy of prediction while protecting privacy.

[0035] Step 3: Combining the vehicle dynamic clustering results, construct a hierarchical prediction system of cluster-local model, sparse region-local model, and global model: In dense regions, aggregate vehicle request features at the cluster level to generate a cluster-local popularity model; in sparse regions, directly construct a sparse region-local model based on the request history of a single vehicle; the global model asynchronously aggregates local models, integrates regional preferences and global trends, and ultimately provides multi-granularity popularity prior knowledge for caching decisions; the specific process is as follows: Step 3.1, Construction of the local model; The local model directly reflects the content preferences of vehicles or clusters. To address the issues of strong time-series requirements and rapid decay of popularity over time in vehicle-to-everything (V2X) requests, the local model adopts a time-weighted request frequency calculation method, making recent requests contribute more to popularity prediction; The specific process is as follows; Step 3.1.1, Local Model of Dense Areas; The cluster-local model is built on a cluster-by-cluster basis, based on the request history of all vehicles within the cluster. Each cluster... Record of every content request for the vehicle for; (5); in, For content; This represents the cumulative number of requests for this content. The time slot in which this content was most recently requested; When cluster In-vehicles in time slots Initiate content When making a request, request history The update rule formula is: (6); in, This is the set of content that has been requested. For each subsequent request, the cumulative count is incremented by 1, and the most recent time slot request history is updated. Let the time-weighted request frequency be cluster. Local model for content Popularity score The calculation formula is: (7); in, This is the sum of time decay factors between the most recent request time slots; For the current time slot; For content The most recently requested time slot; For content The number of content requests; For content The number of content requests is used to reduce the weight of historical requests and highlight the impact of recent requests; thus, the cluster-local model is saved as... ; Step 3.1.2, Local Model of Sparse Region; Local Model of Vehicles in Sparse Region (Content) Popularity score The calculation formula is consistent with the cluster local model, only the statistical object is changed from all vehicles within the cluster to vehicles in a single sparse area, and thus the local model is saved as... This is for subsequent global aggregation. Step 3.2, Hierarchical Asynchronous Aggregation: Based on the vehicle clustering results, a global popularity model is generated to address regional request pattern differences. The cluster-local model has been constructed using the request history of all vehicles within the cluster. Essentially, it is an implicit aggregation of vehicle request preferences within the cluster, eliminating the need for additional collection and aggregation of individual vehicle models within the cluster. This reduces communication and computational overhead, and the model is directly generated based on the cluster-level request history. This is equivalent to first calculating the local model of a single vehicle within the cluster and then averaging the results, thus realizing implicit aggregation of preferences within the cluster. Let the content type function be , The total number of content types, including content Mapping to type collection This facilitates aggregating the popularity trends of similar content by type; for types Corresponding content collection Global popularity score The prediction results of the fusion cluster local model and the sparse region vehicle local model are calculated using the following formula; (8); in, For content type For type Cluster local model for content Popularity score; For type Sparse vehicle local model for content The average popularity score; The weighting coefficients are used to balance the contributions.

[0036] To adapt to the high dynamism of vehicle-to-everything (V2X) communication and control communication overhead, an asynchronous triggering mechanism is adopted, whereby the cumulative number of local model updates reaches a threshold. During periods of frequent changes in request patterns, update the global model promptly.

[0037] Step 4: Construct a cache placement problem that jointly minimizes the average latency and average energy consumption in the system, and decouple it into two sub-problems. The specific process is as follows: Step 4.1: Construct the system's cache model and UAV trajectory model. The specific process is as follows: Step 4.1.1: Construct the system's caching model; assuming... For a collection of cache nodes, Indicates the first A cluster head, Indicates the first One roadside unit, Indicates the first One drone; define that within the same time slot, a vehicle cannot simultaneously request content from multiple nodes; assume the cache capacity of the cache node is... ,content The content size is The cache decision variable is At the same time, cache capacity constraints must be met. (9); Step 4.1.2: Construct the UAV trajectory model for the system; time slot The location of the drone at that time ,in The coordinates are the x-coordinate, y-coordinate, and altitude of the drone's position, and the drone's flight speed. Calculated as; (10); in, This is the time interval for movement; For time slots The location of the drone at that time; The motion constraint formula for a drone is expressed as: (11); The speed of the drone is limited to the maximum and minimum value And its location is within the permitted flight zone. Inside.

[0038] Step 4.2: Construct an average latency model for the system content request service process. The specific process is as follows: Step 4.2.1, Components of System Delay; System delay includes transmission delay, processing delay, queuing delay, and source latency; Transmission delay is determined by data size and link rate; processing delay is inversely proportional to node processing rate and modeled according to exponential distribution; queuing delay is introduced under multi-user resource contention, and the average queuing delay is approximated by the M / M / 1 queuing model; when neither the drone nor the roadside unit hits the cache, the content is transmitted back to the requesting vehicle by the macro base station, resulting in back-to-origin delay. Step 4.2.2, delay calculation in sparse areas: Vehicles prioritize sending requests to the roadside unit with the closest geographical distance. The path follows a two-layer routing logic: if the roadside unit's cache is hit, transmission is performed directly; if the roadside unit's cache is not hit, transmission is performed back to the source macro base station. Queuing delays for roadside units in sparse areas are negligible; delays when hitting roadside units are negligible. Represented as; (12); in, For the transmission delay of the roadside unit; This is for the processing delay of the roadside unit; Miss delay Represented as: (13); in, This refers to the transmission delay from the macro base station to the roadside unit; For origin retrieval delay; Single request latency Represented as: (14); in, This is an indicator of whether the roadside unit has been hit; Step 4.2.3, delay calculation in dense areas; the path follows a three-level routing strategy: priority to cluster head hit, second hit by UAV, and base station backhaul when neither hit, making full use of the complementarity between the mobility of UAV and the close-range coverage of ground cluster head; set up Assuming the signaling interaction delay between the UAV and the cluster head; For the transmission delay from the macro base station to the drone; Hit cluster head delay Represented as: (15); in, , , These are the cluster head's transmission delay, processing delay, and queuing delay, respectively. Hitting the drone delay Represented as: (16); in, , , These are the transmission delay, processing delay, and queuing delay of the drone, respectively. All misses delay Represented as: (17); Single request latency Represented as: (18); in, , These are indicators that represent whether the cluster head and the drone have been hit; Step 4.2.4, Calculation of system average latency; global arbitrary vehicle In the time slot Request content latency for: (19); in, For vehicles In the time slot Related nodes Indicator; vehicles In the time slot Related nodes Request content Transmission delay, processing delay, and queuing delay; Global average latency for: (20); in, This represents the total time slots.

[0039] Step 4.3: Construct an average energy consumption model for the system's content request service process. The specific process is as follows: Step 4.3.1, Calculation of system energy consumption; The system energy consumption consists of the energy consumption of the UAV, the energy consumption of the roadside unit, the energy consumption of the cluster head vehicle, and the backhaul energy consumption of the macro base station. The energy consumption of drones includes energy consumption during drone movement, energy consumption related to communication, hovering energy consumption when drones hover to provide content services to vehicles, and processing energy consumption. Among them, the movement energy consumption is modeled based on the power model of rotary-wing drones, the hovering energy consumption is the basic power consumption to maintain the stability of the aircraft, and the communication and processing energy consumption is related to the service duration and the number of requests. The energy consumption of roadside units and cluster head vehicles mainly comes from the communication and processing energy consumption of service vehicles; the energy consumption of macro base stations is only generated when the cache is not hit, including the communication and processing energy consumption of backhaul content. Step 4.3.2: Energy consumption calculation for sparse and dense regions; similar to the latency model, the energy consumption of a single request under different paths in the sparse region. Represented as: (twenty one); in, The communication power consumption of the roadside unit; The communication energy consumption from the macro base station to the roadside unit; This represents the total energy consumption of the roadside unit. For macro base station back-to-source power consumption; In densely populated areas, the energy consumption of a single request under different paths for: (twenty two); in, This indicates the total energy consumption of the leading vehicle. Communication power consumption for cluster-head vehicles; Indicates the mobile energy consumption of the drone; Indicates the hovering energy consumption of the drone; This indicates the communication and processing energy consumption of the drone; This indicates the energy consumption for transmission from the cluster head to the drone; This indicates the energy consumption for transmission from the drone to the macro base station; Step 4.3.3: Calculation of system average energy consumption; global calculation for any vehicle. In the time slot Request content energy consumption for: (twenty three); in, For vehicles In the time slot Related nodes Request content Communication energy consumption and processing energy consumption; Global average energy consumption for: (twenty four); in, For drones in time slots The sum of the energy consumption during movement and the energy consumption during hovering.

[0040] Step 4.4: Define the optimization problem, with the ultimate optimization objective being to jointly minimize the system's average latency and average energy consumption. Introducing weights A linear normalization method is used to map all targets to the [0,1] interval, normalizing the average delay and average energy consumption. The formula is: (25); (26); in, , These represent the estimated maximum latency and maximum energy consumption, respectively. The optimization problem is represented as: (27); Satisfy constraints: (28); Among them, C1 indicates that the total amount of content stored in the edge node cache must not exceed its cache capacity; C2 indicates that the cached variables are binary variables; C3 and C5 indicate that the vehicle speed must be within the range allowed by physical performance, and the calculation method of speed is specified; C4 indicates that the drone must move within the preset feasible flight range.

[0041] Step 4.5: Decouple the optimization problem into two sub-problems: cache decision and UAV trajectory optimization; Based on discrete and continuous variables, the original optimization problem is decoupled into two sub-problems: the buffer decision sub-problem and the UAV trajectory optimization sub-problem. The caching decision subproblem is represented as: (29); Satisfy constraints C1 and C2; The drone trajectory optimization subproblem is represented as: (30); The constraints C3, C4, and C5 are satisfied.

[0042] like Figure 5As shown, after initializing the dual-agent network, federated learning components, and experience pool, the algorithm configures hyperparameters and resets the environment to obtain the joint state of the cache and trajectory. The cache agent selects discrete cache actions using an ε-greedy method based on the Dueling network, while the trajectory agent generates noisy continuous trajectory actions through the Actor network. After execution, it calculates the joint reward of fusion latency, energy consumption, UAV coverage, and hit rate, and stores the experience. When the experience pool is sufficient, the cache agent updates the network by minimizing the temporal difference error, while the trajectory agent alternately updates the Critic network and Actor network and softly updates the target network. At the end of each round, the federated model is asynchronously aggregated to generate global popularity to guide pre-caching, iterating until convergence and outputting the optimal collaborative strategy.

[0043] Step 5: The competitive dual-deep Q-network algorithm is used to solve the cache decision problem, and the depth-determined strategy gradient algorithm is used to solve the UAV trajectory optimization problem. Through the dual-agent collaborative optimization framework, the cache placement strategy is obtained. The specific process is as follows: Step 5.1: Use a competitive dual-depth Q-network algorithm to solve the cache decision problem; Define the state and action for the algorithm; Status: Time Slot The global state is represented as ;in As a region indicator, it achieves a quantitative representation of the region type of the requesting vehicle through one-hot encoding; For cluster attributes, first define the distinguishing regions, and then normalize the valid cluster IDs of the dense region, that is, map them to the [0,1] interval; Content characteristics, including normalized content size. and the one-heat type vector obtained after PCA dimensionality reduction ; This refers to the instantaneous utilization of resources, including RSU, UAV, and cluster cache. Action: Time Slot The action space is represented as Among them, action 0 means caching to the nearest roadside unit, action 1 means caching to the cluster head, action 2 means caching to the drone, and action 3 means not caching but returning to the source from the base station. A region-aware mask is applied to the actions of vehicles in sparse areas, so that vehicles in sparse areas can only choose action 0 or 3. Step 5.2: Use the depth-deterministic gradient algorithm to solve the UAV trajectory optimization problem; Define the state and action for the algorithm; Status; Time Slot The global state is represented as ;in The current location information of the drone is converted into a two-dimensional trajectory optimization with its fixed flight altitude. For drone speed; The content request distribution density within the UAV service area is defined as follows: , This represents the normalized vehicle density of the k-th cluster in time slot t; Action; time slot The action space is represented as ;in Indicates time slot The displacement increment along the x-axis; Indicates time slot The displacement increment along the y-axis; where the velocity constraint determines the displacement constraint. ; Range constraints To define the flight area; Step 5.3: Collaborative optimization based on dual-agent reinforcement learning algorithm; The algorithm employs a dual-agent reinforcement learning approach for collaborative optimization, based on a three-layer progressive architecture. The bottom layer is the vehicle-to-everything (V2X) environment layer, providing status and feedback; the middle layer is the dual-agent decision-making layer, enabling differentiated optimization; and the top layer is the system-level optimization objective layer, unifying evaluation standards. Through the design concepts of functional division of labor, temporal collaboration, and information linkage, the decoupling and optimization of the two types of sub-problems are achieved. The global reward function is: (31); in, This is the global reward function; These represent the reward weight and normalized metric for cache hits, respectively. These represent the reward weight and normalized index for vehicles requesting drone coverage, respectively. These represent the reward weight and normalized index for the collaboration between the two agents, respectively.

[0044] The specific process of the dual-agent reinforcement learning collaborative optimization algorithm is as follows; Single agent independent training: The cache agent (D3QN) uses a dual network architecture to sample experience and update parameters; the trajectory agent (DDPG) generates continuous actions, updates parameters using the policy gradient method, and introduces exploration noise to ensure action diversity; Dual-agent collaborative feedback: The two agents interact and feedback through the environment and reward function, influencing each other's decision-making direction; the system performance is evaluated in fixed rounds, and the reward weights and hyperparameters are dynamically adjusted to ensure collaborative convergence to the global optimum; Training Termination and Deployment: Training is terminated when the number of iterations reaches the upper limit, or when the reward value is stable and the performance indicators meet the target. The optimized parameters are then deployed to achieve dynamic decision-making and adaptation.

[0045] The optimization results of the caching strategy subproblem provide key constraints such as content request distribution and cache node location for trajectory optimization. The output of trajectory optimization serves as the environmental state input for caching strategy updates. Through the joint reward mechanism of the dual-agent collaborative framework, the optimization results of the two subproblems are globally integrated, ultimately ensuring that the overall optimization performance of the original problem is not compromised by decoupling.

[0046] Table 1: Pseudocode table of the joint optimization algorithm based on dual-agent reinforcement learning; .

[0047] Step 5.4, Experimental Evaluation; In a vehicle-to-everything (V2X) simulation environment, a heterogeneous network including macro base stations, roadside units, drones, and vehicles is constructed. Content catalogs and learning hyperparameters are configured, with average system latency, average energy consumption, cache hit rate, and drone coverage as indicators. Comparative algorithms are selected for comparative experiments, including baseline methods such as cache decision-making based on deep Q-networks, trajectory optimization based on dual-latency deterministic policy gradients, single-agent end-to-end joint optimization, periodic hotspot tracking of drone trajectories, and fixed cache replacement strategies. Ablation experiments are conducted by removing the hierarchical asynchronous federated prediction model (using centralized training) and fixing the vehicle clustering structure (stopping dynamic updates) to evaluate the contribution of each module. Results show that the proposed dual-agent collaborative optimization framework outperforms existing technologies in reducing latency and energy consumption, and improving cache hit rate and coverage. Furthermore, the hierarchical asynchronous federated prediction, dynamic clustering, and dual-agent collaborative decision-making mechanisms work together to effectively enhance the overall service quality of the system. The specific process is as follows: Step 5.4.1, Experimental Setup; To evaluate the performance of the proposed dual-agent reinforcement learning collaborative caching and trajectory optimization algorithm framework, simulation experiments were conducted in a vehicle-to-everything (V2X) scenario. The hardware platform was equipped with an Intel i7-13700HX CPU, 16GB DDR5 memory, and a 64-bit Windows 11 system. The algorithm stack was implemented in PyCharm 2025.1.3.1 environment based on Python 3.9 and PyTorch 2.0.1.

[0048] The simulation scenario is a 2 km × 2 km urban area; the base station is located at the origin, and four roadside units are symmetrically deployed in the corners (coverage radius 500 m); the rotary-wing UAV cruises at an altitude of 50 m above the ground with a coverage radius of 300 m, and its initial position is optimized by the proposed trajectory agent and updated every 1 second; 200 vehicles are deployed, with vehicle speeds uniformly distributed between 30-50 km / h; the on-board cache capacity is randomly allocated from 10 to 20 MB, while the roadside units and UAVs are equipped with 200 MB and 100 MB caches, respectively.

[0049] The content catalog contains 100 video clips, each ranging from 1 to 5 MB. These clips were extracted from the Movielens-1M dataset (a publicly available movie rating dataset). During the extraction process, data preprocessing was performed to remove inactive users to ensure the authenticity of content preferences, providing a realistic content resource foundation for subsequent caching strategies.

[0050] Regarding learning hyperparameters, both agents used a discount factor of 0.95, the experience replay pool stored 10,000 transition data points, and the number of mini-batch samples for gradient updates was 128; the initial value of ε-greedy exploration was 0.99, and it decayed exponentially with a factor of 0.995; each agent used a fully connected competitive network (two hidden layers: 128 and 64 neurons); all reported metrics were averaged over five independent runs with different random seeds.

[0051] Step 5.4.2, Evaluation Indicators and Comparison Methods; The evaluation algorithm framework is mainly based on four performance metrics: 1) Average latency: the average time to process content requests from all vehicle users in the system; 2) Average energy consumption: the average energy consumption of all service nodes in the system when processing vehicle user requests; 3) Cache hit rate: the ratio of cached content at edge nodes to total content requests; 4) UAV coverage: the proportion of requests within the communication coverage area of ​​UAVs to the total number of global requests.

[0052] The main algorithms considered for comparison are as follows: 1) Proposed: A dual-agent reinforcement learning collaborative optimization algorithm; 2) DA-DDQN-DDPG: The cache agent is replaced with a double deep Q-network (DQN), verifying the advantages of D3QN in bias estimation and state discriminability; 3) DA-D3QN-TD3: The trajectory agent is replaced with a double-delay deterministic policy gradient (TD3), evaluating the stability of DDPG in a high-dimensional continuous action space; 4) SA-PPO: A single-agent proximal policy optimization (PPO) algorithm is adopted; 5) SA-D3QN-HSF: The cache agent uses a single-agent D3QN to optimize the cache, and the UAV moves to the hotspot following (HSF) area with the highest historical request density according to a periodic trajectory strategy, without updating with environmental feedback; 6) SA-DDPG-IFLRU: The trajectory agent uses a single-agent DDPG to optimize the trajectory, and the cache strategy is fixed as a hybrid mechanism of "initial-frequency-based LRU" (Initial-Frequency-based LRU). 7) Proposed without HAFL: All node data is uploaded to the central server for centralized training without distributed aggregation, i.e., the ablation experiment of the hierarchical asynchronous federated prediction model; 8) Proposed without dynamiccluster: After the initial clustering is completed, the subsequent fixed vehicle positions, interests and clustering structures are not dynamically updated. The caching strategy and federated learning model loading are based on the initial clustering features to achieve pre-caching and data processing, i.e., the ablation experiment of the vehicle dynamic clustering module.

[0053] Step 5.4.3, Experimental Results and Analysis; The training results all used the sliding window averaging method, with a sliding window size of 200 to smooth the original data; (1) Performance comparison of dual-agent reinforcement learning collaborative optimization algorithms.

[0054] Figures 6-9 The convergence curves of each algorithm are shown in terms of average latency, average energy consumption, cache hit rate, and drone coverage. For example... Figure 6 As shown, the Proposed algorithm achieves the lowest average system latency, reducing it by approximately 0.52%, 1%, 2%, 2.5%, and 6.3% compared to DA-D3QN-TD3, DA-DDQN-DDPG, SA-PPO, SA-D3QN-HSF, and SA-DDPG-IFLRU, respectively. Figure 7 As shown, the Proposed algorithm achieves the lowest average system power consumption, reducing it by 6.8%, 14.1%, 27.8%, 34.4%, and 42% respectively compared to other algorithms; Figure 8 As shown, the Proposed algorithm achieves the highest cache hit rate, improving upon other algorithms by approximately 6%, 8%, 15.7%, 17.5%, and 36.2%, respectively. Figure 9 As shown, the Proposed algorithm achieves the highest drone coverage with a cache hit rate, improving by approximately 7.3%, 14.6%, 17.5%, 52%, and 13.5% compared to other algorithms.

[0055] The performance advantages of the Proposed algorithm stem from the synergistic effect of its core design. First, the Hierarchical Asynchronous Federated Prediction Model (HAFL) combines temporal weighting with regional preference aggregation to provide accurate popularity priors for caching decisions. Second, the bias-free estimation capability of the D3QN caching agent and the continuous spatial optimization characteristics of the DDPG trajectory agent work together, with the former achieving accurate matching of content and nodes and the latter tracking request hotspots in dense areas in real time. Third, location-interest-based dynamic clustering aggregates vehicles that are geographically close and have similar interests, reducing the dispersion of requests within clusters.

[0056] In the DA-D3QN-TD3 algorithm, the conservative policy update of TD3 causes the UAV to lag in exploration in scenarios with rapidly changing hotspots, resulting in trajectory response speed lagging behind the caching strategy and weakening the collaborative effect. In the DA-DDQN-DDPG algorithm, due to the lack of value-advantage decoupling capability of Dueling structure, DDQN has difficulty distinguishing the value of key actions in the high-dimensional vehicle network state space, causing caching decisions to deviate from the optimal and triggering trajectory-caching collaboration mismatch, which weakens the collaborative effect. In the SA-PPO algorithm, PPO needs to handle discrete caching and continuous trajectory actions simultaneously, resulting in a surge in policy gradient variance and an inability to take both objectives into account, ultimately making it difficult to stably learn an effective policy. In the SA-D3QN-HSF algorithm, the static hotspot path cannot track sudden changes in request distribution, causing the UAV trajectory update to lag behind the caching decision and resulting in a disconnect between service coverage and content placement. In the SA-DDPG-IFLRU algorithm, the caching strategy cannot predict the time-varying nature of content popularity, resulting in long-term mismatch of cached content and a large amount of back-to-origin and invalid movement energy consumption.

[0057] like Figure 10 As shown, the average reward of all algorithms increases with the number of training rounds, but the Proposed algorithm remains in the lead and converges optimally, improving the reward by approximately 9.6%, 11.6%, 19%, 31.8%, and 37.2% compared to DA-D3QN-TD3, DA-DDQN-DDPG, SA-PPO, SA-D3QN-HSF, and SA-DDPG-IFLRU, respectively, indicating that the proposed algorithm is cooperatively optimal.

[0058] (2) Ablation experiment of vehicle dynamic clustering and asynchronous federated prediction module.

[0059] Figure 11 This paper presents ablation experiments on the vehicle dynamic clustering and hierarchical asynchronous federated learning prediction modules. Since the average reward comprehensively reflects the optimization effect of multiple objectives such as cache latency and hit rate, and is highly correlated with system service efficiency and cache effectiveness, it was selected as the evaluation metric. With increasing training epochs, the average reward of all compared algorithms showed an upward trend, but the Proposed algorithm consistently maintained its lead, with a particularly significant advantage after convergence. The results indicate that the absence of either module leads to performance degradation, verifying their irreplaceable role in the algorithm. The hierarchical asynchronous federated learning prediction module provides accurate demand perception capabilities for dual-agent collaboration through dynamic content popularity prediction; the vehicle dynamic clustering module adapts to vehicle distribution and interest evolution in real time, optimizing service group partitioning. Together, they form a closed loop of "demand perception—service partitioning," jointly supporting the algorithm's performance advantages in latency control, multi-objective optimization, and cache efficiency.

[0060] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A collaborative caching method based on dual-agent reinforcement learning with UAV assistance, characterized in that, Specifically, the steps include the following: Step 1: In drone-assisted vehicle-to-everything (V2X) communication, construct a V2X edge collaborative caching model; Step 2: Utilize the location and preference information of vehicle users to achieve dynamic vehicle clustering, dividing vehicles into multiple interest clusters in dense areas and independent vehicles in sparse areas. Step 3: Calculate the popularity of the requested content using a hierarchical asynchronous federated learning prediction model; The hierarchical asynchronous federated learning prediction model includes: building a cluster-local model based on the request history of each interest cluster in the dense region, building a vehicle-local model based on the request history of independent vehicles in the sparse region, and asynchronously aggregating the cluster-local model and the vehicle-local model to generate a global popularity model. Step 4: Construct a cache placement optimization problem with the joint goal of minimizing the system's average latency and average energy consumption, and decouple this optimization problem into a cache decision subproblem and a UAV trajectory optimization subproblem; Step 5: Solve the cache decision subproblem using a competitive dual-depth Q-network algorithm to obtain the optimal cache placement strategy; The UAV trajectory optimization subproblem is solved using a deep deterministic strategy gradient algorithm to obtain the optimal UAV flight trajectory. Through a dual-agent collaborative optimization framework, the caching decision agent and the trajectory optimization agent interact and provide feedback, ultimately obtaining the system's optimal collaborative caching strategy.

2. The collaborative caching method based on dual-agent reinforcement learning with UAV assistance according to claim 1, characterized in that, In step 1, the vehicle-to-everything (V2X) edge collaborative caching model includes: Macro base stations serve as content origination centers; Roadside units are fixedly deployed on both sides of the road and have buffering and communication functions; Unmanned aerial vehicles (UAVs) serve as mobile aerial buffers and communication nodes. Vehicles move on city roads and have local caching capabilities.

3. The collaborative caching method based on dual-agent reinforcement learning with UAV assistance according to claim 2, characterized in that, In step 2, the process of implementing dynamic vehicle clustering includes: Step 2.1: Regional division and candidate vehicle screening; Step 2.2: Extraction and quantification of interest features; Step 2.3: Dimensionality reduction of interest features; Step 2.4: Cluster based on interests and preferences; Step 2.5: Dynamic update of vehicle clusters.

4. The collaborative caching method based on dual-agent reinforcement learning with UAV assistance according to claim 3, characterized in that, In step 3, the process of hierarchical asynchronous federated learning predicting content popularity includes: Step 3.1: Building the local model; For interest clusters in dense areas, a cluster-local model is constructed by calculating the popularity score of content based on the request history of all vehicles within the cluster and using time-weighted request frequency. For independent vehicles in sparse regions, the popularity score of the content is calculated using the same time-weighted method based on their own request history, and a local vehicle model is constructed. Specifically, the steps include the following: Step 3.1.1, Local Model of Dense Areas; In dense areas, vehicles within a cluster form stable cooperative units through inter-vehicle communication, and their content requests are similar; The cluster local model is built on a cluster-by-cluster basis, based on the request history of all vehicles within the cluster; Step 3.1.2, Local Model for Sparse Regions: Vehicles are distributed sparsely in sparse regions and their request patterns are highly independent. Therefore, a local model is built directly based on the request history of a single vehicle to capture individual preferences. Step 3.2, Layered asynchronous aggregation; When the cumulative number of local model updates reaches a preset threshold, global aggregation is triggered. The prediction results of all cluster local models and sparse area vehicle local models are aggregated according to content type, and the global popularity score of the content is obtained by weighted calculation to form a global popularity model.

5. The collaborative caching method based on dual-agent reinforcement learning with UAV assistance according to claim 4, characterized in that, The specific process of step 4 is as follows: Step 4.1: Construct the system's cache model and UAV trajectory model; Step 4.2: Construct an average latency model for the system's content request service process; Step 4.3: Construct an average energy consumption model during the system content request service process; Step 4.4: Define the optimization problem, with the ultimate optimization objective being to jointly minimize the system's average latency and average energy consumption. Step 4.5: Decouple the optimization problem into two sub-problems: cache decision and UAV trajectory optimization.

6. The collaborative caching method based on dual-agent reinforcement learning with UAV assistance according to claim 5, characterized in that, The specific process of step 4.1 is as follows: Step 4.1.1: Construct the system's caching model; Step 4.1.2: Construct the UAV trajectory model for the system.

7. The collaborative caching method based on dual-agent reinforcement learning with UAV assistance according to claim 5, characterized in that, The specific process of step 4.2 is as follows: Step 4.2.1, Components of System Delay; System delay includes transmission delay, processing delay, queuing delay, and source latency; Step 4.2.2: Calculation of time delay in the sparse region; Step 4.2.3: Delay calculation for dense areas; Step 4.2.4: Calculation of system average delay.

8. The collaborative caching method based on dual-agent reinforcement learning with UAV assistance according to claim 5, characterized in that, The specific process of step 4.3 is as follows: Step 4.3.1, Calculation of UAV energy consumption; Step 4.3.2, Calculation of energy consumption of roadside units; Step 4.3.3: Calculation of energy consumption of the leading vehicle; Step 4.3.4: Calculation of backhaul power consumption of macro base stations; Step 4.3.5: Energy consumption calculation for sparse and dense regions; Step 4.3.6: Calculation of average system energy consumption.

9. The collaborative caching method based on dual-agent reinforcement learning with UAV assistance according to claim 5, characterized in that, In step 4.4, with the goal of minimizing the weighted sum of time delay and energy consumption, weights are introduced and a linear normalization method is used to map all objectives to the [0,1] interval; In step 4.5, the original optimization problem is decoupled into two sub-problems based on discrete and continuous variables: the buffer decision sub-problem and the UAV trajectory optimization sub-problem.

10. The collaborative caching method based on dual-agent reinforcement learning with UAV assistance according to claim 1, characterized in that, The specific process of step 5 is as follows: Step 5.1: Use a competitive dual-depth Q-network algorithm to solve the cache decision problem; Step 5.2: Use the depth-deterministic gradient algorithm to solve the UAV trajectory optimization problem; Step 5.3: Collaborative optimization based on dual-agent reinforcement learning algorithm.

Images