Intelligent cooperative caching strategy based on asynchronous federated learning and social perception in Internet of Vehicles

Abstract

The invention discloses an intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles. The intelligent cooperative caching strategy comprises the steps that a user vehicle sends a caching request to a local RSU; local RSUs, nearby user vehicles, adjacent RSUs and macro base stations with request contents are selected according to priorities, cache requests of the user vehicles are responded, and a social network is adopted for communication between the vehicles; after the user vehicle receives the request content, the local cache content is updated by adopting a vehicle local local model, and the local local model is updated based on the vehicle local experience pool data; the selected vehicle uploads the updated local model to a local RSU; when the local RSU receives the local model, the global model is updated based on the vehicle weight; the local RSU uses the global model to update the local cache content based on the requested content of the vehicle within its service area. According to the method, the cache is cooperatively updated through the DQN training under the asynchronous FL, so that the transmission delay of the request content can be reduced, and the cache hit rate is improved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to an intelligent cooperative caching strategy based on asynchronous federated learning and social perception in an Internet of Vehicles. Background Art

[0002] With the development of Internet of Vehicles and edge computing, caching technology has become a key technology to support cars to enjoy diversified multimedia real-time applications such as intelligent navigation, efficient pattern recognition, and rich multimedia entertainment experience, greatly improving the convenience and entertainment of driving. Considering the high demand for computing, communication, and storage resources of in-vehicle applications, the requirements for service response speed and network bandwidth are very strict. Relying on its powerful computing and caching capabilities, smart vehicles achieve real-time interaction with edge devices, pedestrians, and vehicles through communication technologies such as C-V2X and LTE-V. Edge caching technology reduces repeated data transmission, optimizes transmission efficiency, responds to the challenges of telematic communication data, reduces system burden and energy consumption, and meets the needs of computationally intensive, low-latency in-vehicle applications. In addition, the edge cache capacity is limited, which makes the design of an efficient collaborative caching scheme an important research hotspot.

[0003] In the collaborative caching system, the selection of service nodes and the update of cache content are two key issues that need to be solved urgently. The rise of federated learning and some learning algorithms has solved some of the problems. For example, some studies use federated learning algorithms to select high-quality service nodes, aggregate all vehicle models, and build a global model. Based on this model, popular content is predicted and cached in advance to cooperative vehicles or roadside units. Collaborative caching reduces access delays and improves vehicle service quality. In traditional federated learning, the global model can be periodically updated by aggregating the local models of all vehicles. However, vehicles may often drive out of the coverage of the edge server and fail to upload all local models as expected, which will lead to a decrease in the accuracy of the global model. Asynchronous federated learning does not need to aggregate the local models of all vehicles. The more local models uploaded, the more accurate the global model can be. Vehicle mobility will greatly affect the performance of asynchronous federated learning. At the same time, the rise of intelligent transportation also applies deep reinforcement learning methods to edge caching systems, introduces reinforcement learning caching strategies, and each roadside unit agent independently selects cache actions to optimize resources to maximize cumulative rewards. However, the huge state-action space in the learning algorithm has led to an increasingly prominent problem of resource consumption. Summary of the invention

[0004] The present invention provides an intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles. Social perception between vehicles is added, and the cache is collaboratively updated through deep reinforcement learning training under the asynchronous federated learning framework, which can reduce the transmission delay of the requested content and improve the cache hit rate.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] An intelligent cooperative caching strategy based on asynchronous federated learning and social perception in Internet of Vehicles, including:

[0007] The user vehicle sends a cache request to the local RSU;

[0008] After receiving the cache request, the local RSU determines whether there is a request content corresponding to the cache request in its own cache content: if yes, the request content is sent to the user vehicle; if not, a nearby user vehicle, a neighboring RSU or a macro base station with the request content is selected according to a preset priority to respond to the cache request of the user vehicle, and the request content is also sent to the local RSU;

[0009] Each user vehicle within the coverage of the local RSU: after receiving the request content corresponding to each cache request, the local cache update model of the vehicle is used to update the local cache content of the vehicle, and the local cache update model of the vehicle is updated and trained based on the current experience pool data of the vehicle;

[0010] After the local cache update model of each selected vehicle is updated, it is uploaded to the local RSU; each selected vehicle is selected by its local RSU according to the residence time of the vehicle in its coverage area;

[0011] When the local RSU receives any local cache update model, it updates the global cache update model based on the corresponding vehicle weight;

[0012] The local RSU uses the current global cache update model to update the cache content of the local RSU based on the request content corresponding to the cache requests of all user vehicles in its service area.

[0013] Furthermore, the preset priorities, from high to low, are: nearby user vehicles, adjacent RSUs and macro base stations, and the methods of sending request contents are:

[0014] The nearby user vehicles use social perception between vehicles to send the request content to the user vehicle;

[0015] The neighboring RSU first sends the request content to the local RSU, which then sends it to the user vehicle;

[0016] The macro base station receives the cache request from the user vehicle and then directly sends the request content to the user vehicle.

[0017] Furthermore, the local cache update model of the vehicle is abbreviated as the local model, and the global cache update model of the RSU is abbreviated as the global model; the initial local model of each vehicle is obtained by downloading the global model of the local RSU; the global model and the local model use a deep reinforcement learning algorithm to decide whether to update their respective cache contents;

[0018] The global model uses a deep reinforcement learning algorithm to decide whether to update the cache content as follows:

[0019] Status: includes cache status and request status of preset multiple cache contents, expressed as in, represents the state of the global model of the nth RSU in time slot t, It represents the cache status of K preset cache contents corresponding to the global model of the nth RSU in the t time slot, represents the cache status of the global model of the nth RSU for the kth preset cache content in the tth time slot, Represents cached. Represents not cached; represents the request status of the i-th user vehicle for K preset cache contents in time slot t, represents the request status of the i-th vehicle for the k-th preset cache content in the t-th time slot, Send a request on behalf of Indicates that the request was not sent; P k represents the cache location of the kth preset cache content; V is the set of user vehicle numbers within the coverage area of ​​the nth RSU;

[0020] The action is: whether to replace multiple cache contents with the request contents corresponding to the cache request, expressed as in, represents the action of the global model of the nth RSU in time slot t, Indicates whether the global model of the nth RSU replaces the kth preset cache content with the request content corresponding to the cache request currently received from the i-th user vehicle in the tth time slot, Indicates replacement, Indicates no replacement;

[0021] The rewards are: minimize the cache request latency of each user’s vehicle;

[0022] The local model of the i-th user vehicle uses a deep reinforcement learning algorithm to decide whether to update the cache content as follows:

[0023] Status: includes cache status and request status of preset multiple cache contents, expressed as in, represents the state of the current local model at time slot t, represents the cache status of the current local model for K preset cache contents in time slot t, represents the cache status of the current local model for the kth preset cache content in time slot t, Represents cached. Represents not cached; represents the request status of the i-th user vehicle for K preset cache contents in time slot t, represents the request status of the i-th vehicle for the k-th preset cache content in the t-th time slot, Send a request on behalf of Indicates that the request was not sent; P k represents the cache location of the kth preset cache content; V is the set of user vehicle numbers within the coverage area of ​​the nth RSU;

[0024] The action is: whether to replace multiple cache contents with the request contents corresponding to the cache request, expressed as in, represents the action of the current local model in time slot t, Indicates whether the current local model replaces the k-th preset cache content with the requested content obtained by the cache request in the t time slot. Indicates replacement, Indicates no replacement;

[0025] The reward is: minimize the cache request latency of the i-th user's vehicle.

[0026] Furthermore, the reward functions of the global model and each local model are:

[0027]

[0028] In the reward function of the global model, (S t ,A t )Pick In the reward function of the local model, (S t ,A t )Pick D t represents the delayed reward at time t; λ1, λ2, λ3, and λ4 represent the learning rates in the reward function of nearby vehicles, local RSU, nearby RSU, and macro base station response cache requests, respectively; d V，i d R，i , d M，i They represent the transmission delays of nearby vehicles, local RSU, nearby RSU, and macro base station in response to the cache request of the i-th vehicle.

[0029] Furthermore, the transmission delay d of the nearby vehicles in response to the cache request of the i-th vehicle is V，i The calculation formula is:

[0030]

[0031]

[0032] Where s represents the size of the request content in response to the cache request; R V，i Represents the neighboring vehicles V and the i-th vehicle V i The transmission rate between them; α represents the contact rate of the nearby vehicle V, that is, the probability of the nearby vehicle V establishing communication with the vehicle Vi; N represents the total number of vehicles with requested content cached in time slot t within the coverage area of ​​the local RSU. t represents the total number of vehicles in the coverage area of ​​the local RSU in time slot t; B is the available bandwidth; p V Indicates the transmit power level used by nearby vehicles, dis(V, V i ) represents the neighboring vehicle V and vehicle V i The distance between i (dis(V,V i )) indicates vehicle V i The channel gain with the nearby vehicle V, is the noise power;

[0033] The transmission delay d of the local RSU in responding to the cache request of the i-th vehicle R，i The calculation formula is:

[0034]

[0035]

[0036] In the formula, R R，i Represents the local RSU and the user vehicle V i The transmission rate between B Indicates the transmit power level used by the local RSU; dis(S, V i ) represents the local RSU and vehicle V i The distance between i (dis(S,V i )) indicates the local RSU and vehicle V i The channel gain between

[0037] Transmission delay of the nearby RSU in response to the cache request of the i-th vehicle The calculation formula is:

[0038]

[0039] In the formula, R R-R Indicates the transmission rate between adjacent RSUs;

[0040] The transmission delay d of the macro base station in responding to the cache request of the i-th vehicle M，i The calculation formula is:

[0041]

[0042]

[0043] In the formula, R B，i Represents the macro base station and the user vehicle V i The transmission rate between M Indicates the transmit power level used by the macro base station; dis(M, V i ) represents the macro base station and the vehicle V i The distance between i (dis(M,V i )) represents the macro base station and the vehicle V i The channel gain between .

[0044] Furthermore, let x = V, M, S represent nearby vehicles, macro base stations, and local RSUs, respectively. Then the nearby vehicles, local RSUs, macro base stations, and vehicles V i The channel gain calculation formula between is:

[0045] h i (dis(x,V i ))=α i (dis(x,V i ))g i

[0046] In the formula, α i (dis(x,V i )) is the large-scale fading effect including path loss and shadow, where the path loss calculation formula is 128.1+37.6log 10 (dis(x,V i ), the shadow follows a log-normal distribution; g i It is the small-scale fading effect.

[0047] Furthermore, the method for the local RSU to select a vehicle according to the vehicle's residence time within its coverage area is as follows:

[0048] Calculate V for any vehicle i The duration of stay within the coverage area of ​​the local RSU

[0049]

[0050] Where, L s is the coverage of the local RSU, P i For user vehicle V i The distance that the current round has traveled within the local RSU coverage area, S i For user vehicle V i speed;

[0051] If the vehicle V i The duration of stay within the coverage area of ​​the local RSU Greater than the average training time T T and the inference time T I The sum of vehicle V i Participate in asynchronous federated training as a selected vehicle.

[0052] Furthermore, the vehicle weight used to update the global cache update model is calculated as follows:

[0053]

[0054] In the formula, χ i For user vehicle V i weight; μ1 and μ2 are the coefficients of position weight and transmission weight respectively, μ1+μ2=1; L s is the coverage of the local RSU, P i For user vehicle V i The distance that the current round has traveled within the local RSU coverage area, R R，i Represents the local RSU and the user vehicle V i The transmission rate between R，i′ Represents the local RSU and any vehicle V within its coverage area i′ The transmission rate between them, V′ is the number set of selected vehicles within the coverage of the local RSU.

[0055] An intelligent cooperative caching system based on asynchronous federated learning and social perception in an Internet of Vehicles includes a macro base station, several RSUs and several user vehicles. There are several user vehicles within the coverage of each RSU. The user vehicles send a cache request to the RSU within the coverage, and the RSU that receives the cache request is a local RSU, and the remaining RSUs are nearby RSUs. The intelligent cooperative caching system based on asynchronous federated learning and social perception in an Internet of Vehicles is used to implement any of the above-mentioned intelligent cooperative caching strategies based on asynchronous federated learning and social perception in an Internet of Vehicles.

[0056] Compared with the prior art, the significant advantages of the present invention include:

[0057] 1. By considering the mobility characteristics such as vehicle position and speed, an asynchronous federated learning algorithm is proposed to improve the accuracy of the global model.

[0058] 2. Use the DQN reinforcement learning algorithm to learn the request content data of vehicle users in each edge device. DQN can make the best caching decisions, reduce the average request latency of the vehicle and improve the caching performance of each edge device.

[0059] 3. Use the social network of vehicle users to obtain vehicle contact rates in different areas, and dynamically perceive the popularity of content based on changes in content requests. At the same time, use opportunistic V2V communication and data sharing formed by the concept of vehicle social network, integrate multiple vehicle communication methods, and significantly improve cache utilization.

[0060] 4. The collaborative caching strategy (AFDQN) based on asynchronous federation and deep reinforcement learning proposed in this invention makes full use of idle road resources, greatly improves edge caching performance, and reduces the average vehicle request latency. Experimental results show that AFDQN has better cache hit rate and average vehicle request latency than other baseline caching schemes in a highly dynamic vehicle network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is the technical process of the embodiment of the present application;

[0062] Figure 2 is the network architecture of the embodiment of the present application;

[0063] Figure 3 It is the DQN algorithm under asynchronous federation in the embodiment of the present application;

[0064] Figure 4 is the cache hit rate under different local RSU cache capacities in the embodiment of the present application;

[0065] Figure 5 is the request delay under different local RSU cache capacities in the embodiment of the present application;

[0066] Figure 6 is the cache hit rate under different total amounts of content in the embodiments of the present application;

[0067] Figure 7 is the request delay under different content amounts in the embodiment of the present application;

[0068] Figure 8 is the average training time per round of asynchronous federated learning and federation in the embodiment of the present application; DETAILED DESCRIPTION

[0069] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0070] This embodiment provides an intelligent cooperative caching strategy based on asynchronous federated learning and social perception in a vehicle network. The network architecture consists of a macro base station (MBS), several road side units (RSUs) and several user vehicles, and the MBS is connected to the cloud center. In the network architecture, there are several user vehicles within the coverage of each RSU. The user vehicle sends a cache request to the RSU in its coverage area, and the RSU that receives the cache request is the local RSU, and the remaining RSUs are nearby RSUs. The vehicle stores a large amount of historical data of vehicle users, including personal information, request content and ratings. RSUs communicate with each other through wired links, and edge servers for computing and caching are deployed, so RSUs can cache various content to meet the content service needs of vehicle users. The vehicle will look for nearby vehicles or the nearest RSU that have cached the requested content, and establish communication to obtain the content. Otherwise, the nearest RSU of the vehicle will download the content from nearby RSUs that have cached the content or directly from the MBS to provide services for the vehicle.

[0071] The intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles of this embodiment is shown in Figure 2. Figure 1 As shown, the following process is included:

[0072] S1. The user vehicle sends a cache request to the local RSU to obtain the corresponding request content.

[0073] S2. After receiving the cache request, the local RSU determines whether there is request content corresponding to the cache request in its own cache content: if so, the request content is sent to the user vehicle; if not, a nearby user vehicle, neighboring RSU or macro base station with the request content is selected according to the preset priority to respond to the cache request of the user vehicle, and the request content is also sent to the local RSU.

[0074] Local RSU: If the cache content of the local RSU already contains the request content corresponding to the cache request, the local RSU sends the request content back to the vehicle.

[0075] Nearby vehicles: Cached vehicles can share content with other vehicles. Due to the limited coverage of wireless signals, the premise of vehicle-to-vehicle communication is that the distance between vehicles is less than a certain value. The event in which two vehicles are very close and establish a communication relationship with each other is called vehicle contact. These areas may have different vehicle densities and driving modes, resulting in different vehicle social attributes. Therefore, the vehicle contact rate can be used as a key indicator to characterize vehicle social relationships. The vehicle contact time also depends mainly on the contact rate α v , the contact time follows an exponential distribution. In the service area of ​​RSU n, the contact probability between vehicles is: Among them, social-aware caching uses the concepts of social networks such as node centrality and social communities to ensure that the selected collaborative vehicles achieve the required data transmission ratio within a given time. If the requested content is not cached in the local RSU, the vehicle directly sends the request to the nearby vehicles. If the nearby vehicles cache the requested content, the requested content is sent to the vehicle.

[0076] Neighboring RSU: If the requested content is not cached in the local RSU or in the nearby vehicle, the local RSU forwards the request to the neighboring RSU. If the neighboring RSU caches the requested content, the requested content is sent to the local RSU. After that, the local RSU sends the content back to the vehicle.

[0077] MBS: If the requested content is neither cached in the local RSU nor in nearby vehicles and adjacent RSUs, the vehicle sends a cache request to the MBS, and the MBS directly sends the requested content back to the vehicle.

[0078] S3. Each user vehicle within the coverage of the local RSU: After receiving the request content corresponding to each cache request, the local cache update model of the vehicle is used to update the local cache content of the vehicle, and the local cache update model of the vehicle is updated and trained based on the current experience pool data of the vehicle.

[0079] The local cache update model of the vehicle and the global cache update model of the RSU in the present invention are used to: give the current optimal decision based on the cache status and request status of the current cache content of the vehicle itself: whether to use the request content to update its own cache content so that the cache content can respond to subsequent cache requests more quickly.

[0080] The local cache update model of the vehicle is abbreviated as the local model, and the global cache update model of the RSU is abbreviated as the global model. The initial local model of each vehicle is obtained by downloading the global model of the local RSU; the global model and the local model use a deep reinforcement learning algorithm to decide whether to update their respective cache contents.

[0081] (1) Global model.

[0082] The global model uses a deep reinforcement learning algorithm to decide whether to update the cache content, which is defined as:

[0083] Status: includes cache status and request status of preset multiple cache contents, expressed as in, represents the state of the global model of the nth RSU in time slot t, It represents the cache status of K preset cache contents corresponding to the global model of the nth RSU in the t time slot, represents the cache status of the global model of the nth RSU for the kth preset cache content in the tth time slot, Represents cached. Represents not cached; represents the request status of the i-th user vehicle for K preset cache contents in time slot t, represents the request status of the i-th vehicle for the k-th preset cache content in the t-th time slot, Send a request on behalf of Indicates that the request was not sent; P k represents the cache location of the kth preset cache content; V is the set of user vehicle numbers within the coverage of the nth RSU.

[0084] Action: Preset whether multiple cache contents are replaced with the request contents corresponding to the cache request, expressed as

[0085] in, represents the action of the global model of the nth RSU in time slot t, Indicates whether the global model of the nth RSU replaces the kth preset cache content with the request content corresponding to the cache request currently received from the i-th user vehicle in the tth time slot, Indicates replacement, Indicates no replacement.

[0086] The reward is: Minimize the cache request delay of each user's vehicle. It is expressed as:

[0087]

[0088] In the formula, λ1+λ2+λ3+λ4=1, 0<λ1<λ2<<λ3<λ4; D t represents the delayed reward at time t; λ1, λ2, λ3, and λ4 represent the learning rates in the reward functions of nearby vehicles, local RSU, nearby RSU, and macro base station response cache requests, respectively; d V，i,d R，i , d M，i They represent the transmission delays of nearby vehicles, local RSU, nearby RSU, and macro base station in response to the cache request of the i-th vehicle.

[0089] In this embodiment, the vehicle uses orthogonal frequency division multiplexing technology to communicate with the RSU and cache vehicles, so interference is not considered in the communication model. The communication between the local RSU and the adjacent RSU uses a wired link. Each vehicle maintains the same communication model in one round and changes its communication model in different asynchronous rounds. Among them, the transmission delay of the nearby vehicle, the local RSU, the nearby RSU, and the macro base station in response to the i-th vehicle cache request is calculated as follows:

[0090] ① The transmission delay d of the nearby vehicles responding to the cache request of the i-th vehicle V，i The calculation formula is:

[0091]

[0092]

[0093] Where s represents the size of the request content in response to the cache request; R V，i Represents the neighboring vehicles V and the i-th vehicle V i The transmission rate between them; α represents the contact rate between nearby vehicles V, that is, the contact rate between nearby vehicles V and vehicle V i Probability of establishing communication; N represents the total number of vehicles with requested content cached in time slot t within the coverage area of ​​the local RSU. t represents the total number of vehicles in the coverage area of ​​the local RSU in time slot t; B is the available bandwidth; p V Indicates the transmit power level used by nearby vehicles, dis(V, V i ) represents the neighboring vehicle V and vehicle V i The distance between i (dis(V,V i )) indicates vehicle V i The channel gain with the nearby vehicle V, is the noise power.

[0094] The vehicle model in this embodiment assumes that all vehicles travel in the same direction and arrive at one RSU based on Poisson distribution, with an average arrival rate of λ v. Once a vehicle enters the coverage area of ​​the local RSU, it will send a request message to the local RSU. We assume that no vehicle enters or leaves the coverage area of ​​the local RSU in a round, so the number of vehicles in the local RSU in a round is fixed. Each vehicle maintains the same mobility characteristics, including position and speed, within a round, and may change its mobility characteristics at the beginning of each round. The speeds of different vehicles follow independent and identical distributions. The speed of each vehicle is generated by a truncated Gaussian distribution, which is flexible and conforms to the real dynamic vehicle environment. For any r-th round, the number of vehicles traveling in the coverage area of ​​the local RSU is N, and the vehicle set is recorded as Where V i Let {S1, S2, ..., S i , ..., S N} is the speed of all vehicles traveling in the local RSU, where S i V i speed. i The probability density function of is calculated as:

[0095]

[0096] S max and S min are the maximum and minimum speed thresholds for each vehicle, For S i With mean μ and variance σ 2 The Gaussian error function under For user vehicle V i The distance traveled within the local RSU coverage area, T(r) is the duration of the rth round, then V i The distance traversed can be calculated as:

[0097] ② The transmission delay d of the local RSU responding to the cache request of the i-th vehicle R，i The calculation formula is:

[0098]

[0099]

[0100] In the formula, R R，i Represents the local RSU and the user vehicle V i The transmission rate between B Indicates the transmit power level used by the local RSU; dis(S, V i ) represents the local RSU and vehicle V i The distance between i (dis(S,V i)) indicates the local RSU and vehicle V i The channel gain between

[0101] ③ Transmission delay of the nearby RSU in response to the cache request of the i-th vehicle The calculation formula is:

[0102]

[0103] In the formula, R R-R Indicates the transmission rate between adjacent RSUs;

[0104] ④ Transmission delay d of the macro base station responding to the cache request of the i-th vehicle M，i The calculation formula is:

[0105]

[0106]

[0107] In the formula, R B，i Represents the macro base station and the user vehicle V i The transmission rate between M Indicates the transmit power level used by the macro base station; dis(M, V i ) represents the macro base station and the vehicle V i The distance between i (dis(M,V i )) represents the macro base station and the vehicle V i The channel gain between .

[0108] Assume x = V, M, S represent nearby vehicles, macro base stations, and local RSUs, respectively. Then the nearby vehicles, local RSUs, macro base stations, and vehicles V i The channel gain calculation formula in the current round is:

[0109] h i (dis(x,V i ))=α i (dis(x,V i ))g i

[0110] In the formula, α i (dis(x,V i )) is the large-scale fading effect including path loss and shadow, where the path loss calculation formula is 128.1+37.6log 10 (dis(x,V i ), the shadow follows a log-normal distribution; g i is the small-scale fading effect, which is assumed to be exponentially distributed with unit mean.

[0111] In addition, the global model needs to be pre-trained using a training set initially so that the local model can be downloaded as an initial local model.

[0112] The global model uses a deep reinforcement learning algorithm and is pre-trained using the training set:

[0113] Estimate the action-value function Q(S) by updating the Q-learning iteration t , A t ), as follows:

[0114]

[0115] Where γ≥0 represents the learning rate. The current RSU takes system action A t And get instant reward R(S t , A t ) and then the current state S t Update to the next state S t+1 Then, the optimal control strategy for each RSU is obtained according to the c-∈-greedy strategy, which is represented by A*:

[0116]

[0117] Where ∈ represents the probability of selecting a random action. This greedy strategy can effectively avoid saddle points by trying random new actions. The size of each RSU cache is f m , store historical data through the experience pool M and update it according to the latest experience. t Through Action A t Transform into action t+1 And get a reward R(S t , A t ), we can get a batch training set (S t , A t ,R(S t , A t ), S t+1 ), and store the training set in the experience playback M. During the entire system training process, at each moment, RSU randomly selects a small batch of experience pools from the experience pool M And by minimizing Q(S t , A t ;ω t ) to train QNN. Then, the prediction loss function of QNN is defined as follows:

[0118] Loss(ω t )=E[(q(S t+1 , A t+1 )-Q(S t , At ;ω t )) 2 ]

[0119] In the formula, ω t represents the weight parameter of each iteration of QNN training, Q(S t , A t ;ω t ) is the output of QNN at time slot t, and its approximate value function is defined as follows:

[0120]

[0121] in, is the QNN in a given state S t+1 The maximum output obtained when selecting an action is A t+1 . Then, the update is performed using the semi-gradient algorithm as follows:

[0122]

[0123] Where ρ>0 is the gradient step size, Yes t The updated gradient.

[0124] (2) Local model.

[0125] The initial local model of each vehicle is obtained by downloading the global model of the local RSU. That is, the local model of the i-th user vehicle also uses the deep reinforcement learning algorithm to decide whether to update the cache content, which is defined as:

[0126] Status: includes cache status and request status of preset multiple cache contents, expressed as in, represents the state of the current local model at time slot t, represents the cache status of the current local model for K preset cache contents in time slot t, represents the cache status of the current local model for the kth preset cache content in time slot t, Represents cached. Represents not cached; represents the request status of the i-th user vehicle for K preset cache contents in time slot t, represents the request status of the i-th vehicle for the k-th preset cache content in the t-th time slot, Send a request on behalf of Indicates that the request was not sent; P k represents the cache location of the kth preset cache content; V is the set of user vehicle numbers within the coverage area of ​​the nth RSU;

[0127] The action is: whether to replace multiple cache contents with the request contents corresponding to the cache request, expressed as in, represents the action of the current local model in time slot t, Indicates whether the current local model replaces the k-th preset cache content with the requested content obtained by the cache request in the t time slot. Indicates replacement, Indicates no replacement;

[0128] The reward is: Minimize the cache request delay of the i-th user vehicle. The cache request delay of the user vehicle It is consistent with the global model and is expressed as:

[0129]

[0130] After obtaining the initial local model of each vehicle by downloading the global model of the local RSU, the local model is retrained on the local data set.

[0131] S4. After the local cache update model of each selected vehicle is updated, it is uploaded to the local RSU; each selected vehicle is selected by its local RSU according to the residence time of the vehicle within its coverage area.

[0132] The present invention builds, trains, and updates global and local models based on an asynchronous federation framework. For each round of asynchronous federation, after a vehicle uploads a local model, the local RSU aggregates and updates its global model. This embodiment considers the mobility of vehicles and proposes an asynchronous federation algorithm that can be applied to the Internet of Vehicles.

[0133] (1) Vehicle selection: Select vehicles that stay long enough at the local RSU to ensure that they have the opportunity to participate in the asynchronous federation and complete the training process. Each vehicle first sends its mobility characteristics to the local RSU, including speed and position. The local RSU then selects a vehicle based on the residence time calculated by the vehicle's mobility characteristics.

[0134] User Vehicle V i The dwell time within the local RSU coverage area is calculated as:

[0135]

[0136] Among them, L s is the coverage of the local RSU.

[0137] V i The residence time should be greater than the average training time TT and inference time T I The sum of the user's vehicle V ican complete a round of training. Therefore, if User Vehicle V i Otherwise, the user vehicle V i is ignored.

[0138] (2) Download model: Send the parameters trained by RSU as initial values ​​to the selected vehicle.

[0139] (3) Local training: The selected vehicle V in the local RSU i The downloaded global model ω is set as the initial local model, and the local model is updated through training iterations. Afterwards, the updated local model is uploaded to the local RSU. For each iteration k, V i Randomly extract some training data n from the original training set k The loss function under the local model is calculated as:

[0140]

[0141] Calculate n k After the loss function of all data in is calculated, the local loss function after k iterations is calculated as:

[0142]

[0143] Among them, |N k | for n k The number of data in .

[0144] Then calculate the regularized local loss function to reduce the local model ω i，k The deviation from the global model ω improves the convergence of the algorithm:

[0145]

[0146] where ρ is the regularization parameter.

[0147] set up is g(ω i,k ) is called the local gradient. In the previous round, some vehicles may fail to upload the updated local model due to the delay in training time, thus affecting the convergence of the global model. Here, these vehicles are called stragglers, and the local gradient of the stragglers in the previous round is called the delayed local gradient. Aggregate the delayed local gradient into the local gradient of the current round:

[0148]

[0149] Where β is the attenuation coefficient, is the delayed local gradient. If Vi is uploaded successfully in the previous round, then

[0150] Then the local model for the next iteration is updated as:

[0151]

[0152] Where η is the local learning rate of the current round, calculated as: : η = ηmax{1, log(r)}. l is the initial value of the local learning rate. r is the current round.

[0153] Then iteration k ends, V i Randomly sample some training data again and start the next iteration. When the number of iterations reaches the threshold e, V i Complete local training and update the local model ω i ,Right now Upload to the local RSU. Further get the global objective function:

[0154]

[0155] where d i V i The size of the local data in , d is the total size of the local data of the selected vehicle. The overall goal of asynchronous federated learning is to find a global model ω* to minimize the objective function, that is:

[0156] ω * =MinG(ω)

[0157] (4) Updating and uploading the model: After completing the current round of training of the local model, the vehicle uploads the updated local model to the local RSU.

[0158] S5. When the local RSU receives any local cache update model, it updates the global cache update model based on the corresponding vehicle weight.

[0159] Considering the mobility of vehicles, an asynchronous aggregation of the global model is designed based on mobility awareness:

[0160]

[0161] Among them, χ i Yes V i The weight of the asynchronous aggregation.

[0162] By considering the user vehicles V within the coverage area of ​​the local RSU i The traversing distance and the distance from the local RSU to V i The content transmission delay is calculated by asynchronous aggregation V j , in order to improve the accuracy of the global model and reduce content transmission delay. Specifically, if V jIf the traversal distance of is large, it may take a long time to participate in training, so its local model should occupy a larger weight for aggregation to improve the accuracy of the global model. j The content transmission delay is important because when the content is cached in the local or neighboring RSU, V i Eventually, the content will be downloaded from the local RSU. Therefore, if the local RSU to V i The content transmission delay of is small, and its local model should also occupy a larger weight for aggregation to reduce the content transmission delay. The weight of asynchronous aggregation χ i Calculated as:

[0163]

[0164] Where μ1 and μ2 are the coefficients of location weight and transmission weight respectively (i.e., μ1+μ2=1). The values ​​of μ1 and μ2 can be set according to the location and transmission in the actual application. s is the size of the request content. Therefore, from the local RSU to V i The content transmission delay is affected by the local RSU and V i The impact of the transmission rate between and R R，i Further calculation of χ i ,Right now:

[0165]

[0166] Since the local RSU knows each vehicle V at the beginning of the asynchronous federation i dis(S,V i ) and P i , so the local RSU can calculate R R，i , further calculate χ i So far, the rth round of asynchronous federated learning has been completed, and the updated global model ω is obtained. r .

[0167] S6. The local RSU uses the current global cache update model to update the cache content of the local RSU based on the request content corresponding to the cache requests of all user vehicles in its service area.

[0168] To verify the technical effect of the present invention, the following experimental hardware environment is Windows 10 operating system, CPU is Intel Core i7-8750k (2.20GHz), memory is 8GB, and it is developed and verified on the PyCharm platform. Multiple roadside units RSU are distributed on a road, and the local roadside unit, adjacent roadside unit, macro base station and vehicle are cached in coordination. The overall architecture in the experiment of this embodiment is as follows Figure 2As shown, the red car is a mobile vehicle. The vehicle moves in the city, the local roadside unit communicates with the vehicles within its coverage area, the neighboring roadside unit communicates with the local roadside unit, the macro base station communicates with the vehicle, and the vehicles communicate with each other.

[0169] Figure 3 This is the asynchronous federation DQN algorithm training adopted by the embodiment of the present invention, where red represents the vehicles selected to participate in the training and gray represents the vehicles not selected. The local roadside unit performs DQN training locally to obtain the initial global parameters, and the selected vehicles download the global parameters for local training. Considering the mobility of the vehicles, the first trained vehicles upload the model first, and finally aggregate according to the asynchronous aggregation weight of each vehicle, and finally upload the global model.

[0170] The cache hit rate under different local RSU cache capacities is as follows: Figure 4 As shown. The selection range of the RSU cache capacity is controlled between 50 and 400, and the cache hits of the present invention and the other three baseline schemes. It can be seen that as the capacity increases, the cache hit rate of all schemes increases. The cache hit rate of the present invention is improved by an average of about 45%, 5%, 11% and 2% respectively compared with other strategies. This is because the local RSU caches more content with a larger capacity, and under the limited cache capacity of the RSU, the RSU will naturally generate more cache updates. Therefore, it is more likely to obtain the requested content of the vehicle from the local RSU. In addition, the random scheme provides the worst cache hit rate because the scheme only randomly selects content. In addition, the performance of the present invention, c-∈-greedy and CAFR is better than that of the random scheme and Thompson sampling. This is because the random and Thompson sampling schemes cannot predict the cache content through learning, while the present invention, c-∈-greedy and CAFR schemes determine the cache content by observing the historical request content. In addition, the present invention is slightly better than c-∈-greedy and CAFR.

[0171] Request latency under different local RSU cache capacities is as follows: Figure 5 As shown. As the cache capacity increases, the content transmission delay of each scheme is decreasing. The request delay of the present invention is reduced by an average of about 15%, 7%, and 11% compared to the random, c-∈-greedy, and Thompson sampling schemes, respectively. This is because as the cache capacity increases, each RSU will cache more content, and each vehicle is more likely to obtain content from the local RSU, thereby reducing the delay in content transmission. The content transmission delay of the present invention is always in a better position. This is because the cache hit rate of the present invention is higher than other schemes, and more vehicles can obtain content directly from the local RSU, thereby reducing the delay in content transmission.

[0172] The cache hit rate under different content amounts is as follows Figure 6 As shown in the figure. By changing the number of contents (from 200 to 900), as the number of contents increases, the cache hit rates of the five algorithms decrease under the limited RSU cache capacity. However, the present invention can better observe the historical requests and rewards of cached contents, so its cache hit rate decreases less than other algorithms.

[0173] Request latency under different content amounts is as follows Figure 7 As shown in the figure, the content transmission delay of different schemes under different total content amounts. As the amount of content increases, the transmission delay of the five algorithms increases slightly. Because the more content there is, the cache hit rate decreases under the limited cache capacity of RSU. However, the increase in the request delay of the present invention is smaller than that of other algorithms, and it is more stable overall.

[0174] The average training time per round for asynchronous federation and federation is as follows Figure 8 As shown in the figure, the training time of each round of the present invention is within 2.5s and 3s, while the training time of each round of the FedAVG scheme is within 31s and 33s. This shows that the training time of the present invention is much shorter than that of the FedAVG scheme. This is because the FedAVG scheme needs to aggregate the local models of all vehicles for each round of global model update, while the asynchronous federation aggregates the local models of the vehicles when receiving them in each round.

[0175] The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.

Claims

1. An intelligent cooperative caching strategy based on asynchronous federated learning and social perception in Internet of Vehicles, characterized in that: include: The user vehicle sends a cache request to the local RSU; After receiving the cache request, the local RSU determines whether there is a request content corresponding to the cache request in its own cache content: if yes, the request content is sent to the user vehicle; if not, a nearby user vehicle, a neighboring RSU or a macro base station with the request content is selected according to a preset priority to respond to the cache request of the user vehicle, and the request content is also sent to the local RSU; Each user vehicle within the coverage of the local RSU: after receiving the request content corresponding to each cache request, the local cache update model of the vehicle is used to update the local cache content of the vehicle, and the local cache update model of the vehicle is updated and trained based on the current experience pool data of the vehicle; After the local cache update model of each selected vehicle is updated, it is uploaded to the local RSU; each selected vehicle is selected by its local RSU according to the residence time of the vehicle in its coverage area; When the local RSU receives any local cache update model, it updates the global cache update model based on the corresponding vehicle weight; The local RSU uses the current global cache update model to update the cache content of the local RSU based on the request content corresponding to the cache requests of all user vehicles in its service area.

2. The intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles according to claim 1 is characterized in that: The preset priorities, from high to low, are: nearby user vehicles, adjacent RSUs and macro base stations, and the methods for sending request contents are: The nearby user vehicles use social perception between vehicles to send the request content to the user vehicle; The neighboring RSU first sends the request content to the local RSU, which then sends it to the user vehicle; The macro base station receives the cache request from the user vehicle and then directly sends the request content to the user vehicle.

3. The intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles according to claim 1 is characterized in that: The local cache update model of the vehicle is abbreviated as the local model, and the global cache update model of the RSU is abbreviated as the global model; the initial local model of each vehicle is obtained by downloading the global model of the local RSU; the global model and the local model use a deep reinforcement learning algorithm to decide whether to update their respective cache contents; The global model uses a deep reinforcement learning algorithm to decide whether to update the cache content as follows: Status: includes cache status and request status of preset multiple cache contents, expressed as in, represents the state of the global model of the nth RSU in time slot t, It represents the cache status of K preset cache contents corresponding to the global model of the nth RSU in the t time slot, represents the cache status of the global model of the nth RSU for the kth preset cache content in the tth time slot, Represents cached. Represents not cached; represents the request status of the i-th user vehicle for K preset cache contents in time slot t, represents the request status of the i-th vehicle for the k-th preset cache content in the t-th time slot, Send a request on behalf of Indicates that the request was not sent; P k represents the cache location of the kth preset cache content; V is the set of user vehicle numbers within the coverage area of ​​the nth RSU; The action is: whether to replace multiple cache contents with the request contents corresponding to the cache request, expressed as in, represents the action of the global model of the nth RSU in time slot t, Indicates whether the global model of the nth RSU replaces the kth preset cache content with the request content corresponding to the cache request currently received from the i-th user vehicle in the tth time slot, Indicates replacement, Indicates no replacement; The rewards are: minimize the cache request latency of each user's vehicle; The local model of the i-th user vehicle uses a deep reinforcement learning algorithm to decide whether to update the cache content as follows: Status: includes cache status and request status of preset multiple cache contents, expressed as in, represents the state of the current local model at time slot t, represents the cache status of the current local model for K preset cache contents in time slot t, represents the cache status of the current local model for the kth preset cache content in time slot t, Represents cached. Represents not cached; represents the request status of the i-th user vehicle for K preset cache contents in time slot t, represents the request status of the i-th vehicle for the k-th preset cache content in the t-th time slot, Send a request on behalf of Indicates that the request was not sent; P k represents the cache location of the kth preset cache content; V is the set of user vehicle numbers within the coverage area of ​​the nth RSU; The action is: whether to replace multiple cache contents with the request contents corresponding to the cache request, expressed as in, represents the action of the current local model in time slot t, Indicates whether the current local model replaces the k-th preset cache content with the requested content obtained by the cache request in the t time slot. Indicates replacement, Indicates no replacement; The reward is: minimize the cache request latency of the i-th user's vehicle.

4. The intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles according to claim 3 is characterized in that: The reward function of the global model and each local model is: In the reward function of the global model, (S t ,A t )Pick In the reward function of the local model, (S t ,A t )Pick D t represents the delayed reward at time t; λ1, λ2, λ3, and λ4 represent the learning rates in the reward function of nearby vehicles, local RSU, nearby RSU, and macro base station response cache requests, respectively; d V,i ,d R,i , d M,i They represent the transmission delays of nearby vehicles, local RSU, nearby RSU, and macro base station in response to the cache request of the i-th vehicle.

5. The intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles according to claim 1 is characterized in that: The transmission delay d of nearby vehicles responding to the cache request of the i-th vehicle V,i The calculation formula is: Where s represents the size of the request content in response to the cache request; R V,i Represents the neighboring vehicles V and the i-th vehicle V i The transmission rate between them; α represents the contact rate between nearby vehicles V, that is, the contact rate between nearby vehicles V and vehicle V i Probability of establishing communication; N represents the total number of vehicles with requested content cached in time slot t within the coverage area of ​​the local RSU. t represents the total number of vehicles in the coverage area of ​​the local RSU in time slot t; B is the available bandwidth; p V Indicates the transmit power level used by nearby vehicles, dis(V,V i ) represents the neighboring vehicle V and vehicle V i The distance between i (dis(V,V i )) indicates vehicle V i The channel gain with the nearby vehicle V, is the noise power; The transmission delay d of the local RSU in responding to the cache request of the i-th vehicle R,i The calculation formula is: In the formula, R R,i Represents the local RSU and the user vehicle V i The transmission rate between B Indicates the transmit power level used by the local RSU; dis(S,V i ) represents the local RSU and vehicle V i The distance between i (dis(S,V i )) indicates the local RSU and vehicle V i The channel gain between Transmission delay of the nearby RSU in response to the cache request of the i-th vehicle The calculation formula is: In the formula, R R-R Indicates the transmission rate between adjacent RSUs; The transmission delay d of the macro base station in responding to the cache request of the i-th vehicle M,i The calculation formula is: In the formula, R B，i Represents the macro base station and the user vehicle V i The transmission rate between M Indicates the transmit power level used by the macro base station; dis(M, V i ) represents the macro base station and the vehicle V i The distance between o (dis(M,V i )) represents the macro base station and the vehicle V i The channel gain between .

6. The intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles according to claim 5 is characterized in that: Assume x = V, M, S represent nearby vehicles, macro base stations, and local RSUs, respectively. Then the nearby vehicles, local RSUs, macro base stations, and vehicles V i The channel gain calculation formula between is: h i (dis(x,V i ))=α i (dis(x,V i ))g i In the formula, α i (dis(x,V i )) is the large-scale fading effect including path loss and shadow, where the path loss calculation formula is 128.1+37.6log 10 (dis(x,V i ), the shadow follows a log-normal distribution; g i It is the small-scale fading effect.

7. The intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles according to claim 1 is characterized in that: The method by which the local RSU selects a vehicle based on the vehicle's residence time within its coverage area is: Calculate V for any vehicle i The duration of stay within the coverage area of ​​the local RSU Where, L s is the coverage of the local RSU, P i For user vehicle V i The distance that the current round has traveled within the local RSU coverage area, S i For user vehicle V i speed; If the vehicle V i The duration of stay within the coverage area of ​​the local RSU Greater than the average training time T T and the inference time T I The sum of vehicle V i Participate in asynchronous federated training as a selected vehicle.

8. The intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles according to claim 1 is characterized in that: The vehicle weight used to update the global cache update model is calculated as: In the formula, χ i For user vehicle V i weight; μ1 and μ2 are the coefficients of position weight and transmission weight respectively, μ1+μ2=1; L s is the coverage of the local RSU, P i For user vehicle V i The distance that the current round has traveled within the local RSU coverage area, R R，i Represents the local RSU and the user vehicle V i The transmission rate between R，i′ Represents the local RSU and any vehicle V within its coverage area i′ The transmission rate between them, V′ is the number set of selected vehicles within the coverage of the local RSU.

9. An intelligent cooperative caching system based on asynchronous federated learning and social perception in an Internet of Vehicles, characterized in that: It includes a macro base station, several RSUs and several user vehicles. There are several user vehicles within the coverage of each RSU. The user vehicle sends a cache request to the RSU in the coverage area, and the RSU that receives the cache request is the local RSU, and the remaining RSUs are nearby RSUs; the intelligent cooperative caching system based on asynchronous federated learning and social perception in the Internet of Vehicles is used to implement the intelligent cooperative caching strategy based on asynchronous federated learning and social perception in the Internet of Vehicles as described in any one of claims 1-8.

Images