D2D auxiliary edge caching algorithm based on deep reinforcement learning

Abstract

The invention discloses a D2D auxiliary edge caching algorithm based on deep reinforcement learning, relates to the technical field of edge caching and deep reinforcement learning, and optimizes the performance of edge caching by introducing a user preference learning mechanism. The content preference of the user is accurately predicted by analyzing multi-dimensional data such as a content request mode, a social relation and a time sequence behavior of the user; based on the prediction results, a caching strategy is dynamically adjusted, so that the hit rate of content can be maximized and the average delay of content transmission can be minimized under limited caching resources; modeling a cache problem into a Markov decision process, and solving the problem by using a near-end strategy optimization algorithm in deep reinforcement learning; through optimization of the PPO algorithm, the method provided by the embodiment of the invention can effectively balance the popularity of the cached content and the personalized demand of the user, so that a higher hit rate and a lower transmission delay are realized in different user request scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of edge caching and deep reinforcement learning, and in particular to a D2D-assisted edge caching algorithm based on deep reinforcement learning. Background Art

[0002] In modern mobile communication networks, with the emergence of emerging application scenarios such as the Internet of Things (IoT) and social networks, user data traffic has shown explosive growth. This surge in data volume has put tremendous pressure on the already congested cellular networks and backbone networks, resulting in higher transmission delays, which seriously affects the user experience. In order to solve this problem, edge caching technology came into being and became a key means to effectively alleviate data traffic pressure and reduce transmission delays.

[0003] The core idea of ​​edge caching is to deploy cache nodes at the edge of the network. When a user requests data, the system first checks these edge cache nodes. If the data is already stored on a nearby cache node, it can be provided directly to the user, avoiding the delay caused by long-distance transmission. This approach not only improves the user experience, but also reduces the burden on centralized servers and optimizes overall network performance. However, traditional edge caching strategies still have certain shortcomings when facing problems such as diversity of user preferences and time volatility.

[0004] To this end, the researchers proposed an active edge caching strategy based on user preference learning, which fully considers the delay of content transmission and the dynamic changes of user preferences. Specifically, the strategy introduces device-to-device (D2D) communication technology to further reduce the delay of content transmission through direct communication between user devices. At the same time, combined with the user's content requests, content similarity and social relationships, a user preference prediction model based on deep learning is proposed.

[0005] In this model, the researchers used a combination of Stacked Autoencoder (SAE) and BiLSTM (BiLSTM) to learn users' temporal preferences. Stacked Autoencoder can effectively extract features of high-dimensional data, while BiLSTM can capture the law of user preferences changing over time. In addition, considering the impact of social relationships between users on content requests, the model also introduced Convolutional Block Attention Module (CBAM) to learn the weights of social relationships, thereby improving the accuracy of user preference prediction.

[0006] Based on the learned user preference content, the researchers further proposed an active caching strategy based on content popularity. In this strategy, the Hawkes process model is used to calculate the popularity of the content, and the Proximal Policy Optimization (PPO) algorithm is used for training to obtain the optimal caching decision. Experimental results show that this caching strategy based on content popularity performs well in both content download delay and cache hit rate.

[0007] To further improve the performance of edge caching, the study also considered the role of user equipment cache (UE cache) in traffic offloading. UE cache can effectively offload traffic to neighboring user devices through D2D communication, thereby alleviating the pressure on the network. However, a major challenge of UE cache is to determine which content should be cached. To solve this problem, the researchers used machine learning technology to predict the popularity of content through historical data and dynamically adjust the cache strategy based on the prediction results.

[0008] In general, the edge caching strategy based on D2D communication combines the advantages of deep learning and reinforcement learning, which not only effectively reduces the delay of content transmission, but also improves the cache hit rate, providing a new direction for future mobile network optimization. With the further development of IoT and social networks, the complexity and diversity of user preferences will put higher requirements on edge caching technology. Therefore, future research will need to further explore how to optimize caching strategies in more complex network environments to cope with changing user needs and network conditions.

[0009] In summary, with the continuous increase in the number of mobile terminal devices and application requirements, traditional cache strategies are difficult to cope with the growing amount of data and user personalized needs. Especially in edge computing scenarios, how to efficiently manage cache resources to reduce transmission delays and improve cache hit rates has become an urgent problem to be solved. Summary of the invention

[0010] In order to solve the above technical problems, the present invention provides a D2D assisted edge caching algorithm based on deep reinforcement learning, comprising the following steps:

[0011] S1, establish an active edge caching system for a D2D-assisted cellular network consisting of user devices;

[0012] S2, establish a time slot model, each time slot is generally divided into non-peak period and peak period;

[0013] S3, establishing a D2D model to model the D2D shared link between two user devices based on the physical domain and the social domain;

[0014] S4. Establish a content transmission delay model. The user equipment has three ways to cache content in the D2D-assisted cellular network: local cache, D2D cache, and cellular network cache.

[0015] S5. Establish a content popularity model and a content cache model, calculate popularity based on the Hawkes model, use the content hit rate as an indicator to measure the effectiveness of cache operations, and use the future hit rate to estimate the response of the active edge cache system to user requests in future time slots;

[0016] S6, predicting the user's future content requests by combining the correlation between different contents, the temporal model of user content requests, and the influence of social relationships, and learning and updating parameters by minimizing the loss between the predicted number of user content requests and the actual number of user content requests;

[0017] S7. Calculate the content popularity based on the learned user preference content, and then transform the active caching problem based on content popularity into a Markov decision process problem. By continuously interacting with the environment and optimizing and updating its own network, the optimal caching strategy is obtained.

[0018] The beneficial effects of the present invention are:

[0019] (1) The present invention can cope with complex and changeable communication scenarios. Most traditional algorithms assume that the channel conditions are known, while the present invention adopts a model-free deep reinforcement learning algorithm, which can effectively cope with the time-varying nature of information transmission and is more realistic.

[0020] (2) In the present invention, PPO is a currently popular reinforcement learning algorithm. The present invention improves the traditional PPO algorithm so that it can process mixed action spaces, including both continuous and discrete actions, thereby reducing data loss and improving system performance. The algorithm has the advantages of easy implementation, low sample complexity, and stable training, and is very suitable for complex and changeable communication scenarios.

[0021] (3) In the present invention, user preference learning and cache strategy optimization are carried out. By introducing a user preference learning method based on content and social perception, it is possible to more accurately predict user content needs, optimize cache strategies, reduce data transmission delays, and improve cache hit rates;

[0022] (4) In the present invention, efficient cache strategy training is carried out. The PPO algorithm is combined with the Actor-Critic method to achieve efficient cache strategy training. Through the synergy of the strategy network and the value network, a more stable strategy update is ensured;

[0023] (5) In the present invention, a high content hit rate and a user request response ratio are set. The PPO-based cache strategy significantly improves the hit rate of cached content and the ratio of the system successfully responding to user requests by optimizing the content popularity matrix. Compared with the traditional LRU and LFU strategies, the overall system performance is better.

[0024] (6) In the present invention, by dynamically adjusting parameters, the system can better adapt to changes in user preferences, thereby maintaining high performance in different environments and having higher flexibility and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0026] Figure 2 Schematic diagram of an active edge cache system of a D2D assisted cellular network in an embodiment of the present invention;

[0027] Figure 3 It is a sequence diagram of the execution of the model in each time slot in an embodiment of the present invention;

[0028] Figure 4 A diagram of a user preference learning framework based on content and social perception in an embodiment of the present invention;

[0029] Figure 5 This is a flow chart of an active caching strategy optimization algorithm based on deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] This embodiment provides a D2D assisted edge caching algorithm based on deep reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0031] S1. Establish an active edge caching system of a D2D-assisted cellular network consisting of user devices; Figure 2 As shown in FIG. 1 , the active edge caching system includes a base station (BS) and N randomly distributed user equipments (UEs). Represented as a collection of user devices, Indicates that the user device is from the content library The requested content in the above example is , and F represents the number of contents. Assuming that the cache capacity of the base station is large enough to store the entire content library Each user device can Request arbitrary content in, where the length of content f is represented as lf.

[0032] In order to avoid traffic congestion caused by repeated content download requests, the user equipment can also act as a cache node and provide cache services, and share cache content with other user equipment through D2D communication; the cache capacity of user equipment n is represented by S n , the cache status is recorded as When content f is cached in user device n at time slot t, otherwise

[0033] S2. Establish a time slot model, such as Figure 3 As shown in the figure, each time slot is generally divided into two parts, namely non-peak hours and peak hours. At the beginning of each time slot, the deep learning (DL) model is trained offline for the data of historical content requests, and then the user device preferences for different contents are predicted by the trained DL model. Based on the prediction of user device preferences, the corresponding cache decision is obtained and the cache content on all user devices is updated. Then, during the peak period, content is delivered according to the needs of user devices. represents the cache content request vector of user device n, where represents the number of requests for content f and

[0034] Considering the continuous emergence of new content, if content f is not available within time slot t, then A resource scheduler is set up to dynamically allocate subchannels to user devices to avoid interference. Since the active edge caching system suffers from both large-scale and small-scale fading, the transmission rate of content f from user device m and base station to user device n is expressed as:

[0035]

[0036] In the formula, B D2D and B BS They represent the bandwidth of D2D link transmission and cellular link channel, P UE and P BS Respectively represent the transmit power of the user equipment and the base station, h m,n and h s，n Denote the Rayleigh fast fading coefficients of the communication link between user equipment m and user equipment n and between the base station and user equipment n, d m,n and d s，n They represent the physical distances from user equipment n to user equipment m and the base station, α is the path loss index, σ 2 Represents the noise power.

[0037] Therefore, the delay of user equipment n downloading content f from the base station and user equipment m respectively is expressed as:

[0038]

[0039] Since in D2D communication, the distance between two UEs is generally shorter than the distance between a base station and the UE, this embodiment assumes that the download content delay is

[0040] S3. Establish a D2D model and model the D2D shared link between two user devices based on the physical domain and the social domain. Due to the limitation of physical transmission power and the factor of social trust, the D2D communication link is considered to be successfully established only when the distance between the two UEs is within the maximum D2D communication range and the pair of UEs involved have a direct social relationship. Therefore, this embodiment models the D2D shared connection between the two UEs based on the physical domain and the social domain.

[0041] Physical domain: Due to the mobility of user devices, the physical distance between two users is always changing. In order to show the user's mobility pattern, an interactive contact model is adopted. The timeline of the user device contact process in the model is composed of contact time and contact interval time alternating. The contact time is the time when two user devices are within the communication range and may share the required content through the D2D link. The contact interval time is the time between two consecutive contacts. By analyzing the real user trajectory, the connection time and interval time between user device n and user device m are respectively subject to the parameters λ n，m and μ n，m The exponential distribution of .

[0042] This example defines a binary random variable The connection indicator at time slot t is that if user device n and user device m can establish a D2D communication link to transmit the requested content at time slot t, then otherwise The connection between user device n and user device m follows a continuous-time Markov chain model, expressed as:

[0043]

[0044] Where t0 represents the initial time slot.

[0045] Social domain: For UEs in social activities, they prefer to share their content with people with whom they have strong social relationships. Based on this, this embodiment establishes a social graph based on D2D communication. in Represents a collection of user devices, represents the set of social relationships between user devices; when two user devices have a close social relationship, e n,m =1, otherwise en,m = 0; use the similarity of content preference to measure the social relationship between two user devices, and use cosine similarity to calculate the similarity of content preference between user device n and user device m It is expressed as:

[0046]

[0047] In the formula, and denote the number of requests for content f by user device n and user device m in time slot t respectively; if the content preference similarity between user device n and user device m is greater than the threshold θ, it is considered that user device n and user device m have a social relationship, i.e. Otherwise n,m =0.

[0048] D2D shared link: based on physical connection and social relationships n，m ,pass e n，m The condition determines whether user equipment n and user equipment m have established a D2D shared link in time slot t. and e n,m = 1 at the same time, the D2D shared link is successfully established; when When t is t, user device m is called the neighbor of user device n, and the potential neighbor set of user device n in time slot t is expressed as If no shared connection is obtained, When, define

[0049] S4. Establish a content transmission delay model. In the D2D-assisted cellular network, the user equipment has three content caching methods: local caching, D2D caching, and cellular network caching. Each content caching method has a corresponding delay in time slot t.

[0050] Local caching method: If the content f requested by user device n is cached in its local cache content list, then the content f is allowed to be obtained locally, so the delay of obtaining the content locally is recorded as Can be considered as zero.

[0051] D2D caching method: If the user request cannot be obtained through the local cache content list, the user device n will seek help from the surrounding user devices that are allowed to establish D2D shared links. At this time, the delay corresponding to obtaining content f is recorded as It should be noted that when there are multiple user devices around that are allowed to establish a shared connection with user device n and successfully cache the requested content f, user device n will obtain the requested content from user device m with the minimum transmission delay, which is recorded as

[0052] Cellular network caching method: If the requested content cannot be obtained locally or through surrounding user devices, then user device n will send a request for the required content to the base station. At this time, the delay for user device n to obtain the preferred content f is recorded as Therefore, the delay of the user equipment obtaining the content in time slot t is expressed as:

[0053]

[0054] In the formula, A variable ranging from 0 to 1, indicating whether content f is cached in the neighbor device of the shared link of user device n if and only if there is hour,

[0055] Based on the delay of preferred content transmission, the average download delay of preferred content for all user devices in the active edge caching system is obtained as:

[0056]

[0057] When no user device is selected as a cache node to assist content transmission in the active edge caching system, the average transmission delay of all user devices to obtain their required content is expressed as:

[0058]

[0059] The average transmission delay reduced by the user device caching strategy is expressed as:

[0060] τ b =τ nc -τ c .

[0061] S5. Establish a content popularity model and a content cache model, calculate popularity based on the Hawkes model, use content hit rate as an indicator to measure the effectiveness of cache operations, and use future hit rate to estimate the response of the active edge cache system to user requests in future time slots.

[0062] In the process of user device n sending content request to the base station, an active caching scheme based on content popularity is proposed; at the beginning of each time slot, the active edge caching system collects and records the preferred content f requests made by user device n, and calculates the popularity based on the Hawkes model; the Hawkes process is a mathematical model used to model self-excited processes, which is used to describe a series of events that occur over time. The occurrence of each event will stimulate the possibility of the next thing happening, and the effect of this incentive will decay over time.

[0063] Assuming that the content popularity is the same in the area of ​​all user devices, the content popularity of each time slot is expressed as:

[0064]

[0065] In the formula, It indicates the influence of historical content on the future, and δ indicates the attenuation coefficient of historical content.

[0066] After calculating the popularity of content f, all content popularity is normalized and stored in a matrix, denoted as M; this helps to provide a clearer representation of the relative popularity of different content in the user device environment; in addition to calculating the content popularity of the current time slot, this embodiment also uses the Hawkes model to predict the content popularity of future time slots based on the current popularity and historical request information.

[0067] In each time slot, user device n actively perceives the request content f, and the active edge cache system makes a cache decision in indicates not to cache, otherwise the content f is cached on the corresponding user device n;

[0068] Assuming that all user devices have the same cache capacity limit C, the content cache state of the user device is expressed as:

[0069] I n，f =[i n，f ] N*F

[0070] In the formula, i n，f ∈{0, 1} represents the cache status of user device n for content f. n，f =1 means caching, otherwise not caching.

[0071] At the beginning of each time slot, each user device checks its cached content and actively removes any expired content from the cache space, and then records the corresponding position in the cache state matrix I as i n，f and set to 0; Assume that user device n perceives task f and makes a cache decision in time slot t Then the update process of each element in matrix I is expressed as:

[0072]

[0073] In the formula, is an indicator function, when hour, is 1 if the condition is true, otherwise it is zero.

[0074] In the cache model based on content popularity, the content hit rate is used as an indicator to measure the effectiveness of the cache operation. The hit rate of cache content f in time slot t is expressed as:

[0075]

[0076] In the formula, g n (t) represents the traffic situation near user device n.

[0077] When making a cache decision, the future benefit of the current cache decision is usually considered. Therefore, this embodiment uses the future hit rate to estimate the response of the active edge cache system to the user request in the future time slot, which is expressed as:

[0078]

[0079] In the formula, a represents the index that controls the estimated length of future time; represents the future popularity predicted by the Hawkes model, so when When , the trade-off between the current hit rate and the future hit rate is expressed as:

[0080]

[0081] Where w(m) is the discount factor, which is used to control the weight of future hit rate.

[0082] In each time slot t, the average delay of content downloading and the cache decision greatly affect the content hit rate. The goal of this embodiment is to minimize the average delay of content downloading and improve the content hit rate by continuously optimizing the download delay and cache decision.

[0083] From the above we can see that τ nc The value of has nothing to do with the cache strategy, so it will maximize the average transmission delay τ reduced by the user device cache strategy b The problem is transformed into minimizing the average download delay τ of the content c The problem is to minimize the average download delay of the preferred content and continuously optimize the cache decision to improve the content hit rate. The optimization problem is modeled as:

[0084]

[0085] Where β is a weight factor that weighs the average download delay and content hit rate, constraint C1 represents the maximum cache capacity of each user device, and constraint C2 ensures is a binary variable, and constraint C3 indicates that the cache decision takes a value between 0 and the number of user devices.

[0086] S6, predicts the user's future content requests by combining the correlation between different contents, the temporal model of user content requests and the influence of social relationships, and learns and updates parameters by minimizing the loss between the predicted number of user content requests and the actual number of user content requests.

[0087] This paper proposes a user preference content learning (CS-PL) based on content and social relationships, which predicts users' future content requests by combining the relevance between different contents, the temporal model of user content requests and the influence of social relationships. Figure 4 As shown in the figure, firstly, a stacked autoencoder combined with a bidirectional long short-term memory network (SAE-BiLSTM) is used to obtain the temporal pattern of the content requested by the user device, so as to learn a higher-order feature representation of the data; then, based on the constructed content graph, a graph convolutional network (GCN) is used to learn the similarity representation between different contents; then, considering the richer contextual feature information, a convolutional block attention module (CBAM) mechanism is used to learn the social influence representation through the user social network; finally, a fully connected network is used to combine the learned content similarity, social relationship and time series to improve the performance of user preference learning.

[0088] SAE-BiLSTM learning time series: Since most user device requests in daily life are in chronological order and have a certain temporal nature, a bidirectional long short-term memory network (BiLSTM) is used that can efficiently process sequence data. At the same time, considering that the multi-layer structure can help the model learn more complex data representations and abstract features, which helps to improve the performance of the model, a network combining stacked autoencoders (SAE) and BiLSTM is proposed, which helps to reduce the dimension of the input data and improve the computational efficiency of the model.

[0089] An autoencoder is an artificial neural network that can learn an efficient representation of input data through unsupervised learning. It uses the historical number of content requests in an observation window of length ρ time periods as input, expressed as in Represents the number of different content requests of user device n in time slot t. Use the encoder to map the input to the hidden layer representation z, and then use the decoder to map the hidden representation z back to reconstruct it. Use two latent layers to find the optimal hidden feature z to represent the input data. Train the stacked autoencoder model by minimizing the reconstruction loss function through back propagation:

[0090]

[0091] In the formula, W and b represent the weight matrix and bias vector respectively. The stacked autoencoder model is trained layer by layer. First, the input data is used as the input and output of training layer 1 to train hidden layer 1. The output of hidden layer 1 is used as the input and output of training layer 2 to train hidden layer 2. After obtaining hidden layer 2, hidden layer 1 and hidden layer 2 are stacked to obtain SAE as the input of BiLSTM. Then, the input gate is used to Forget Gate and output gate To control the memory of historical information in long and short time periods, it is expressed as:

[0092]

[0093] In the formula, are the weights and biases learned by the bidirectional long short-term memory network, and δ and tanh are both activation functions; then, in order to enhance the memory capacity of the network, BiLSTM contains an additional back-propagation layer to concatenate the context information in both directions, expressed as:

[0094]

[0095] In the formula, h t and h′ t is the output of the bidirectional long short-term memory network; finally, the output is fused through the fully connected layer to obtain the time series representation:

[0096] H n =σ(q t W n +b n ).

[0097] GCN learns content similarity: There is a strong correlation between different contents, so similar content is likely to be requested by the same user device; content similarity can well assist in learning user preferences, and the GCN module is used to learn content similarity from the constructed content graph.

[0098] In the graph convolutional network, we first construct an interaction graph based on the relevance of the content. in Represents the content set, and the sparse adjacency matrix is ​​expressed as The similarity between content i and content j is e i，j In this embodiment, the content similarity is determined by a method based on binary categories. i，j ∈{0, 1}, when content i and content j belong to the same category, e i,j =1, otherwise zero; interaction diagram Each content node f in contains a feature vector, which is the number of content requests in the historical time period ρ, recorded as And the feature matrix of all content nodes is expressed as The feature matrix As the input of the graph convolutional network, it is denoted as The forward propagation rule for each layer is expressed as:

[0099]

[0100] In the formula, and are the output and weight of the lth layer respectively, is the sum of the adjacency matrix and the identity matrix. Since ε F The diagonal of is zero, so when multiplying with the feature matrix, only the weighted sum of the features of all neighbors of a node is calculated, thus ignoring its own features; for The degree matrix of Normalize, σ is the activation function ReLU; finally, use the long short-term memory network to capture the temporal correlation of the graph convolutional network outputs in different time slots to obtain the content similarity H c .

[0101] CBAM learning social relationships: Since users with strong social relationships may have the same interest preferences, this embodiment converts the user social network graph and social similarity as input, select the top k similar users (top-k) similar to user device n, and the selection of k value is based on the setting of the experimental part; the social relationships of different users are not exactly the same, so the features of the top k users selected have different effects on user n; based on this, the CBAM attention mechanism is proposed to generate different weights for the top k users to learn the representation of social relationships. At the beginning, the fully connected layer (FCN) is used to capture the preference representation of the top k similar users, which is recorded as:

[0102]

[0103] In the formula, represents the number of different content requests of the first k similar users in time slot t; after obtaining the preferences of the first k similar users of different users, the unified input to the convolutional block attention module; the channel attention and spatial attention mechanisms of the convolutional block attention module are used to obtain their respective weights, expressed as:

[0104] M c (H)=σ(MLP(AvgPool(H))+MLP(MaxPool(H)))

[0105] M s (H′)=σ(f 7×7([AvgPool(H′);MaxPool(H′)]))

[0106] Where M c (H) and M s (H′) represents the weighted results of channel attention and spatial attention respectively; AvgPool and MaxPool represent average pooling and maximum pooling respectively; MLP represents a multi-layer perceptron, which contains one hidden layer; f 7×7 Represents the size of the convolution kernel used; σ represents the activation function; It represents the result of the calculation after the channel attention, so the representation of social influence is recorded as:

[0107]

[0108] After learning the time series H n , content similarity H c After the social influence is obtained, in order to enhance the performance of user preference learning, it is fused through the Concatenate operation and the fully connected layer. First, the time series and content similarity are fused to obtain the impact of contextual content on user preferences. Then, the social influence is combined with the time series. Finally, the three are combined to predict future user content requests. The specific process is expressed as follows:

[0109] H c，n =[H c , H n ]W c，n +b c，n

[0110] H s，n =[H s , H n ]W s，n +b s，n

[0111]

[0112] The learning and updating is done by minimizing the loss between the predicted number of user content requests and the number of user actual content requests, expressed as:

[0113]

[0114] S7. Calculate the content popularity based on the learned user preference content, then transform the active caching problem based on content popularity into a Markov decision process problem, and finally solve it using a deep reinforcement learning algorithm; at the beginning of each time slot t, the user device actively perceives the content f, and then the agent represents the state as s(t)∈S, generates an action a(t)∈A based on the state, and after executing the action, updates the system state to s(t+1) and obtains the reward r(t) after executing the action.

[0115] State space: The user device actively senses the content f in each time slot t and collects information from all computing nodes, expressed as:

[0116] s(t)={f(t), P(t), I(t), g(t)}

[0117] Where P(t) represents the content popularity matrix, I(t) represents the cache state matrix, and g(t) represents the set of popular situations near different user devices;

[0118] Action space: The agent perceives the state and makes a cache decision, recorded as

[0119] Reward function: Set as the objective function to maximize optimization, expressed as:

[0120] r(t)=max-(1-β)τ c +βH(t).

[0121] How to use the PPO algorithm in an active caching system to train a more efficient caching strategy, including the following modules: a policy network π with parameters θ θ (a|s), the value network V with parameter φ φ (s), experience pool, and environment.

[0122] like Figure 5 As shown in the figure, in each training cycle, the current policy network and the environment are interacted to collect samples, and actions are selected according to the policy, the rewards obtained are observed, and then the experience group is stored in the experience pool; then, after a period of time, a small batch of historical data is sampled from the experience pool to update the network; in the training process, the parameters of the Actor network and the Critic network are mainly updated. Different from the traditional AC algorithm, the Actor network loss function in the PPO algorithm is defined as:

[0123]

[0124] Where: It represents the ratio between the new and old strategies, which allows the PPO algorithm to control the magnitude of strategy changes each time the strategy is updated, thereby improving the stability of the algorithm; πθ (a t |s t ) is generated by the real-time interaction between the agent and the environment, is the old strategy for the agent to interact with the environment, which can be sampled from historical data in the experience pool.

[0125] In the first stage, the agent uses the current strategy to interact with the environment and generates a batch of sample data which is stored in the experience replay pool. In this stage, neither the policy network (Actor) nor the value network (Critic) is updated. In the second stage, when the sample data in the experience replay pool reaches the set amount, multiple learning processes of the policy network and the value network are started. The algorithm uses the stored sample data to optimize the two networks and gradually improve the expressive power of the policy and value functions.

[0126] The last state value generated by the environment is called s_(s_∈S), where S represents the set of all state values ​​in the entire experience pool; s_ is input into the Critic network to calculate the estimated value function of the last state Input S into the Critic network to obtain The advantage function is expressed as:

[0127]

[0128] The formula adopts the form of General Advantage Estimation (GAE), δ t =r t +γV(s t+1 )-V(s t ) represents the time difference error between time t and time t+1; γ∈[0,1] is the discount factor; λ∈[0,1] is the attenuation coefficient of GAE, which controls the degree of attenuation of future rewards. A smaller λ value will reduce the importance of future rewards, while a larger λ value will increase the importance of future rewards.

[0129] Therefore, the loss of the Critic network is back-propagated and updates the Critic network, which can be calculated as follows:

[0130] C_loss=mean(square(A t )).

[0131] The method of this embodiment optimizes the performance of edge cache by introducing a user preference learning mechanism. In modern communication networks, with the continuous increase in the number of mobile terminal devices and application requirements, traditional cache strategies are difficult to cope with the growing amount of data and user personalized needs. Especially in edge computing scenarios, how to efficiently manage cache resources to reduce transmission delays and improve cache hit rates has become an urgent problem to be solved.

[0132] To this end, this embodiment adopts an innovative user preference learning method to accurately predict users' content preferences by analyzing multi-dimensional data such as users' content request patterns, social relationships, and temporal behaviors; based on these prediction results, the cache strategy is dynamically adjusted to maximize the content hit rate and minimize the average delay of content transmission under limited cache resources.

[0133] Specifically, this embodiment models the caching problem as a Markov decision process (MDP) and uses the proximal policy optimization (PPO) algorithm in deep reinforcement learning to solve the problem; through the optimization of the PPO algorithm, the method of this embodiment can effectively balance the popularity of the cached content with the user's personalized needs, thereby achieving a higher hit rate and lower transmission delay in different user request scenarios.

[0134] Experimental results show that compared with the existing caching strategies, the method proposed in this embodiment shows better performance in different experimental scenarios, especially in complex multi-user scenarios, significantly improving the system's cache hit rate and user request response efficiency; in addition, the method of this embodiment also has good scalability and can adapt to more complex scenario requirements in future communication networks. By combining cloud computing services and other communication resources, the caching strategy of the method of this embodiment is expected to further improve the overall system performance and provide important technical support for future intelligent edge computing networks.

[0135] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A D2D-assisted edge caching algorithm based on deep reinforcement learning, characterized by: The following steps are involved: S1, establish an active edge caching system for a D2D-assisted cellular network consisting of user devices; S2, establish a time slot model, each time slot is generally divided into non-peak period and peak period; S3, establishing a D2D model to model the D2D shared link between two user devices based on the physical domain and the social domain; S4. Establish a content transmission delay model. The user equipment has three ways to cache content in the D2D-assisted cellular network: local cache, D2D cache, and cellular network cache. S5. Establish a content popularity model and a content cache model, calculate popularity based on the Hawkes model, use the content hit rate as an indicator to measure the effectiveness of cache operations, and use the future hit rate to estimate the response of the active edge cache system to user requests in future time slots; S6, predicting the user's future content requests by combining the correlation between different contents, the temporal model of user content requests, and the influence of social relationships, and learning and updating parameters by minimizing the loss between the predicted number of user content requests and the actual number of user content requests; S7. Calculate the content popularity based on the learned user preference content, and then transform the active caching problem based on content popularity into a Markov decision process problem. By continuously interacting with the environment and optimizing and updating its own network, the optimal caching strategy is obtained.

2. A D2D assisted edge caching algorithm based on deep reinforcement learning according to claim 1, characterized in that: In step S1, the active edge cache system includes a base station and N randomly distributed user equipments. Represented as a collection of user devices, Indicates that the user device is from the content library The content requested in the above example is , F represents the number of contents, assuming that each user device can select the content from the content library Request any content in the f ; The user equipment can also act as a cache node and provide cache services, and share cache content with other user equipment through D2D communication; The cache capacity of user equipment n is represented by S n , the cache status is recorded as When content f is cached in user device n at time slot t, otherwise 3. According to the D2D assisted edge caching algorithm based on deep reinforcement learning according to claim 1, it is characterized by: In step S2, at the beginning of each time slot, a deep learning model is trained offline on the data of historical content requests, and then the user device preferences for different contents are predicted by the trained deep learning model; Based on the prediction of user device preferences, corresponding cache decisions are obtained and cache content on all user devices is updated; Then, content is delivered according to the needs of user devices during peak hours. represents the cache content request vector of user device n, where represents the number of requests for content f and If content f is not available in time slot t, then A resource scheduler is set up to dynamically allocate subchannels to user devices. Since the active edge caching system suffers from both large-scale and small-scale fading, the transmission rate of content f from user device m and base station to user device n is expressed as: In the formula, B D2D and B BS They represent the bandwidth of D2D link transmission and cellular link channel, P UE and P BS Respectively represent the transmit power of the user equipment and the base station, h m，n and h s，n Denote the Rayleigh fast fading coefficients of the communication link between user equipment m and user equipment n and between the base station and user equipment n, d m，n and d s，n They represent the physical distances from user equipment n to user equipment m and the base station, α is the path loss index, σ 2 represents the noise power; Therefore, the delay of user equipment n downloading content f from the base station and user equipment m respectively is expressed as: In which, assuming that the download content is delayed 4. The D2D-assisted edge caching algorithm based on deep reinforcement learning according to claim 1, characterized in that: The physical domain in step S3 adopts an interactive contact model, in which the timeline of the user equipment contact process in the model is composed of contact time and contact interval time alternating, the contact time is the time when two user equipments are within the communication range and may share the required content through the D2D link, and the contact interval time is the time between two consecutive contacts; By analyzing the real user trajectory, the connection time and interval time between user device n and user device m are respectively subject to the parameters λ n,m and μ n，m exponential distribution of Define a binary random variable The connection indicator at time slot t is that if user device n and user device m can establish a D2D communication link to transmit the requested content at time slot t, then otherwise The connection between user device n and user device m follows a continuous-time Markov chain model, expressed as: Where t0 represents the initial time slot.

5. The D2D assisted edge caching algorithm based on deep reinforcement learning according to claim 1, characterized in that: The social domain in step S3 establishes a social graph based on D2D communication. in Represents a collection of user devices, represents the set of social relationships between user devices; when two user devices have a social relationship, e n,m =1, otherwise e n,m =0; The similarity of content preferences is used to measure the social relationship between two user devices, and the cosine similarity is used to calculate the content preference similarity between user device n and user device m. It is expressed as: In the formula, and denote the number of requests for content f by user device n and user device m in time slot t respectively; if the content preference similarity between user device n and user device m is greater than the threshold θ, it is considered that user device n and user device m have a social relationship, i.e. Otherwise n,m =0.

6. The D2D assisted edge caching algorithm based on deep reinforcement learning according to claim 1, characterized in that: The D2D shared link in step S3 is based on the physical connection and social relationships n，m ,pass The condition determines whether user equipment n and user equipment m have established a D2D shared link in time slot t. and e n,m = 1 at the same time, the D2D shared link is successfully established; when When t is t, user device m is called the neighbor of user device n, and the potential neighbor set of user device n in time slot t is expressed as If no shared connection is obtained, When, define 7. The D2D-assisted edge caching algorithm based on deep reinforcement learning according to claim 1, characterized in that: In step S4, each content caching mode has a corresponding delay in time slot t. For the local caching mode, if the content f requested by the user device n is cached in its local cache content list, then the content f is allowed to be obtained locally, so the delay in obtaining the content locally is recorded as That is zero; For the D2D caching method, if the user request cannot be obtained through the local cache content list, the user device n will seek help from the surrounding user devices that are allowed to establish a D2D shared link. At this time, the delay corresponding to obtaining the content f is recorded as When multiple user devices are allowed to establish a shared connection with user device n and successfully cache the requested content f, user device n will obtain the requested content from user device m with the minimum transmission delay, which is recorded as For the cellular network cache method, if the requested content cannot be obtained locally or through surrounding user devices, then user device n will send a request for the required content to the base station. At this time, the delay for user device n to obtain the preferred content f is recorded as Therefore, the delay of the user equipment obtaining the content in time slot t is expressed as: In the formula, A variable ranging from 0 to 1, indicating whether content f is cached in the neighbor device of the shared link of user device n if and only if there is hour, Based on the delay of preferred content transmission, the average download delay of preferred content for all user devices in the active edge caching system is obtained as: When no user device is selected as a cache node to assist content transmission in the active edge caching system, the average transmission delay of all user devices to obtain their required content is expressed as: The average transmission delay reduced by the user device caching strategy is expressed as: t b =t nc -t c 。 8. The D2D-assisted edge caching algorithm based on deep reinforcement learning according to claim 1, characterized in that: In step S5, at the beginning of each time slot, the active edge cache system collects and records the requests for preferred content f made by the user device n, and calculates the popularity based on the Hawkes model; assuming that the content popularity is the same in the areas of all user devices, the content popularity of each time slot is expressed as: In the formula, It indicates the influence of historical content on the future, and δ indicates the attenuation coefficient of historical content; After calculating the popularity of content f, all content popularity is normalized and stored in a matrix, denoted as M; the Hawkes model is used to predict the content popularity of future time slots based on the current popularity and historical request information; In each time slot, user device n actively perceives the request content f, and the active edge cache system makes a cache decision in indicates not to cache, otherwise the content f is cached on the corresponding user device n; Assuming that all user devices have the same cache capacity limit C, the content cache state of the user device is expressed as: I n，f =[i n，f ] N*F In the formula, i n，f ∈{0, 1} represents the cache status of user device n for content f. n，f =1 means caching, otherwise no caching; At the beginning of each time slot, each user device checks its cached content and actively removes any expired content from the cache space, and then records the corresponding position in the cache state matrix I as i n，f and set to 0; Assume that user device n perceives task f and makes a cache decision in time slot t Then the update process of each element in matrix I is expressed as: In the formula, is an indicator function, when hour, is 1, otherwise it is zero; In the content cache model, the content hit rate is used as an indicator to measure the effectiveness of the cache operation. The hit rate of cache content f in time slot t is expressed as: In the formula, g n (t) represents the traffic situation near user device n; The future hit rate is used to estimate the response of the active edge cache system to the user request in the future time slot, which is expressed as: In the formula, a represents the index that controls the estimated length of future time; represents the future popularity predicted by the Hawkes model, so when When , the trade-off between the current hit rate and the future hit rate is expressed as: Where w(m) is the discount factor, which is used to control the weight of future hit rate; Maximize the average transmission delay τ reduced by the user device caching strategy b The problem is transformed into minimizing the average download delay τ of the content c The problem is to minimize the average download delay of the preferred content and continuously optimize the cache decision to improve the content hit rate. The optimization problem is modeled as: Where β is a weight factor that weighs the average download delay and content hit rate, constraint C1 represents the maximum cache capacity of each user device, and constraint C2 ensures is a binary variable, and constraint C3 indicates that the cache decision takes a value between 0 and the number of user devices.

9. The D2D-assisted edge caching algorithm based on deep reinforcement learning according to claim 1, characterized in that: In step S6, a stacked autoencoder combined with a bidirectional long short-term memory network is first used to obtain the temporal pattern of the content requested by the user device; then, based on the constructed content graph, a graph convolutional network is used to learn the similarity representation between different contents; then, a convolutional block attention module is used to learn the social influence representation through the user social network; finally, the learned content similarity, social relationship and time series are combined through a fully connected network.

10. The D2D assisted edge caching algorithm based on deep reinforcement learning according to claim 9, characterized in that: In step S6, the stacked autoencoder is combined with the bidirectional long short-term memory network, and the historical number of content requests in the observation window with a length of ρ time periods is used as input, which is expressed as in Represents the number of different content requests of user device n in time slot t. Use the encoder to map the input to the hidden layer representation z, and then use the decoder to map the hidden representation z back to reconstruct it. Use two latent layers to find the optimal hidden feature z to represent the input data. Train the stacked autoencoder model by minimizing the reconstruction loss function through back propagation: Where W and b represent the weight matrix and bias vector respectively; the stacked autoencoder model is trained layer by layer, and then the input gate Forget Gate and output gate To control the memory of historical information in long and short time periods, it is expressed as: In the formula, are the weights and biases learned by the bidirectional LSTM network, and δ and tanh are both activation functions; the bidirectional LSTM network contains an additional back-propagation layer to concatenate the context information in both directions, expressed as: In the formula, h t and h′ t is the output of the bidirectional long short-term memory network; finally, the output is fused through the fully connected layer to obtain the time series representation: H n =σ(q t W n +b n ) In the graph convolutional network, we first construct an interaction graph based on the relevance of the content. in Represents the content set, and the sparse adjacency matrix is ​​expressed as The similarity between content i and content j is e i，j , judging content similarity through a binary category-based approach i，j ∈{0, 1}, when content i and content j belong to the same category, e i，j =1, otherwise zero; interaction diagram Each content node f in contains a feature vector, which is the number of content requests in the historical time period ρ, recorded as And the feature matrix of all content nodes is expressed as The feature matrix As the input of the graph convolutional network, it is denoted as The forward propagation rule for each layer is expressed as: In the formula, and are the output and weight of the lth layer respectively, is the sum of the adjacency matrix and the identity matrix. Since ε F The diagonal of is zero, so when multiplying with the feature matrix, only the weighted sum of the features of all neighbors of a node is calculated, thus ignoring its own features; for The degree matrix of Normalize, σ is the activation function ReLU; finally, use the long short-term memory network to capture the temporal correlation of the graph convolutional network outputs in different time slots to obtain the content similarity H c ; In the convolutional block attention mechanism, the user social network graph and social similarity as input, select the top k similar users similar to user device n, first capture the preference representation of the top k similar users through the fully connected layer, recorded as: In the formula, represents the number of different content requests of the first k similar users in time slot t; after obtaining the preferences of the first k similar users of different users, the unified input to the convolutional block attention module; the channel attention and spatial attention mechanisms of the convolutional block attention module are used to obtain their respective weights, expressed as: M c (H)=σ(MLP(AvgPool(H))+MLP(MaxPool(H))) M s (H′)=σ(f 7×7 ([AvgPool(H′);MaxPool(H′)])) Where M c (H) and M s (H′) represents the weighted results of channel attention and spatial attention respectively; AvgPool and MaxPool represent average pooling and maximum pooling respectively; MLP represents a multi-layer perceptron, which contains one hidden layer; f 7×7 Represents the size of the convolution kernel used; σ represents the activation function; It represents the result of the calculation after the channel attention, so the representation of social influence is recorded as: After learning the time series H n , content similarity H c After the social influence is obtained, it is fused through the Concatenate operation and the fully connected layer. First, the time series and content similarity are fused to obtain the impact of contextual content on user preferences. Then, the social influence is combined with the time series. Finally, the three are combined to predict future user content requests. The specific process is expressed as follows: H c，n =[H c ,H n ]W c，n +b c，n H s，n =[H s ,H n ]W s，n +b s，n The learning and updating is done by minimizing the loss between the predicted number of user content requests and the number of user actual content requests, expressed as:

Images