5G cloud edge collaborative content caching method

Abstract

The present application relates to the field of 5G and digital information transmission, and particularly relates to a 5G cloud edge collaborative content caching method, which comprises the following steps: in a cloud edge collaborative scenario, considering factors such as hit rate, transmission delay and traffic load, a user experience is modeled; the constructed cloud edge collaborative caching model is solved, wherein the solving process comprises content prediction based on a Transformer and multi-agent reinforcement learning technology; the caching model is divided into two parts; the caching prediction scheme based on the Transformer, a deep neural network and hyper-dimensional calculation is used for solving; and the deep reinforcement learning method is used for solving; the present application comprehensively considers user preference information, popular information and user historical browsing records, and designs a differentiated feature extraction scheme, so that a high prediction accuracy is achieved.

Description

Technical Field

[0001] This invention relates to the field of 5G and digital information transmission, and specifically to a 5G cloud-edge collaborative content caching method. Background Technology

[0002] With the rapid development of 5G, cloud computing, and edge computing technologies, cloud-edge collaborative architecture has gradually become the mainstream architecture for next-generation information systems. Cloud computing, relying on its powerful computing capabilities and elastic scalability, provides users with faster and more efficient computing and storage services. Edge computing decentralizes some nodes to areas closer to users, further reducing network transmission latency. Cloud-edge collaboration technology combines cloud computing and edge computing technologies, leveraging the advantages of both: on the one hand, it enables the entire network service to have efficient computing and storage capabilities; on the other hand, because edge nodes are close to users, they can provide efficient network services. These two characteristics have led to the widespread application of cloud-edge collaboration technology in various scenarios to improve service quality and enhance user experience.

[0003] Users' overall subjective perception of network quality directly reflects the quality of network services, also known as experience quality. This metric is closely related to traffic load and network latency. Cloud-edge collaboration technology can offload some storage or computing tasks to edge nodes. These edge nodes, being closer to users, can intelligently, periodically, and proactively cache and update cached content, thereby reducing network load and latency, and ultimately improving user experience quality. Existing methods for improving user experience quality using cloud-edge collaboration technology focus on content prediction, that is, accurately predicting the content that users are about to access and caching it in advance to edge nodes, thereby reducing load and latency. Existing content prediction-based user experience quality improvement solutions can be divided into two categories: statistical content prediction solutions and artificial intelligence-based content prediction solutions.

[0004] Among these, content prediction schemes based on statistics rely on statistical analysis to summarize user preferences for content and thus make predictions. For example, the Least Frequently Used (LFU) strategy makes cache replacement decisions based on the access frequency of a specific individual user; frequently accessed data items are retained in the cache, while infrequently accessed items may be deleted. The K-th Least Frequently Used (LRU-K) strategy determines the caching policy based on data from K users to improve prediction accuracy. These schemes are relatively simple and have lower computational and storage costs.

[0005] AI-based content prediction solutions leverage the powerful data mining capabilities of AI algorithms to extract relevant features from historical user data and predict the content users will access in the next moment, thereby improving user experience. These solutions can be divided into two types: the first is time-series prediction, which uses algorithms such as Long Short-Term Memory (LSTM) and Transformer to predict popular content among users based on historical browsing information, improving the accuracy of cached content; the second is reinforcement learning-based solutions, which use rewards received by the agent during interaction with the environment to optimize parameters and obtain the optimal caching strategy.

[0006] While existing solutions can improve user experience by predicting cached content, they still have the following problems:

[0007] 1. Existing solutions rely on coarse-grained prediction criteria when predicting cached content, only considering users' historical data while ignoring factors such as popular information and user preferences. Furthermore, existing solutions do not design differentiated feature extraction methods based on data characteristics, resulting in low prediction accuracy and consequently a poor user experience.

[0008] 2. Existing solutions based on reinforcement learning only consider the decisions of a single agent when making decisions. In addition, the limited storage capacity of the experience replay area makes the data quality relatively random, which leads to performance degradation and large decision errors during the training process.

[0009] Therefore, a 5G cloud-edge collaborative content caching method needs to be designed to address the current shortcomings. Summary of the Invention

[0010] This invention addresses the problems existing in the prior art by providing a 5G cloud-edge collaborative content caching method. This method comprehensively considers multiple factors, such as user preference information, browsing history, regional hot topics, network load, and network latency, to improve cache hit rate, thereby minimizing traffic load and network latency and enhancing user experience.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A prediction accuracy model is constructed based on cosine similarity theory. and cache hit rate The way of expression;

[0013] According to the user at any time The method for obtaining the requested content and the method for constructing a representation of the overall transmission latency;

[0014] Construct a representation of the total traffic cost based on the way the overall transmission delay is expressed;

[0015] Taking into account hit rate, transmission latency, and traffic load, we construct a way to express the user experience and further build a caching model;

[0016] Split the caching model into and Two parts;

[0017] The solution is obtained through a cache prediction scheme based on Transformer, deep neural networks, and hyperdimensional computation. ;

[0018] Solving the problem using deep reinforcement learning methods. .

[0019] In a specific feasible implementation, prediction accuracy The expression is as follows: ;

[0020] Cache hit rate The expression is as follows: ;

[0021] In the above formula, for the first... edge nodes ,

[0022] At any moment It predicts the content the user will visit in the next moment. ,

[0023] Users The content actually accessed at any given time is ,

[0024] The cached content after the caching decision is completed is represented as follows: ,

[0025] Indicates that the edge node is The first time cache Contents.

[0026] In a specific feasible implementation, the user at any time The methods for obtaining the requested content include:

[0027] Local communication: If edge nodes If the content is already cached locally, then directly... from Transmitted to user ;

[0028] Proximity communication: If the content requested by the user is on an edge node If there is no local cache, then Try from its adjacent edge nodes Retrieve this content;

[0029] Cloud communication: If the content requested by the user is not cached on the edge node or its adjacent nodes, then The cache is retrieved from the cloud computing center via a backhaul link.

[0030] In a specific feasible implementation, the overall transmission delay is expressed as: ;

[0031] in, Indicates transmission delay.

[0032] represent A nearby edge node and cloud computing center;

[0033] This indicates that the content is related to... from The delay in retrieving this content; Represents edge server At any moment Received content The number of requests;

[0034] Indicates at time exist China's target Was the request accepted? If satisfied, then ,otherwise ;

[0035] Total traffic cost Represented as: ;

[0036] in, express and The cost of traffic between them.

[0037] In a specific feasible implementation, the user experience is represented as:

[0038] In the formula, express from Obtain content from Time required;

[0039] Content The function of utilization;

[0040] Representatives obtain through direct communication The cost;

[0041] Representative equipment in Speed ​​of movement in time;

[0042] Representative equipment in Transmission radius over time;

[0043] and All of these are communication weight parameters;

[0044] The constraints are satisfied:

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051] In the above formula, and The expected value of the prediction accuracy. , , These are the weight parameters.

[0052] In a specific feasible implementation, Represented as: ;

[0053] Represented as: .

[0054] In a specific feasible implementation, for The solution process is as follows: preprocess the data, and use high-dimensional calculation methods to further map the vectorized cached content to a hyperdimensional space;

[0055] The preprocessed data is used as input, and user preferences, historical information and popular information are extracted by passing through the user preference extraction network, historical information extraction network and popular information extraction network respectively. The user preference extraction network consists of a fully connected neural network with only one layer.

[0056] The Transformer temporal neural network is used to predict historical and trending information for the next time step. The encoder-decoder structure in the Transformer is used to predict the sequence, and the embedding layers of the encoder and decoder are combined with high-dimensional vectors, where vector operations replace the original floating-point operations.

[0057] The prediction results are combined to obtain the final content matrix.

[0058] In a specific feasible implementation, a temporal sequence neural network, Transformer, is used to predict historical and trending information for the next time step. The encoder-decoder structure in Transformer is used to predict the sequence, and the embedding layers of the encoder and decoder are combined with high-dimensional vectors, including the following steps:

[0059] Hot News Prediction Network Historical information extraction network Employs the encoder-decoder architecture of Transformer;

[0060] For known time series data The predicted samples are split into: and ,in, It is the input data. It is the output target;

[0061] The encoder's embedding layer is used to generate hyperdimensional vectors, and the vectorized matrix is ​​represented as: The Transformer uses different trigonometric functions for encoding based on whether the position is odd or even, namely:

[0062]

[0063]

[0064] in Indicates position code, Indicates the position of the token. This represents the index of each element in the vector. It is the dimension of the positional encoding vector;

[0065] Establish three different matrices, each called a query matrix. Keywords Sum Each matrix is ​​obtained through a linear transformation, i.e. The trainable weight matrix is ​​represented as ;

[0066] The self-attention module in Transformer calculates pairwise similarities between each query and all keys, capturing... The relationships between various vectors within the input are used to capture key information from the input, denoted as... , This includes a total One vector;

[0067] The output is all values Weighted sum: softmax() is the activation function, where yes , and The dimension;

[0068] Repeat line execution Next, the relationship between elements at the same position and elements at multiple positions is extracted. The multi-head attention module includes... Each head is equipped with a different trainable weight matrix. ;

[0069] Input for each head Afterwards, The outputs of the head are combined into a unified matrix. The outputs of multiple attention heads are combined to form the total output; the output of the multi-head attention module is... ,in, These are the weight parameters of the neural network;

[0070] For popular information The prediction, during the update process, uses complete historical browsing data at the next moment. To calculate the loss and perform gradient updates; for predictions based on historical information, a historical information extraction network is used. The update is its output data and Results from the linear layer: .

[0071] In a specific feasible implementation, the state space, action space, and reward are defined;

[0072] Represented as The Actor network is represented as The Critic network, represented as by The target actor network, and represented as The target critic network's parameter updates during training are divided into two parts: updating the critic network and updating the Actor network.

[0073] When updating the Critic network, first select a set of samples. and And use the target Actor network to calculate the state. The action taken in the state, i.e. ;

[0074] Calculate state-action combinations using a target critic network. , The target value, i.e. The done parameter indicates whether training is complete; it is 1 if training is complete, and 0 otherwise.

[0075] Calculate the evaluation value of the state-action combination using the Critic network: The gradient descent algorithm is used to reduce the difference between the evaluated and expected values, thereby updating the parameters of the Critic network.

[0076] When updating the Actor network, first use the Actor network to calculate the state. The action in, namely calculation ;

[0077] The Critic network is used to calculate the evaluation value of the state-action combination, i.e., the cumulative expected return. ;

[0078] Use the gradient ascent algorithm to maximize the cumulative expected return.

[0079] In a specific feasible implementation, solve During the process, a priority value was established, which was evaluated by the variance between the current and target distributions, denoted as... The goal is to reduce The priority value is mapped to a range of 0-1 and treated as the probability of experience replay.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] (1) This invention addresses the problem of decreased user experience quality caused by inaccurate proactive caching by providing an intelligent solution from cached content prediction to decision-making. This solution comprehensively considers multiple indicators, including user preference information, historical browsing records, regional popular information, network load, network latency, and other factors, to improve user experience quality. Specifically, when predicting cached content, this invention designs a fine-grained feature extraction scheme. On the one hand, it comprehensively considers user preference information, popular information, and historical browsing data. On the other hand, it designs differentiated feature extraction schemes based on the characteristics of different data, thereby efficiently and comprehensively extracting user data features and achieving a high result prediction accuracy.

[0082] (2) The present invention uses an efficient and accurate decision-making scheme, comprehensively considers the decisions of multiple agents, and utilizes a priority experience playback strategy to improve the performance during the training process, making the decision more accurate and improving the quality of user experience. Attached Figure Description

[0083] Figure 1 This is a flowchart of the method of the present invention;

[0084] Figure 2 This is a schematic diagram of the existing cloud-edge collaborative network architecture;

[0085] Figure 3 This is a diagram of the content prediction framework based on Transformer of the present invention. Detailed Implementation

[0086] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments. The technical features of each embodiment of the present invention can be combined accordingly, provided that there is no mutual conflict.

[0087] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0088] In the description of this invention, it should be understood that when an element is considered to be "connected" to another element, it can be a direct connection to the other element or an indirect connection, i.e., there is an intermediate element. Conversely, when an element is said to be "directly" connected to another element, there is no intermediate element.

[0089] Combination Figure 2 As shown, a typical cloud-edge collaborative network includes the following components:

[0090] 1. A cloud computing center (labeled as a cloud server in the diagram) is represented as: It has ample computing and storage capabilities;

[0091] 2. An edge computing node (hereinafter referred to as "edge node") is represented as... Compared to cloud computing centers, each edge node has weaker computing and storage capabilities. Edge nodes can communicate with cloud computing centers and also communicate with each other. Edge nodes are typically deployed at 5G base stations, caching content for mobile users to access, thereby reducing transmission latency and improving user experience. An edge node... It serves a number of mobile users, each of whom is only within the range of one edge node at any given time. It can interact with other nodes It communicates with the cloud computing center to understand the contents of other edge nodes and the cloud computing center's cache, and based on this, combines... To determine the preferences of users within the service area Its own caching decisions.

[0092] 3. Each mobile user (hereinafter referred to as "user") is represented as Each user is within the service range of a certain edge node at any given time. Users will access Internet content according to their own preferences and expect to obtain a better user experience, such as smooth playback, low latency, and high bandwidth.

[0093] In the above scenario, the present invention will use the user At any moment The accessed content is represented as , The user's historical browsing content is represented as The most popular information in the current area is The edge node to which this user belongs The currently cached content is , , This represents the total storage capacity of the current edge node. The predicted user access content is In this scenario, this method optimizes the caching decisions and cache hit rate of edge nodes, ensuring that the content requested by users is hit on edge nodes as much as possible, thereby improving the cache hit rate. This allows user requests to be satisfied on their respective edge nodes as much as possible, while minimizing the need to retrieve content from other nodes or cloud computing centers, thus improving the user experience.

[0094] Example

[0095] Combination Figure 1As shown, this embodiment provides a 5G cloud-edge collaborative content caching method, which comprehensively considers user preference information, popular information, and user browsing history, and designs a differentiated feature extraction scheme to achieve a high accuracy rate in predicting cached content. Furthermore, based on the proposed high-accuracy content prediction scheme, this method designs an efficient and accurate decision-making scheme based on deep reinforcement learning, resulting in a high cache hit rate, lower network load and latency, and improved user experience quality. Specifically, it includes the following steps:

[0096] S1. In cloud-edge collaboration scenarios, user experience is modeled by comprehensively considering factors such as hit rate, transmission latency, and traffic load.

[0097] In this embodiment, the hit rate analysis process is as follows: for edge nodes At any moment It predicts the content the user will visit in the next moment. Users The content actually accessed at any given time is The cached content after the caching decision is completed is represented as .

[0098] Based on cosine similarity theory, the prediction accuracy will be... Represented as: .

[0099] ,

[0100] cache hit rate The above formula is expressed as follows, where, Represents edge nodes exist The first time cache Contents.

[0101] In this embodiment, the analysis process of transmission delay and traffic load is as follows:

[0102] At any moment ,user Requested content It can be obtained in the following three ways:

[0103] First method: Local communication: If the edge node If the content is already cached locally, then directly... from Transmitted to user ;

[0104] The second method: Nearest neighbor communication: If the content requested by the user is on the edge node. If there is no local cache, then Try from its adjacent edge nodes This content can be obtained because the communication links between edge nodes have high bandwidth and low latency, thus enabling [the processing of this content]. Provide a good user experience;

[0105] The third method: Cloud communication: If the content requested by the user is not cached on the edge node or its adjacent nodes, then Retrieving the cache from the cloud computing center via a backhaul link typically introduces relatively higher latency and puts a certain load on the link compared to near-neighbor communication.

[0106] Combining these three methods, the overall transmission delay can be expressed as: ;in, Indicates transmission delay. , represent A nearby edge node and cloud computing center; This indicates that the content is related to... from The delay in retrieving this content; Represents edge server At any moment Received content The number of requests; Indicates at time exist China's target Was the request accepted? If satisfied, then ,otherwise .

[0107] Based on this, the total traffic cost Represented as:

[0108] in express and The traffic cost between them. Correspondingly, the link load will be... Represented as: .

[0109] To improve the user experience, the user experience is represented as:

[0110] In the formula, express from Obtain content from Time required; Content The function of utilization; Representatives obtain through direct communication The cost, among which, Representative equipment in Speed ​​of movement in time Representative equipment in The transmission radius in time. If Cannot be obtained directly Then it needs to be obtained from the base station, at a cost of ,in and These represent speed and transmission radius, respectively. and These are all communication weight parameters.

[0111] Based on the above analyses of hit rate, communication latency and traffic load, and user experience, the completed cloud-edge collaborative caching model is used... Specifically, it means: ;

[0112] The above caching model must meet the following constraints:

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119] In the formula, and The expected value of the prediction accuracy is [0,1]. , , These are weight parameters, used to adjust prediction accuracy, link load, and transmission latency, respectively.

[0120] S2. Solve the constructed cloud-edge collaborative caching model. The solution process includes content prediction based on Transformer and multi-agent reinforcement learning techniques.

[0121] In this embodiment, the solution process is specifically as follows: The cache model... Split into two parts, set as and The corresponding expressions for the two are as follows:

[0122] ;

[0123] .

[0124] Furthermore, a differentiated feature extraction scheme based on Transformer is employed to solve the problem. This method captures the contextual relationships of long-term spatiotemporal sequences through an attention mechanism, thereby achieving high accuracy. Specifically, to improve the hit rate of cached content at edge nodes, this method proposes a cache prediction scheme based on Transformer, deep neural networks, and hyperdimensional computing. This scheme comprehensively considers user preference information, historical information, and popular information, such as... Figure 3 As shown.

[0125] The solution process is as follows:

[0126] First, this method preprocesses the data, using high-dimensional computation methods to further map the vectorized cached content into a hyperdimensional space, effectively avoiding interference from sparse, similar data and thus improving the algorithm's prediction accuracy. Next, the preprocessed data is used as input, passing through user preference extraction networks, historical information extraction networks, and popular information extraction networks to extract user preferences, historical information, and popular information, respectively. For efficient training, the user preference extraction network consists of a single-layer fully connected neural network. Since historical and popular information change with temporal and spatial order, and information from the previous moment influences information from the next moment, this method uses a temporal sequence neural network, the Transformer, to predict historical and popular information for the next moment. This method primarily uses the encoder-decoder structure in the Transformer to predict the sequence, combining the embedding layers of the encoder and decoder with high-dimensional vectors, where original floating-point operations are replaced by vector operations. Finally, the prediction results are synthesized to obtain the final content matrix.

[0127] like Figure 3 As shown, cached content prediction mainly consists of three networks: a user preference extraction network, a user history information extraction network, and a popular information prediction network. Among these three networks, the HD encoder layer is essential; it is responsible for sparsifying the original feature vector and mapping it to a hyperdimensional space, using this as a sample for training the network. The network structure is described in detail in sequence.

[0128] In this embodiment, the encoder layer of hyperdimensional computation is inspired by the workings of neurons in the brain, where the operation is viewed as point-to-point or vector-to-vector pairs in hyperdimensional space. Hyperdimensional vectors in hyperdimensional computation refer to vectors with extremely high dimensions. Due to their ultra-high dimensionality, hyperdimensional vectors are approximately orthogonal. This means that the scheme can easily use hyperdimensional vectors to represent different features. Furthermore, hyperdimensional space has enormous capacity and strong resistance to noise and interference.

[0129] This embodiment uses a record-based encoder for encoding, where a different channel is assigned to each feature position of the input vector in the encoding scheme. . Composed of random hyperdimensional vectors, ensuring that all features are related to the channels. Related, , .in Indicates element-wise similarity between vectors. It represents the dimension of the generated hyperdimensional vector.

[0130] The encoder examines each position of the input feature vector and converts the channels... Applied sequentially to the corresponding hypervector elements The subsequent equations illustrate this. In the class The combination of intrinsic input vectors and hyperdimensional vectors. This method uses This is used to represent the encoding process.

[0131] In user preference extraction network In this context, user personal characteristics, including age, gender, and occupation, are primarily used as input. The output is the user's preference for cached content types based on these characteristics. However, in actual communication, cached information cannot be directly used for training and inference within the network. Therefore, it is necessary to first extract and combine these features. User personal characteristics are represented as a set. The cache information features are represented as Vectorization can be represented as ,in , and These represent the embedding layer, linear layer, and ReLU activation function, respectively. Although individual user characteristics vary greatly, user preferences for cached content are not significantly different. Feature extraction is performed using the encoding layer in hyperdimensional computation: ,in, Used as training samples. Prediction is performed using a DNN with fully connected layers: ,in It is a trainable weight matrix. It's a bias. The loss function of the user preference extraction network is defined as... , The function represents the mean squared error, which is the mean of the sum of squares of the differences between the predicted and actual values.

[0132] The trending information prediction network and the historical information extraction network adopt the Transformer encoder-decoder structure, and are defined as follows: and However, the two networks differ in their input, output, and update processes. For known time series data... The predicted samples are split into: and ,in, It is the input data. It is the output target. In this way, training enables the network to not only infer... Moreover, they were able to learn The prediction results will then be used as historical information for further predictions. The sequence. The encoder's embedding layer is used to generate hyperdimensional vectors, therefore the vectorized matrix is ​​represented as: To ensure minimal positional coding differences and to reflect relative positional information, the Transformer employs different trigonometric functions for encoding based on whether the position is odd or even, as follows: ,

[0133] ,

[0134] in Indicates position code, Indicates the position of the token. This represents the index of each element in the vector. It is the dimension of the positional encoding vector.

[0135] The self-attention module in the Transformer is used to capture... The relationships between various vectors within the input are used to capture key information from the input, denoted as... , This includes a total There are several vectors. To achieve this goal, three different matrices are constructed, called query vectors and query vectors respectively. Keywords Sum Each matrix is ​​obtained through a linear transformation, i.e. The trainable weight matrix is ​​represented as The self-attention module calculates pairwise similarity between each query and all keys. The output is all values Weighted sum: softmax() is the activation function, where yes , and The dimension of multi-head self-attention. Multi-head attention refers to parallel execution based on the above self-attention. This process extracts the relationship between elements at the same position and elements at multiple positions. To improve accuracy, the multi-head attention module includes... Each head is equipped with a different trainable weight matrix. Applying a self-attention mechanism to each head input. Afterwards, The outputs of the head are combined into a unified matrix. Finally, the outputs of multiple attention heads are combined to form the total output; the output of the multi-head attention module is... ,in, These are the weight parameters of the neural network.

[0136] In Transformer, the decoder's structure is similar to that of the encoder. Both encoders and decoders typically consist of multiple layers, each composed of two main parts: a multi-head attention mechanism and a multilayer perceptron. The multilayer perceptron consists of bilinear layers with Gaussian error linear element (GELU) activation functions. For the first... The input and output of a layer are defined as follows: ,in This represents a normalization operation to avoid overfitting. Residual connections are used to mitigate gradient degradation. Finally, the Transformer output is... .

[0137] For popular information The prediction requires complete historical browsing data as input. Therefore, during the update process, it uses the complete browsing history data at the next moment. To calculate the loss and perform gradient updates. For predictions based on historical information, since it has already been based on... The user characteristics described above determine the user's preference type for cached content; therefore, the input data should be historical browsing data that does not include content types. The output is the same. Historical information extraction network The update is its output data and Results from the linear layer: Finally, and The prediction results are merged into And compare it with the content required by the user to obtain the loss: .

[0138] Furthermore, this can be solved using deep reinforcement learning methods. This provides support for caching decisions. The specific solution process is as follows:

[0139] Reinforcement learning consists of two main parts: the agent and the environment. Throughout the reinforcement learning process, the agent interacts with the environment to maximize rewards. When the agent acquires a state from the environment, it outputs an action based on that state. This action is then executed within the environment, which outputs the next state and scores the current action based on the agent's chosen action. The agent's goal is to learn through training to take the most appropriate action and maximize rewards from the environment. In discrete time steps, continuous probabilistic expressions are manipulated to ensure that the model's output closely approximates the expected value during extensive computation. These prediction and decision-making algorithms are collectively referred to as multi-agent reinforcement learning methods.

[0140] Multi-agent deep deterministic policy gradient (MDPR) addresses the dynamics and complexity of multi-agent environments. In such environments, each agent's behavior depends not only on the state of the environment but also on the policies of other agents. MDPR improves stability by employing an independent Actor-Critic architecture for each agent and incorporating policy information from other agents during training. To further enhance training efficiency, a priority experience replay approach is added, resulting in a reinforcement learning scheme based on priority experience replay.

[0141] The reinforcement learning scheme based on priority experience replay consists of three parts: state space, action space, and reward, as defined below.

[0142] 1. Status: In time Each agent's state includes predicted information and cached information. Therefore, the state of a single agent is established as follows: ,in It represents the set of current predictions. This indicates the content that has been cached.

[0143] 2. Action: In time In this process, each agent observes the current state and determines its current decision-making strategy based on that state, i.e., whether to act within the given timeframe. Internal execution cache operations. Continuous probabilities are used to represent the likelihood of discrete actions occurring; the specific action space is defined as follows: The action space represents the probability that the edge server will perform a caching operation. Therefore, a random value is defined when selecting an action. When the probability of an action is greater than this random value, it is considered that the subject has performed the action, and vice versa.

[0144] 3. Rewards: After taking an action, the environment generates a reward based on the agent's action, and each agent receives a reward. This reward is defined as the sum of the cosine similarity between the content matrix in the cache space and the actual demand matrix, and the user experience quality score, expressed as... :

[0145]

[0146] in, and Represents the weighting parameter.

[0147] The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm comprises four distinct networks: (represented as follows) The Actor network is represented as The Critic network, represented as by The target actor network, and represented as The target critic network, in which , , and These are all neural network weight parameters. Parameter updates during training are performed twice: updating the Critic network and updating the Actor network. When updating the Critic network, a set of samples is first selected. , , and And use the target Actor network to calculate the state. The action taken in the state, i.e. Next, using a target critic network, we compute state-action combinations. , The target value, i.e. The `done` parameter indicates whether training is complete; it is 1 if training is complete, and 0 otherwise. Next, the Critic network is used to calculate the evaluation value of the state-action combination: The gradient descent algorithm is used to reduce the difference between the evaluated and expected values, thereby updating the parameters of the Critic network. When updating the Actor network, the state is first calculated using the Actor network. The action in, namely calculation Next, the Critic network is used to calculate the evaluation value of the state-action combination, i.e., the cumulative expected return. Finally, the gradient ascent algorithm is used to maximize the cumulative expected return.

[0148] Prioritizing Experience Replay: As training progresses, the experience replay region gradually fills with experience data. The data quality in the experience replay region is relatively random, and selecting small batches of samples may lead to performance degradation during model training. To enhance model convergence, a priority experience replay strategy is introduced into multi-agent deep reinforcement learning algorithms. A priority value is established, which is evaluated by the variance between the current and target distributions, denoted as... The goal of training is to reduce... Therefore, the higher the priority value, the greater the difference in the results of the sample, and the more continuous training is needed. The priority value is mapped to a range of 0-1 and regarded as the probability of experience replay.

[0149] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention do not depart from the essence and scope of the technical solution of the present invention.

Claims

1. A 5G cloud-edge collaborative content caching method, characterized in that: Includes the following steps: A prediction accuracy model is constructed based on cosine similarity theory. and cache hit rate The way of expression; According to the user at any time The method for obtaining the requested content and the method for constructing a representation of the overall transmission latency; Construct a representation of the total traffic cost based on the way the overall transmission delay is expressed; Taking into account hit rate, transmission latency, and traffic load, we construct a way to express the user experience and further build a caching model; Split the caching model into and Two parts; The solution is obtained through a cache prediction scheme based on Transformer, deep neural networks, and hyperdimensional computation. ; Solving the problem using deep reinforcement learning methods. ; Represented as: in, To improve prediction accuracy, For cache hit rate, This represents the expected value of the prediction accuracy. This represents the expected value of the prediction accuracy. These are weight parameters; The content that the user actually accesses; For a collection of content; represent A nearby edge node or cloud computing center; Represents edge nodes neighboring nodes and cloud computing center The set consisting of both; Represented as: ; This refers to the user experience read from local storage; in, These are weight parameters; These are weight parameters; For link load; For user experience, For content from Delay in retrieving content; For the time delay of reading from local; right The solution process is as follows: preprocess the data, and use high-dimensional calculation methods to further map the vectorized cached content to a hyperdimensional space; The preprocessed data is used as input, and user preferences, historical information and popular information are extracted by passing through the user preference extraction network, historical information extraction network and popular information extraction network respectively. The user preference extraction network consists of a fully connected neural network with only one layer. The Transformer temporal neural network is used to predict historical and trending information for the next time step. The encoder-decoder structure in the Transformer is used to predict the sequence, and the embedding layers of the encoder and decoder are combined with high-dimensional vectors, where vector operations replace the original floating-point operations. The prediction results are combined to obtain the final content matrix; right The solution process is as follows: define the state space, action space, and reward; Represented as The Actor network is represented as The Critic network, represented as by The target Actor network, and represented as The target Critic network undergoes parameter updates in two stages during training: updating the Critic network and updating the Actor network. When updating the Critic network, first select a set of samples. and And use the target Actor network to calculate the state. The action taken in the state, i.e. ; Using a target Critic network, calculate state-action combinations. , The target value, i.e. The done parameter indicates whether training is complete; it is 1 if training is complete, and 0 otherwise. Calculate the evaluation value of the state-action combination using the Critic network: The gradient descent algorithm is used to reduce the difference between the evaluated value and the expected value, thereby updating the parameters of the Critic network. When updating the Actor network, first use the Actor network to calculate the state. The action in, namely calculation ; The Critic network is used to calculate the evaluation value of the state-action combination, i.e., the cumulative expected return. ; Use the gradient ascent algorithm to maximize the cumulative expected return.

2. The 5G cloud-edge collaborative content caching method according to claim 1, characterized in that: Prediction accuracy The expression is as follows: ; Cache hit rate The expression is as follows: ; In the above formula, for the first... edge nodes , At any moment It predicts the content the user will visit in the next moment. , Users The content actually accessed at any given time is , The cached content after the caching decision is completed is represented as follows: , Represents edge nodes exist The first time cache One item; S represents the total storage capacity of the edge nodes.

3. The 5G cloud-edge collaborative content caching method according to claim 1, characterized in that: User at any time The methods for obtaining the requested content include: Local communication: If edge nodes If the content is already cached locally, then directly... from Transmitted to user ; Proximity communication: If the content requested by the user is on an edge node If there is no local cache, then Try from its adjacent edge nodes Retrieve this content; Cloud communication: If the content requested by the user is not cached on the edge node or its adjacent nodes, then The cache is retrieved from the cloud computing center via a backhaul link.

4. The 5G cloud-edge collaborative content caching method according to claim 2, characterized in that: The overall transmission delay is expressed as: ; in, Indicates transmission delay. represent A nearby edge node or cloud computing center; This indicates that the content is related to... from The delay in retrieving this content; Represents edge server At any moment Received content The number of requests; Indicates at time exist China targets Was the request accepted? If satisfied, then ,otherwise ; Total traffic cost Represented as: ; in, express and The cost of traffic between them.

5. The 5G cloud-edge collaborative content caching method according to claim 4, characterized in that: User experience is represented as: In the formula, express from Get content from Time required; Content The function of utilization; Representatives obtain through direct communication The cost; Representative equipment in Speed ​​of movement in time; Representative equipment in Transmission radius over time; and All of these are communication weight parameters; dt refers to the integration over the communication time; The constraints are satisfied: In the above formula, and The expected value of the prediction accuracy. , , For weight parameters, This is the collection of content cached after the caching decision is completed. , A substitute refers to any element in the set. This represents the time delay when reading from the local machine.

6. The 5G cloud-edge collaborative content caching method according to claim 1, characterized in that: The Transformer temporal neural network is used to predict historical and trending information for the next time step. The encoder-decoder structure in the Transformer is used to predict the sequence, and the embedding layers of the encoder and decoder are combined with high-dimensional vectors, including the following steps: Hot News Prediction Network Historical information extraction network The encoder-decoder structure of Transformer is adopted; For known time series data The predicted samples are split into: and ,in, It is the input data. It is the output target; The encoder's embedding layer is used to generate hyperdimensional vectors, and the vectorized matrix is ​​represented as: The Transformer uses different trigonometric functions for encoding based on whether the position is odd or even, namely: in Indicates position code, Indicates the position of the token. This represents the index of each element in the vector. It is the dimension of the positional encoding vector; Establish three different matrices, each called a query matrix. Keywords Sum Each matrix is ​​obtained through a linear transformation, i.e. The trainable weight matrix is ​​represented as ; The self-attention module in Transformer calculates pairwise similarities between each query and all keys, capturing... The relationships between various vectors within the input are used to capture key information from the input, denoted as... , This includes a total One vector; The output is all values Weighted sum: softmax() is the activation function, where yes , and The dimension; Repeat line execution Next, the relationship between elements at the same position and elements at multiple positions is extracted. The multi-head attention module includes... Each head is equipped with a different trainable weight matrix. ; Input for each head Afterwards, The outputs of the head are combined into a unified matrix. The outputs of multiple attention heads are combined to form the total output; the output of the multi-head attention module is... ,in, These are the weight parameters of the neural network; For popular information The prediction, during the update process, uses complete historical browsing data at the next moment. To calculate the loss and perform gradient updates; for predictions based on historical information, a historical information extraction network is used. The update is its output data and Results from the linear layer: .

7. The 5G cloud-edge collaborative content caching method according to claim 1, characterized in that: Solve During the process, a priority value was established, which was evaluated by the variance between the current and target distributions, denoted as... The goal is to reduce The priority value is mapped to a range of 0-1 and treated as the probability of experience replay.

Images