Low-cost video edge caching method based on federated deep reinforcement learning

Abstract

The application discloses a low-cost video edge caching method based on federal deep reinforcement learning, and belongs to the technical field of edge caching in cloud-edge cooperation. The method comprises the following steps: establishing a network model; a local implicit feedback data is used by a bottom mobile terminal to train a self-encoder, and a federal learning framework is applied by an edge server in an intermediate layer to aggregate local models of the mobile terminal, and a global model is obtained after iteration; prediction is carried out by using the global model; the calculation and communication resources spent in the federal training process are modeled as a comprehensive system cost problem, and the user calculation resource allocation problem is further expressed as a Markov decision process, and under the premise of meeting the resource constraints, the application can effectively allocate the calculation resources of each federal training client, and maximally reduce the total cost of the federal learning process. Experiments show that the application has a better performance in the cache hit rate than the baseline algorithm, and is close to the optimal algorithm, and can effectively reduce the system cost under the condition that the local resources are limited.

Description

TECHNICAL FIELD

[0001] The application relates to a low-cost video edge caching method based on federal deep reinforcement learning, and belongs to the technical field of edge caching in cloud-edge collaboration. BACKGROUND

[0002] With the rapid development of 5G technology and Internet of Things (IoT), the number of mobile devices has increased significantly, which in turn has led to a surge in video traffic. According to the Ericsson Mobility Report in June 2024, 5G mobile subscriptions will approach 5.6 billion, and global mobile data traffic is expected to grow by about three times, reaching 313 EB per month by 2029. Among them, the increase in video content viewing is the main driving force behind the growth of traffic. However, the huge pressure of video transmission may cause network traffic congestion. In particular, with the widespread popularity of 5G and the increase in video applications, the growth rate of video traffic is very fast, requiring greater bandwidth and lower latency to meet the Quality of Service / Experience (QoS / QoE) of mobile users.

[0003] Therefore, Mobile Edge Computing (MEC) is proposed as a new computing paradigm to address the huge resource consumption and time delay problems caused by centralized computing and caching. MEC caches popular video content to edge devices closer to users, usually Base Stations (BS) or IoT devices. Users do not need to obtain content from remote clouds, thereby reducing traffic on backhaul links. MEC-based video caching is an effective way to release traffic load and improve user QoS. Although edge caching has achieved certain results, there are still some specific outstanding problems.

[0004] The first issue is how to determine the popular content to be cached. Since the cache capacity of edge servers (ESs) is limited, it is unrealistic to cache all video contents. Instead, caching popular contents is an effective way to improve cache hit rate. Some early static rule-based caching algorithms do not consider the dynamic changes of popular contents, such as Least Recently Used (LRU), Least Frequently Used (LFU), and First In First Out (FIFO). Since popular contents change over time, these static caching algorithms are no longer applicable. Recently, with the rapid development of artificial intelligence and computing power, more research focuses on machine learning-based popular content prediction methods. By designing a neural network and predicting the movie items that users are likely to request, cache hit rate can be improved. In addition, this method can adapt to dynamically changing cache contents. However, centralized machine learning prediction methods require users to upload local data to cloud servers, which will bring data privacy and security risks. Therefore, many studies use Federated Learning (FL), a distributed machine learning method, to predict dynamically changing popular contents. FL keeps users' local data on the terminal device at all times without uploading to the cloud, helping to protect data privacy. However, most FL-based caching schemes do not consider the overall system cost. The training process of FL requires computing and communication resources, which can lead to a large waste of resources.

[0005] The second and more critical issue is that the energy resources of mobile terminal devices such as tablets and smartphones are not unlimited. Most existing caching schemes mainly focus on how to improve cache efficiency and assume that system resources are large enough. This assumption is not realistic. Terminal devices usually have limited energy and computing resources. An urgent challenge that must be addressed is how to ensure high cache hit rate while reducing system cost in an energy-limited edge cloud environment.

[0006] In addition, users usually only have implicit feedback data without explicit data such as likes, comments, and collection behaviors. The lack of explicit rating data does not mean that users are not interested in videos. Therefore, how to use only implicit feedback data to predict popular contents is also a problem we need to explore.

[0007] In summary, on the one hand, how to obtain popular videos with only user implicit feedback data while addressing user privacy and security issues is a key challenge. On the other hand, how to ensure cache hit rate and reduce system cost in an energy-limited edge cloud system is a key problem that needs to be addressed urgently. SUMMARY

[0008] In order to guarantee cache hit rate and reduce system cost in an energy-limited edge cloud system while ensuring user privacy and security, the application provides a low-cost video edge caching method based on federated deep reinforcement learning, which comprises the following steps:

[0009] Step 1: Establish a network model, including a top layer, an intermediate layer and a bottom layer;

[0010] The top layer is a centralized cloud computing center; the intermediate layer includes a plurality of base stations, each of which is provided with an edge server; the bottom layer includes a plurality of mobile terminals; the centralized cloud computing center is in communication connection with the edge server, and the mobile terminal is in communication connection with the edge server;

[0011] Step 2: The mobile terminal trains a denoising autoencoder model with local implicit feedback data, the edge server of the intermediate layer aggregates local models from the mobile terminal devices at the bottom layer using a federated learning framework, and after multiple iterations, a global model is obtained for completing popular video caching;

[0012] Step 3: Use the global model to predict the movies that users are most likely to access;

[0013] Step 4: Model all the computing and communication resources spent in the federated training process as a comprehensive system cost problem, and further express the user computing resource allocation problem as a Markov decision process to minimize the expected long-term system cost; under the premise of meeting the resource constraints, train the DDQN model to allocate the computing resources of each mobile terminal.

[0014] Optionally, the step 1 comprises:

[0015] Define the users within the coverage range of each edge server as a set C = {c1, c2,..., c m}, wherein c i ∈C; the edge server provides video services for users within the coverage range and accesses the centralized cloud computing center through a backhaul link; describe the communication operation timeline of the mobile terminal and the base station as time slots ε = {1, 2,..., K G};

[0016] Define the key factors affecting system performance: the uplink transmission delay of the mobile terminal to the base station, the downlink transmission delay of the base station to the mobile device and the energy consumption of the mobile terminal to transmit data;

[0017] The uplink transmission delay is represented as:

[0018]

[0019] in, and These represent the local model parameters and their size, respectively. The base station is assigned to mobile user c i The channel bandwidth meets the constraints. It is the channel power gain. It is a mobile device c i The transmission power, where N0 represents the Gaussian white noise power;

[0020] The downlink transmission delay is expressed as:

[0021]

[0022] in, and These represent the global model parameters and their size, respectively. It is an edge server e i Channel bandwidth, It is the channel power gain. This refers to the transmission power, where N0 represents the Gaussian white noise power.

[0023] The energy consumption for data transmission by the mobile terminal is expressed as follows:

[0024]

[0025] Where γ1 is the transmission energy consumption coefficient;

[0026] Mobile terminal c i K is performed in the k-th iteration L The computational latency caused by training the local model in round 1 is:

[0027]

[0028] in, For the dataset size, Indicates mobile terminal c i CPU / GPU cycle frequency for executing sample data Indicates mobile terminal c i The number of CPU / GPU cycles required to execute the sample data;

[0029] The energy consumed by the mobile terminal during local training is defined as:

[0030]

[0031] Where γ2 is the computational energy consumption coefficient relative to the chip architecture.

[0032] Optionally, the step 2 comprises: using federated learning to jointly complete a model training target, i.e., minimizing a global model loss function:

[0033] w * = argmin F(w)

[0034] wherein the process of federated learning comprises:

[0035] Step 21: After the user downloads the global model parameters w, local training is performed, and each client has a training task, and the optimization objective of the local client is:

[0036]

[0037] wherein F i (w) is a local loss function, is a prediction loss;

[0038] The input of the network is a user-item interaction matrix R M×N , let y ∈ R M×N , represent the implicit feedback behavior of whether the user has a request for the video, and the specific definition is:

[0039]

[0040] wherein u represents a user, i represents a video; y u,i = 1 represents that the user has an implicit feedback behavior of requesting the video, and y u,i = 0 represents that there is no any request behavior for the video, which does not mean that the user is not interested in the video, but may represent that the user has not found the existence of the video i;

[0041] Step 22: After the user completes the local training, the local model parameters are uploaded to the edge server;

[0042] Step 23: After the edge server receives the local model parameters, aggregation is performed, and the parameter aggregation rule is:

[0043]

[0044] Step 24: After the edge server obtains the global model, the global model parameters are further distributed to the user, and the user continues the step of step 21.

[0045] Optionally, the step 3 comprises:

[0046] Each mobile terminal uses the trained global model to perform content recommendation, and the output is the probability size of each movie being requested and is uploaded to the server;

[0047] The first H users similar to the active user are obtained by using the similarity, and the movie list clicked by the first H users is taken as the recommendation;

[0048] The movie recommended by the auto-encoder and the movie recommended by the similar user are combined, and the first N movies are taken as the movie recommended by the mobile terminal according to the number of recommended movies.

[0049] Optionally, the step 4 comprises: defining a state space S, an action space A and a reward function R in the DDQN model; the state in the state space S is represented as: s t ∈S, Wherein Π is a Cartesian product, and The state vector combination of the mobile user c at t time is represented as: s The delay, energy and bandwidth of the user c at the current t time are represented as: T(t), Ω(t) and B(t) respectively;

[0050] The t round action in the action space A is represented as: a t ∈A, Wherein The CPU frequency allocation set is represented as: F;

[0051] The reward function R is defined as: r t =-σT(t)-ιΩ(t), wherein σ and ι are two normal numbers, the range is 0 to 1, and T(t) and Ω(t) represent the delay and energy consumption caused by calculation and communication respectively;

[0052] The discount return is used as the total return R t of all rewards from the current time to the end of this round:

[0053]

[0054] Wherein, γ is a discount factor, and γ ∈ (0, 1].

[0055] Optionally, the delay generated by one round of federated learning is the sum of the communication delay and the calculation delay, and is represented as:

[0056]

[0057] The total energy consumption generated by one round of federated learning is:

[0058]

[0059] The total cost of the system is defined as: the weighted sum of the time and energy consumption caused by calculation and communication, and is represented as:

[0060]

[0061] The problem of reasonably allocating computing resources under the energy consumption limit to achieve the minimization of the total system cost is described as follows:

[0062]

[0063] C3:F(ω)<κ

[0064] wherein T max and Ω max are constants; C1 restricts the CPU / GPU frequency allocated to the mobile terminal; C2 restricts that the total energy consumption cannot be higher than an upper limit value; and C3 restricts that the global loss function is less than κ.

[0065] Optionally, one round of federated learning comprises:

[0066] (1) the base station initiates a training task, and transmits a global model to the mobile terminal through a downlink transmission link;

[0067] (2) the mobile terminal trains a local model locally;

[0068] (3) the mobile terminal uploads the trained local model to the base station through an uplink transmission link.

[0069] Optionally, step 3 uses cosine similarity to obtain the top H users similar to the active user.

[0070] Optionally, the state-action value function in the DDQN model is:

[0071]

[0072] wherein Q' and Q networks are two neural networks of the same structure, and r t represents the immediate reward of the current state.

[0073] Optionally, the mobile terminal comprises a smartphone, a smart wearable device, a smart tablet and a portable computer.

[0074] The present application has the following advantages:

[0075] The present application proposes a caching scheme based on federated reinforcement learning and user implicit feedback, which protects user data privacy and does not require explicit rating data, but only needs to use implicit feedback data for federated training. Each client trains a denoising autoencoder model together, and the edge server aggregates the obtained model at the last round of communication to obtain a trained model, which completes the popular video caching. In addition, the scheme not only improves the cache hit rate, but also reduces the system cost based on the resource allocation scheme of reinforcement learning.

[0076] The present application models all the computing and communication resource costs in the federated training process as a system-wide cost, and further formulates the user computing resource allocation problem as a Markov Decision Process (MDP) to minimize the expected long-term system cost (reflecting the user's QoE). Under the condition of meeting resource constraints, a resource allocation method based on DDQN is proposed to solve the optimal resource allocation strategy, which can effectively allocate the computing resources of each federated training client to minimize the total cost in the federated learning process.

[0077] Through verification on real data sets, experiments show that the performance of the present application in cache hit rate is higher than that of the baseline algorithm and close to the Oracle, i.e., the optimal algorithm; in addition, experiments also show that the active video caching method of the present application can effectively reduce the system cost under the condition of limited local resources. BRIEF DESCRIPTION OF DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0079] Figure 1 The technical process diagram of the video edge caching method based on federated deep reinforcement learning of the present application.

[0080] Figure 2 The video caching process diagram of the present application.

[0081] Figure 3 The comparison diagram of the cache hit rate of the caching method of the present application and other four algorithms when the cache capacity is 50-500.

[0082] Figure 4 The comparison diagram of the average system cost of the caching method of the present application and other three algorithms.

[0083] Figure 5 The network model structure diagram of the caching method of the present application.

[0084] Figure 6 The pseudo code screenshot of the DDQN resource allocation algorithm in the embodiment of the present application.

[0085] Figure 7 The pseudo code screenshot of the video caching algorithm based on federated learning in the embodiment of the present application.

[0086] Figure 8 The flowchart of the DDQN resource allocation algorithm in the embodiment of the present application. DETAILED DESCRIPTION

[0087] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0088] Embodiment one:

[0089] The embodiment provides a video edge caching method based on federal deep reinforcement learning, which comprises the following steps:

[0090] Step 1: Establish a network model, including a top layer, an intermediate layer and a bottom layer.

[0091] The top layer is a centralized cloud computing center; the intermediate layer comprises a plurality of base stations, each of which is provided with an edge server, and the bottom layer comprises a plurality of mobile terminals; the centralized cloud computing center and the edge server are in communication connection, and the mobile terminal and the edge server are in communication connection.

[0092] Step 2: The mobile terminal trains the denoising autoencoder model with local implicit feedback data, the edge server of the intermediate layer aggregates the local model from the mobile terminal device of the bottom layer by using the federal learning framework, and the global model is obtained after multiple iterations, which is used for completing the popular video caching;

[0093] Step 3: Use the global model to predict the movies that the user is most likely to access.

[0094] Step 4: Model all the computing and communication resources spent in the federal training process as a comprehensive system cost problem, and further express the user computing resource allocation problem as a Markov decision process to minimize the expected long-term system cost; under the premise of meeting the resource constraints, train the DDQN model to allocate the computing resources of each mobile terminal.

[0095] Embodiment two:

[0096] The embodiment provides a low-cost video edge caching method based on federal deep reinforcement learning, which uses a three-layer network architecture to simulate a real urban scene, including a top layer, an intermediate layer and a bottom layer. The top layer is also called cloud, the intermediate layer is also called edge, and the bottom layer is also called terminal. The method specifically comprises the following steps:

[0097] Step 1: Establish a network model, including a top layer, an intermediate layer and a bottom layer.

[0098] The top layer is a centralized cloud computing center with unlimited computing and storage resources; the middle layer contains a plurality of small base stations (SBSs), and the SBSs have edge servers (ESs) with computing and storage resources, such as Nvidia Jetson TX2; and the bottom layer is composed of a plurality of mobile users, and each user's smart mobile device, such as a smartphone, a smart wearable device, a smart tablet, a laptop, and the like, is equipped with a chip capable of running a lightweight neural network.

[0099] The cloud computing center is connected with the edge end for network connection and data transmission through a backhaul link, and the mobile user / terminal device is connected with the SBS / edge server for connection and communication through a cellular network (also referred to as a mobile network, such as 3G, 4G, 5G, and the like).

[0100] Step 2: The bottom layer mobile terminal device jointly trains a denoising autoencoder model using local implicit feedback data, the edge server in the middle layer aggregates the local model from the bottom layer mobile terminal device using a federated learning framework, and after multiple iterations, a global model is obtained, which is used to complete popular video caching;

[0101] Step 3: The global model is used to predict the movies that the user is most likely to access, and after the autoencoder model, the output is the probability of each movie being requested. The top N movies are taken as the recommended movies of the user. In addition, since the autoencoder has the characteristics of exploring the potential features of data, the movie list of similar users is also taken as the recommended movie of the user according to the collaborative filtering rule.

[0102] Step 4: All computing and communication resources spent in the federated training process are modeled as a comprehensive system cost problem, and further, the user computing resource allocation problem is expressed as a Markov decision process (MDP) to minimize the expected long-term system cost (reflecting the user's QoE). Under the premise of meeting the resource constraints, a resource allocation method based on DDQN is used to solve the optimal resource allocation strategy, effectively allocate the computing resources of each federated training client, and maximize the total cost of the federated learning process.

[0103] The step 1 specifically includes:

[0104] Each ES covers a user in a set C = {c1, c2,..., c m}, where c i ∈ C. The ES can provide video services for users in different geographical locations within its coverage range, and can also access the cloud center through a backhaul link. The communication operation timeline of the mobile user and the SBS is described as a time slot ε = {1, 2,..., KG}。

[0105] In the communication model, the embodiment considers three key factors that affect the system performance: the uplink transmission delay of mobile devices transmitting local models to SBS, the downlink transmission delay of SBS delivering global models to mobile devices, and the energy consumption of mobile devices transmitting data.

[0106] Within the coverage of SBS, mobile users communicate with ES through a wireless transmission link. The embodiment assumes that the total bandwidth provided by SBS to mobile devices is When mobile users train the local model through ξ local iterations and upload it to ES, they communicate with ES through the uplink transmission link. According to the Shannon formula, the maximum uplink transmission rate of mobile users c i is:

[0107]

[0108] wherein, is the channel bandwidth allocated by SBS to mobile user c i , satisfying the constraint is the channel power gain, is the transmission power of mobile device c i , and N0 represents the Gaussian white noise power.

[0109] The uplink transmission delay in the εth round of communication is:

[0110]

[0111] wherein, and represent the local model parameters and their size, respectively.

[0112] When mobile users issue a video request and the requested video is cached on ES, ES communicates with mobile users through the downlink transmission link and transmits the cached video content to the mobile user terminal. In addition, ES also delivers the global model to the mobile user terminal through the downlink transmission link. Mobile user c i communicates with ES e i The maximum downlink transmission rate of mobile users transmitting data to ES is:

[0113]

[0114] wherein, is the channel bandwidth of ES e i , is the channel power gain, is the transmission power, and N0 represents the Gaussian white noise power.

[0115] At the εth round of communication, the downlink transmission delay is:

[0116]

[0117] wherein, and respectively represent the global model parameters and the size thereof.

[0118] The mobile user equipment transmits data through the uplink, which consumes energy. In the embodiment, the transmission energy consumption of the mobile user equipment c i is defined as:

[0119]

[0120] wherein γ1 is a transmission energy consumption coefficient.

[0121] Since the SBS has sufficient resources, the embodiment assumes that the transmission energy consumption of the ES e i has little impact on the system, and therefore, only the transmission energy consumption of the mobile equipment is considered in the model of the embodiment.

[0122] Modern intelligent communication equipment is mostly equipped with a system-level chip (SoC) with computing power, which can train a lightweight neural network model. When training a local model on a mobile user equipment, computing delay and energy consumption are generated.

[0123] For each client , the CPU / GPU cycle number (CPU / GPU cycles) and the CPU / GPU cycle frequency (CPU / GPU cycles frequency) of the sample data executed by the client are respectively and The CPU / GPU cycle frequency represents the local computing power, and it is assumed that the CPU / GPU cycle frequency is divided into l levels, i.e. The mobile user c i performs K L rounds of local model training in the kth iteration, and the computing delay generated by the K rounds of local model training is:

[0124]

[0125] wherein, is the size of the data set.

[0126] Considering that the SBS has enough computing resources, the embodiment assumes that the computing energy consumption of the SBS has little effect on the entire system, and therefore, the embodiment only considers the energy consumed by the mobile device for local training, which is defined as:

[0127]

[0128] wherein γ2 is a computing energy consumption coefficient relative to the chip architecture.

[0129] Step 2 of the embodiment includes using federated learning to jointly complete a model training target, i.e., minimizing a global model loss function:

[0130] w * = argmin F(w)

[0131] wherein the process of federated learning includes:

[0132] Step 21: First, the user downloads the global model parameters w and then performs local training. Each client has a training task, and the optimization goal of the local client is:

[0133]

[0134] wherein F i (w) is a local loss function, and the prediction loss.

[0135] The input of the network is a user-item interaction matrix R M×N . Let y ∈ R M×N , represent the implicit feedback behavior of whether the user has a request for the video, which is specifically defined as:

[0136]

[0137] wherein u represents the user, and i represents the video; y u,i = 1 represents the implicit feedback behavior of the user having a request for the video, and y u,i = 0 represents no request behavior for the video, which does not mean that the user is not interested in the video, but may mean that the user has not yet discovered the existence of the video i. If the number of evaluations of the user-item matrix is not less than 4, the corresponding element of the matrix is 1, otherwise, it is 0.

[0138] The input of the denoising autoencoder is not complete data, but is partial input data. This can enable the hidden layer to discover more robust features, rather than learning too simple features. The embodiment sets the value of some tensors to 0 at random through dropout.

[0139] Since the model trained in this embodiment is an autoencoder network model, the learning goal is whether the video will be requested in the future (1 if requested, 0 if not requested), so it is a binary classification problem. This embodiment adds a sigmoid activation function in the output layer to convert the output of the hidden layer into a probability between 0 and 1. The output layer is whether the user will request the video, that is, a binary classification task. Therefore, the loss function is a binary cross entropy loss function:

[0140]

[0141] where

[0142] The autoencoder model of this embodiment encodes not only the user's interaction behavior with the video (user-item matrix) in the input layer, but also considers the user's features, that is, the user's features are also encoded to obtain the hidden layer, which can make the semantics more rich. The autoencoder model used in this embodiment is a denoising autoencoder, so this embodiment adds noise to the training data through dropout, but restores the original clean data when restoring the data, which is done to make the model more robust.

[0143] Step 22: After the user trains locally, upload the local model parameters to the edge server

[0144] Step 23: After the edge server receives the local model parameters, it will be aggregated. Here, the parameter aggregation rule is:

[0145]

[0146] Step 24: After the edge server obtains the global model, the global model parameters are further distributed to the user, and the user will continue to perform the steps of step 21.

[0147] From the foregoing description, a complete FL communication process is:

[0148] 1. SBS initiates a training task, that is, the global model is downloaded to the mobile device end through the downlink transmission link;

[0149] 2. The mobile device calculates the local model;

[0150] 3. The mobile device uploads the trained local model to the SBS through the uplink transmission link.

[0151] After a round of FL communication, since each mobile user c iThe time spent on training local models varies with different computing resources. SBS starts model aggregation only after receiving all local models. Therefore, the delay of one round of FL communication is the sum of communication delay and computing delay:

[0152]

[0153] The total energy consumption of one round of FL communication is:

[0154]

[0155] The total cost of the system is defined as the weighted sum of computing and communication resources, i.e., the weighted sum of time and energy consumption due to computing and communication:

[0156]

[0157] How to reasonably allocate computing resources to minimize the total cost of the system under the energy consumption constraint is a key problem. This embodiment describes the problem as follows:

[0158]

[0159] C3: F(ω) < κ

[0160] Since the units of time and energy consumption are different, the cost of the entire system cannot be defined as the sum of time and energy consumption. Therefore, this embodiment defines the system cost as the sum of the proportion of time consumption and the proportion of energy consumption. max and Ω max are constants.

[0161] C1 restricts the CPU / GPU frequency allocated to mobile users to be greater than the minimum value that can be provided and less than the maximum CPU / GPU frequency that can be provided. C2 restricts the total energy consumption to be no higher than the upper limit. C3 restricts the global loss function to be less than κ, ensuring the training quality.

[0162] Step 3 of this embodiment includes:

[0163] After the training is completed, each client uses the local data to make user popular content recommendations according to the global model, and uploads to the server. When each client makes a prediction, the input is the existing training set, and the training set data is not destroyed, that is, no noise is added. After passing through the autoencoder model, the output is the probability size of each movie being requested. The top N movies in this embodiment are taken as the recommended movies of the user. In addition, since the autoencoder has the characteristics of exploring the potential features of data, this embodiment also takes the movie list of similar users as the recommended movies of the user according to the collaborative filtering rule. The specific process is that this embodiment uses the hidden layer of the autoencoder to find the top H users similar to the active user by using the cosine similarity, and takes the movie list clicked by the similar users as the recommendation. The active user is defined as a user who has more video interaction behaviors. The cosine similarity formula is as follows:

[0164]

[0165] The movies recommended by the autoencoder and the movies recommended by similar users are combined, and the top top-N movies are taken as the recommended movies of the client according to the number of recommended movies of each movie. The server sorts the recommended results of each client according to the number of recommended movies of each movie, and caches the top N movies according to the cache capacity.

[0166] Step 4 of this embodiment includes:

[0167] The purpose of this embodiment is to minimize the system cost caused by federated learning in an edge cloud system with energy constraints. This embodiment models this problem as a Markov Decision Process (MDP) problem. The state space and action space are discrete. Next, this embodiment describes the state space S, action space A and reward function R in the DDQN model respectively.

[0168] State space S: The dynamic of network bandwidth in the mobile edge network scenario of this embodiment conforms to the rules of the actual scenario. However, the network will not change dramatically in a short time, for example, tens of seconds. According to the above optimization formula, there are many factors that affect the system cost, such as delay, consumed energy and network bandwidth, etc., and the network bandwidth is related to the historical bandwidth. Therefore, the current state space of the DDQN model in this embodiment is set as where Π is the Cartesian product, and represents the state vector combination of mobile user c at time t, where respectively represent the delay, energy and bandwidth of user c at the current time t. Since S is a state space set, s t ∈ S.

[0169] Action space A: The action of the DDQN agent is to allocate CPU resources. CPU frequency θ c For discrete, is the set of CPU frequency allocation. The action at the t-th round is defined as a t , This embodiment defines the action space as A, then a t ∈ A.

[0170] Reward function R: Since the goal of this embodiment is to minimize the system cost under the energy consumption limit, this embodiment defines the immediate reward function to be inversely proportional to the delay and energy consumption: r t =-σT(t)-ιΩ(t), where σ and ι are two positive numbers, ranging from 0 to 1. Since different federated learning tasks have different preferences for learning time and energy efficiency, this embodiment defines two non-negative parameters σ and ι to adjust the preferences in the objective function.

[0171] This embodiment uses the discounted return as the total return R t of all rewards from the current time to the end of this episode:

[0172]

[0173] where γ is the discount factor, γ ∈ (0, 1].

[0174] What action to take in the process involves the policy. This embodiment gives a certain policy π, then a = π(s), which means that the input state s outputs the action a. (This policy π summarizes how to choose behavior in each state. Therefore, the policy π is a set-like thing.) π(a|s) = P(A t =a|S t =s) represents the probability of an agent taking a certain action a given an input state s. The expected return of a state is called the value of the state, and the values of all states form the value function. Under the policy π, this embodiment gives the expected long-term training reward state-action value function as:

[0175]

[0176] By Bellman equation, we can derive:

[0177] Q π (s,a) = E π [r t + γR t+1 |S t =s,A t =a]

[0178] = E π [ r t + γQ π (S t+1 , A t+1 )|S t = s, A t = a]

[0179] This state-action value function is the Q-value function, which represents the expected return of an agent performing an action a in a state s according to a specific policy π, i.e., the maximum possible benefit at the end of the episode.

[0180] The optimal policy π ＊ will be a policy that maximizes the state-action value function, which is obtained by the following optimal objective function:

[0181]

[0182] Therefore, the optimal state-action value function Q * (s, a) is:

[0183]

[0184] That is, the optimal Q-value of the current state and action is equal to the immediate reward r t plus the maximum Q-value of the next state.

[0185] Deep reinforcement learning (DRL) is a popular algorithm used to solve MDP problems, especially when the state space and action space are very large. Since the action of allocating computing resources is discrete, the value-based DQN algorithm is selected in this embodiment. In previous studies, it was found that DQN overestimates the action value, and double deep Q-network (DDQN) can solve the problem of overestimating the action value. Therefore, the DDQN method is selected in this embodiment, which changes the state-action value function in DQN to:

[0186]

[0187] where Q' and Q networks are two neural networks of the same structure.

[0188] In addition, considering the limited available tracking of FL, DDQN is more effective in reusing data than policy gradient methods and actor-critic methods. This part details the neural network training function Q(s, a; ω) of the DDQN-Based Agent to approach Q* (s, a) find the optimal strategy p ＊ . The target value of Q(s, a; w) is And the estimated value is Q(s t , a; w). Therefore, the loss function Loss = MSE(y t , Q(s t , a; w)). Wherein, the pseudo code of DDQN resource allocation algorithm and video caching algorithm based on federated learning are shown in Figure 6 and Figure 7 .

[0189] The flow chart of DDQN resource allocation algorithm is shown in Figure 8 , including the following steps:

[0190] S1: Get the experience replay memory pool size D, initial network weight parameters, regularization coefficient, learning rate, discount factor, maximum resource amount, Q network, target network;

[0191] S2: Randomly initialize the weights of the Q network and the target network;

[0192] S3: Observe the current state s;

[0193] S4: Select action according to greedy strategy;

[0194] S5: Store the converted state;

[0195] S6: Determine whether step exceeds D, if yes, perform network learning and update parameters, and empty the replay memory pool size D;

[0196] S7: Determine whether the precision of FL reaches a or the remaining resource amount is less than 0, if yes, terminate;

[0197] S8: Go back to S3 and continue to execute.

[0198] The video caching algorithm based on federated learning includes the following steps:

[0199] S1: The client and the server establish communication;

[0200] S2: Each client downloads the global model from the server and performs local training;

[0201] S3: All clients upload the local model to the server;

[0202] S4: The server updates the global model;

[0203] S5: Input the current state into the DDQN model, and output the CPU frequency allocated to each client;

[0204] S6: send the allocated CPU frequency to the client;

[0205] S7: go back to S1 and continue.

[0206] The simulation experiment of the cache method of the present application is as follows:

[0207] Figure 3 The comparison diagram of the cache method of the present application and other four algorithms in the cache hit rate when the cache capacity is 50-500 is described. The cache hit rate result increases with the increase of the cache size, and the curve shows an upward trend. This is because the larger the cache space is, the more it can store a few or less popular videos for different user requests. Oracle shows the best cache result because Oracle knows all future user requests in advance. FRPVC and c-ε-greedy both learn from the past implicit feedback data, so the performance is better than TS and Random. The cache performance of FRPVC is closest to Oracle because FRPVC uses a denoising autoencoder to learn the damaged user implicit feedback data and make video recommendations. In addition, FRPVC also uses the collaborative filtering and data feature extraction characteristics of the autoencoder to use the videos requested by similar users to make video recommendations. Therefore, the cache hit rate of FRPVC is high. Since the videos are randomly selected from all the films for caching, the cache hit rate of Random is the worst.

[0208] Figure 4 The comparison diagram of the average system cost of the cache method of the present application and other three algorithms is described. In order to intuitively compare, we selected two comparison algorithms, Heuristic and Static. In Heuristic, since the bandwidth information of the mobile device in each iteration of federated learning can be known, the CPU cycle frequency of the current round can be determined according to this information. The Static algorithm assumes that the network is static, and a unified CPU cycle frequency is used during each iteration of training. The CPU cycle frequency used in the Static algorithm is determined by taking the average value according to the data in the network bandwidth data set. It can be found that the DRL method obtains the best performance.

[0209] In particular, the average system cost of the DDQN method is 0.24, while the average system cost of the Heuristic and Static methods is 0.67 and 0.73, respectively. This is due to the learning ability of the DDQN method, which can obtain experience from previous episodes to allocate the most appropriate CPU frequency. Although the Heuristic algorithm can obtain some experience from the bandwidth information in the last round, it cannot guarantee the lowest long-term average system energy consumption. Static always uses the same CPU frequency during the FL training round, resulting in the highest system energy consumption and time.

[0210] Part of the steps in the embodiments of the present application can be implemented by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk.

[0211] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A low-cost video edge caching method based on federated deep reinforcement learning, characterized in that, The method includes: Step 1: Build a network model, including top layer, middle layer, and bottom layer; The top layer is a centralized cloud computing center; the middle layer contains multiple base stations, each of which is equipped with an edge server; the bottom layer includes multiple mobile terminals; the centralized cloud computing center and the edge servers are connected in communication; the mobile terminals and the edge servers are also connected in communication. Step 2: The mobile terminal uses local implicit feedback data to jointly train the denoising autoencoder model. The edge server of the intermediate layer uses a federated learning framework to aggregate local models from the underlying mobile terminal devices. After multiple iterations, a global model is obtained to complete the caching of popular videos. Step 3: Use the global model to predict the movies that the user is most likely to visit; Step 4: Model all the computational and communication resources spent in the federated training process as a comprehensive system cost problem, and further formulate the user computational resource allocation problem as a Markov decision process to minimize the expected long-term system cost; under the premise of satisfying resource constraints, train the DDQN model to allocate the computational resources of each mobile terminal.

2. The video edge caching method according to claim 1, characterized in that, Step 1 includes: Define the users within the coverage area of ​​each edge server as a set C = {c1, c2, ..., c...} m }, where c i ∈C; The edge server provides video services to users within its coverage area and accesses the centralized cloud computing center via a backhaul link; The communication operation timeline between the mobile terminal and the base station is described as a time slot ε={1,2,...,K G }; The key factors affecting system performance are defined as follows: the uplink transmission delay of the mobile terminal transmitting the local model to the base station, the downlink transmission delay of the base station sending the global model to the mobile device, and the energy consumption of the mobile terminal transmitting data. The uplink transmission delay is expressed as: in, and These represent the local model parameters and their size, respectively. The base station is assigned to mobile user c i The channel bandwidth meets the constraints. It is the channel power gain. It is a mobile device c i The transmission power, where N0 represents the Gaussian white noise power; The downlink transmission delay is expressed as: in, and These represent the global model parameters and their size, respectively. It is an edge server e i Channel bandwidth, It is the channel power gain. This refers to the transmission power, where N0 represents the Gaussian white noise power. The energy consumption for data transmission by the mobile terminal is expressed as follows: Wherein, γ1 is the transmission energy consumption coefficient; Mobile terminal c i K is performed in the k-th iteration L The computational latency caused by training the local model in round 1 is: in, For the dataset size, Indicates mobile terminal c i CPU / GPU cycle frequency for executing sample data Indicates mobile terminal c i The number of CPU / GPU cycles required to execute the sample data; The energy consumed by the mobile terminal during local training is defined as: Where γ2 is the computational energy consumption coefficient relative to the chip architecture.

3. The video edge caching method according to claim 2, characterized in that, Step 2 includes: using federated learning to jointly achieve a model training objective, namely, minimizing the global model loss function. w * =argmin F(w) The federated learning process includes: Step 21: First, the user downloads the global model parameters w and then performs local training. Each client has a training task, and the optimization objective for the local client is: Among them, F i (w) is the local loss function. To predict losses; The network input is the user-item interaction matrix R. M×N Let y∈R M×N This indicates an implicit feedback behavior indicating whether the user has requested a video, specifically defined as: Where u represents the user, i represents the video; y u,i =1 indicates that the user has implicitly requested the video. u,i =0 indicates that there was no request for the video. This does not mean that the user is not interested in the video, but may also mean that the user has not yet discovered the existence of video i. Step 22: After the user has completed local training, they will upload the local model parameters. To the edge server; Step 23: After receiving the local model parameters, the edge server will perform aggregation. The parameter aggregation rules are as follows: Step 24: After the edge server obtains the global model, it continues to distribute the global model parameters to the users, and the users will continue to step 21.

4. The video edge caching method according to claim 3, characterized in that, Step 3 includes: Each mobile device uses a pre-trained global model to recommend content, outputting the probability of each movie being requested and uploading it to the server; By utilizing the hidden layer of the autoencoder, we obtain the top H users who are similar to active users based on similarity, and use their clicked movie list as recommendations. The movies recommended by the autoencoder and the movies recommended by similar users are merged. The top N movies recommended by each movie are selected as the movies recommended by the mobile terminal and cached.

5. The video edge caching method according to claim 4, characterized in that, Step 4 includes: defining the state space S, action space A, and reward function R in the DDQN model; The states in the state space S are represented as: s t ∈S, Where Π is the Cartesian product, and Let represent the combination of state vectors of user c moving at time t, where Let represent the delay, energy, and bandwidth of user c at the current time t, respectively. The action in the action space A in round t is represented as a. t ∈A, in Represents the CPU frequency allocation set; The reward function R is defined as an instantaneous reward function that is inversely proportional to latency and energy consumption: r t = -σT(t)-ιΩ(t), where σ and ι are two positive constants ranging from 0 to 1, and T(t) and Ω(t) represent the delay and energy consumption caused by computation and communication, respectively; Use the discounted reward as the total reward R of all rewards from the current moment until the end of this round. t : Where γ is the loss factor, γ∈(0,1).

6. The video edge caching method according to claim 2, characterized in that, The latency incurred by a round of federated learning is the sum of communication latency and computation latency, expressed as: The total energy consumption of one round of federated learning is: The total system cost is defined as the weighted sum of time and energy consumption incurred due to computation and communication, expressed as: The problem of rationally allocating computing resources under energy consumption constraints to minimize the total system cost is described as follows: C3:F(ω)<κ Among them, T max and Ω max C1 is a constant; C2 constrains the CPU / GPU frequency allocated to the mobile terminal; C3 constrains the total energy consumption to not exceed the upper limit; and C3 constrains the global loss function to be less than κ.

7. The video edge caching method according to claim 1, characterized in that, One round of federal learning includes: (1) The base station initiates a training task and sends the global model to the mobile terminal through the downlink transmission link; (2) The mobile terminal is trained using a local model; (3) The mobile terminal uploads the trained local model to the base station via the uplink transmission link.

8. The video edge caching method according to claim 4, characterized in that, Step 3 uses cosine similarity to obtain the top H users who are similar to the active users.

9. The video edge caching method according to claim 4, characterized in that, The state-action value function in the DDQN model is: Among them, Q' and Q networks are two neural networks with the same structure, r t Indicates the immediate reward for the current state.

10. The video edge caching method according to claim 1, characterized in that, The mobile terminals include: smartphones, smart wearable devices, smart tablets, and portable computers.

Images