Low-cost video edge caching method based on federal deep reinforcement learning

Abstract

The invention discloses a low-cost video edge caching method based on federal deep reinforcement learning, and belongs to the technical field of edge caching of cloud edge collaboration. The method comprises the following steps: establishing a network model; the bottom-layer mobile terminal uses local implicit feedback data to train an auto-encoder, the edge server of the middle layer uses a federated learning framework to aggregate local models of the mobile terminal, and a global model is obtained after iteration; predicting by using the global model; according to the method, calculation and communication resources consumed in the federation training process are modeled as a comprehensive system cost problem, and a user calculation resource allocation problem is further expressed as a Markov decision process, so that the calculation resources of each federation training client can be effectively allocated on the premise of meeting resource constraints, and the resource allocation efficiency is improved. And the total cost of the federal learning process is reduced to the greatest extent. Experiments show that the cache hit rate of the method is higher than that of a baseline algorithm and is close to that of an optimal algorithm, and the system cost can be effectively reduced under the condition that local resources are limited.

Description

Technical Field

[0001] The present invention relates to a low-cost video edge caching method based on federated deep reinforcement learning, and belongs to the technical field of edge caching in cloud-edge collaboration. Background Art

[0002] With the rapid development of 5G technology and the Internet of Things (IoT), the number of mobile devices has increased significantly, leading to a surge in video traffic. According to the Ericsson Mobility Report in June 2024, 5G mobile subscription users will be close to 5.6 billion, and global mobile data traffic is expected to increase by about three times, reaching 313EB per month by 2029. Among them, the increase in video content viewing is the main driver of traffic growth. However, the huge pressure of video transmission may cause network traffic congestion. In particular, with the widespread popularity of 5G and the increase in video applications, video traffic is growing very fast, requiring larger bandwidth and lower latency to meet the quality of service / quality of experience (QoS / QoE) of mobile users.

[0003] Therefore, Mobile Edge Computing (MEC) has been proposed as an emerging computing paradigm to deal with the huge resource consumption and time delay problems caused by centralized computing and caching. MEC caches popular video content to edge devices closer to users, usually base stations (BS) or IoT devices. Users do not need to obtain content from the remote cloud, thereby reducing the traffic on the backhaul link. MEC-based video caching is an effective way to release traffic load and improve user QoS. Although edge caching has achieved certain results, there are still some specific outstanding issues.

[0004] The first question is how to determine the popular content to cache. Due to the limited cache capacity of the edge server (ES), it is unrealistic to cache all video content. Instead, caching popular content is an effective way to improve the cache hit rate. Some early static rule caching algorithms do not consider the dynamic changes of popular content, such as Least Recently Used (LRU), Least Frequently Used (LFU), and First In First Out (FIFO). Since popular content changes over time, these static caching algorithms are no longer applicable. Recently, with the rapid development of artificial intelligence and computing power, more research has focused on popular content prediction methods based on machine learning. By designing neural networks and predicting the movie entries that users may request, the cache hit rate can be improved. In addition, this method is able to adapt to dynamically changing cached content. However, centralized machine learning prediction methods require users to upload local data to cloud servers, which will bring data privacy and security risks. Therefore, many studies adopt Federated Learning (FL), a distributed machine learning method, to predict dynamically changing popular content. FL enables users' local data to always be stored on the terminal device without uploading it to the cloud, which helps protect data privacy. However, most FL-based caching schemes do not consider the overall system cost. The training process of FL requires computing and communication resources, which may lead to a lot of resource waste.

[0005] The second more critical issue is that the energy resources of mobile terminal devices (such as tablets and smartphones) are not unlimited. Most existing cache schemes focus on how to improve cache efficiency and assume that the system resources are large enough. This assumption is not realistic. Terminal devices usually have limited energy and computing resources. An urgent challenge that must be solved is how to ensure a high cache hit rate in an energy-limited edge cloud environment while reducing system costs.

[0006] In addition, users usually only have implicit feedback data but no explicit data, such as likes, comments, and favorites. The lack of explicit rating data does not mean that the user is not interested in the video. Therefore, how to use only implicit feedback data to predict popular content is also a problem we need to explore.

[0007] In summary, on the one hand, how to obtain popular videos with only implicit user feedback data while solving user privacy and security issues is a key challenge. On the other hand, how to ensure cache hit rate and reduce system cost in energy-constrained edge cloud systems is a key problem that needs to be solved urgently. Summary of the invention

[0008] In order to ensure user privacy and security while ensuring cache hit rate and reducing system cost in an energy-constrained edge cloud system, the present invention provides a low-cost video edge caching method based on federated deep reinforcement learning, including:

[0009] Step 1: Establish a network model, including the top layer, middle layer and bottom layer;

[0010] The top layer is a centralized cloud computing center; the middle layer includes multiple base stations, each of which is equipped with an edge server; the bottom layer includes multiple mobile terminals; the centralized cloud computing center is connected to the edge server in communication, and the mobile terminal is connected to the edge server in communication;

[0011] Step 2: The mobile terminal uses local implicit feedback data to jointly train the denoising autoencoder model, and the edge server in the middle layer applies a federated learning framework to aggregate the local models from the underlying mobile terminal devices, and obtains a global model after multiple iterations to complete the popular video caching;

[0012] Step 3: Use the global model to predict the movies that the user is most likely to visit;

[0013] Step 4: Model all computing and communication resources spent in the federated training process as a comprehensive system cost problem, and further formulate the user computing resource allocation problem as a Markov decision process to minimize the expected long-term system cost; train the DDQN model to allocate computing resources to each mobile terminal while satisfying resource constraints.

[0014] Optionally, the step 1 includes:

[0015] The users within the coverage area of ​​each edge server are defined as a set C = {c1, c2, ..., c m}, where c i ∈C; the edge server provides video services to users within the coverage area and accesses the centralized cloud computing center through the backhaul link; the communication operation timeline between the mobile terminal and the base station is described as a time slot ε={1,2,...,K G};

[0016] Define the key factors affecting system performance: uplink transmission delay of the mobile terminal transmitting the local model to the base station, downlink transmission delay of the base station sending the global model to the mobile device, and energy consumption of the mobile terminal transmitting data;

[0017] The uplink transmission delay is expressed as:

[0018]

[0019] in, and Represent the local model parameters and their sizes respectively; is the base station assigned to mobile user c i The channel bandwidth meets the restriction is the channel power gain, Is a mobile device i The transmission power of N0 is Gaussian white noise power;

[0020] The downlink transmission delay is expressed as:

[0021]

[0022] in, and denote the global model parameters and their sizes, respectively. It is the edge server i The channel bandwidth, is the channel power gain, is the transmission power, N0 represents the Gaussian white noise power;

[0023] The energy consumption of transmitting data by the mobile terminal is expressed as:

[0024]

[0025] Among them, γ1 is the transmission energy consumption coefficient;

[0026] Mobile terminalc i In the kth round of iteration, K L The computational delay caused by a round of local model training is:

[0027]

[0028] in, is the dataset size, Indicates mobile terminal c i The CPU / GPU cycle frequency at which the sample data is executed, Indicates mobile terminal c i The number of CPU / GPU cycles to execute the sample data;

[0029] The energy consumed by the mobile terminal for local training is defined as:

[0030]

[0031] Among them, γ2 is the computing energy consumption coefficient relative to the chip architecture.

[0032] Optionally, step 2 includes: using federated learning to jointly complete a model training objective, namely, minimizing a global model loss function:

[0033] w * = argmin F(w)

[0034] The process of federated learning includes:

[0035] Step 21: First, the user downloads the global model parameter w and performs local training. Each client has a training task, and the optimization goal of the local client is:

[0036]

[0037] Among them, F i (w) is the local loss function, To predict losses;

[0038] The input of the network is the user-item interaction matrix R M×N , let y∈R M×N , which indicates whether the user has an implicit feedback behavior for the video, is specifically defined as:

[0039]

[0040] Among them, u represents the user, i represents the video; y u,i =1 indicates that the user has implicit feedback behavior of requesting the video, y u,i =0 means that there is no request for the video, which does not mean that the user is not interested in the video, but may also mean that the user has not yet discovered the existence of video i;

[0041] Step 22: After the user completes local training, the local model parameters will be uploaded To the edge server;

[0042] Step 23: After receiving the local model parameters, the edge server will aggregate them. The parameter aggregation rules are as follows:

[0043]

[0044] Step 24: After the edge server obtains the global model, it continues to distribute the global model parameters to the user, and the user will continue with step 21.

[0045] Optionally, step 3 includes:

[0046] Each mobile terminal uses the trained global model to recommend content, and the output is the probability of each movie being requested and uploaded to the server;

[0047] Using the hidden layer of the autoencoder, we use similarity to obtain the top H users who are similar to the active user, and take the movie lists they have clicked as recommendations;

[0048] The movies recommended by the autoencoder and the movies recommended by similar users are merged, and the top N movies are taken as the recommended movies for the mobile terminal according to the number of recommendations of each movie and cached.

[0049] Optionally, step 4 includes: defining a state space S, an action space A, and a reward function R in the DDQN model; the state in the state space S is represented as: s t ∈S, where Π is the Cartesian product, and represents the state vector combination of mobile user c at time t, where They represent the delay, energy and bandwidth of user c at the current time t respectively;

[0050] The t-th round action in the action space A is represented by a t ∈A, in Indicates the CPU frequency allocation set;

[0051] The reward function R defines an immediate reward function that is inversely proportional to the delay and energy consumption: t = -σT(t)-ιΩ(t), where σ and ι are two positive constants ranging from 0 to 1, T(t) and Ω(t) represent the delay and energy consumption caused by computing and communication respectively;

[0052] Use the discounted return as the total return R of all rewards from the current moment to the end of this episode t :

[0053]

[0054] Among them, γ is the loss factor, γ∈(0,1].

[0055] Optionally, the delay incurred by a round of federated learning is the sum of communication delay and computation delay, expressed as:

[0056]

[0057] The total energy consumption of one round of federated learning is:

[0058]

[0059] The total system cost is defined as the weighted sum of the time and energy consumption due to computation and communication, expressed as:

[0060]

[0061] The problem of reasonably allocating computing resources under energy consumption constraints to minimize the total system cost is described as follows:

[0062]

[0063] C3:F(ω)<κ

[0064] Among them, T max and Ω max is a constant; C1 constrains the CPU / GPU frequency allocated to the mobile terminal; C2 constrains the total energy consumption not to be higher than the upper limit; C3 constrains the global loss function to be less than κ.

[0065] Optionally, a round of federated learning includes:

[0066] (1) The base station initiates a training task and sends the global model to the mobile terminal via a downlink transmission link;

[0067] (2) The mobile terminal is trained with a local model;

[0068] (3) The mobile terminal uploads the trained local model to the base station via an uplink transmission link.

[0069] Optionally, in step 3, the cosine similarity is used to obtain the top H users similar to the active user.

[0070] Optionally, the state-action value function in the DDQN model is:

[0071]

[0072] Among them, Q' and Q network are two neural networks with the same structure, r t Indicates the immediate reward of the current state.

[0073] Optionally, the mobile terminal includes: a smart phone, a smart wearable device, a smart tablet and a portable computer.

[0074] The beneficial effects of the present invention are:

[0075] The present invention proposes a caching scheme based on federated reinforcement learning and implicit user feedback. While protecting user data privacy, the scheme does not require explicit rating data and only requires implicit feedback data for federated training. Each client jointly trains a denoising autoencoder model, and the model aggregated by the edge server in the last round of communication is the trained model, completing the popular video cache. In addition, the scheme not only improves the cache hit rate, but also reduces the system cost through the resource allocation scheme based on reinforcement learning.

[0076] The present invention models all computing and communication resource costs in the federated training process as a system comprehensive cost, and further formulates the user computing resource allocation problem as a Markov decision process (MDP) to minimize the expected long-term system cost (reflecting the user's QoE). Under the condition of satisfying resource constraints, a DDQN-based resource allocation method is proposed to solve the optimal resource allocation strategy, which can effectively allocate the computing resources of each federated training client and minimize the total cost in the federated learning process.

[0077] Through verification on real data sets, experiments show that the cache hit rate of the present invention is higher than that of the baseline algorithm and is close to Oracle, i.e. the optimal algorithm. In addition, experiments also show that the active video caching method of the present invention can effectively reduce system costs when local resources are limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0079] Figure 1 This is a schematic diagram of the technical process of the video edge caching method based on federated deep reinforcement learning of the present invention.

[0080] Figure 2 Schematic diagram of the video caching process described in the present invention.

[0081] Figure 3 The figure is a comparison diagram of cache hit rate between the cache method of the present invention and other four algorithms when the cache capacity is 50-500.

[0082] Figure 4 It is a schematic diagram comparing the average system cost of the cache method of the present invention and the other three algorithms.

[0083] Figure 5 This is a diagram of the network model structure of the caching method described in the present invention.

[0084] Figure 6 This is a screenshot of the pseudo code based on the DDQN resource allocation algorithm in an embodiment of the present invention.

[0085] Figure 7 This is a screenshot of the pseudo code of the video caching algorithm based on federated learning in an embodiment of the present invention.

[0086] Figure 8 This is a flow chart of a DDQN resource allocation algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0087] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0088] Embodiment 1:

[0089] This embodiment provides a video edge caching method based on federated deep reinforcement learning, including:

[0090] Step 1: Establish a network model, including the top layer, middle layer and bottom layer;

[0091] The top layer is a centralized cloud computing center; the middle layer includes multiple base stations, each of which is equipped with an edge server; the bottom layer includes multiple mobile terminals; the centralized cloud computing center and the edge server are connected in communication, and the mobile terminals and the edge server are connected in communication;

[0092] Step 2: The mobile terminal uses local implicit feedback data to jointly train the denoising autoencoder model. The edge server in the middle layer uses the federated learning framework to aggregate the local models from the underlying mobile terminal devices. After multiple iterations, the global model is obtained to complete the popular video caching.

[0093] Step 3: Use the global model to predict the movies that the user is most likely to visit;

[0094] Step 4: Model all computing and communication resources spent in the federated training process as a comprehensive system cost problem, and further formulate the user computing resource allocation problem as a Markov decision process to minimize the expected long-term system cost; train the DDQN model to allocate computing resources to each mobile terminal while satisfying resource constraints.

[0095] Embodiment 2:

[0096] This embodiment provides a low-cost video edge caching method based on federated deep reinforcement learning. The method uses a three-layer network architecture to simulate a real city scene, including a top layer, a middle layer, and a bottom layer. The top layer is also called the cloud, the middle layer is also called the edge, and the bottom layer is also called the terminal. The method specifically includes the following steps:

[0097] Step 1: Build a network model, including the top layer, middle layer and bottom layer.

[0098] The top layer of the network is a centralized cloud computing center with unlimited computing and storage resources; the middle layer contains multiple small base stations (SBS), which have edge servers (ES) with computing and storage resources, such as Nvidia Jetson TX2; the bottom layer is composed of multiple mobile users, and each user's smart mobile device, such as smartphones, smart wearable devices, smart tablets and portable computers, is equipped with chips capable of running lightweight neural networks.

[0099] The cloud computing center connects to the edge and transmits data through a backhaul link, and mobile users / terminal devices connect and communicate with the SBS / edge server through a cellular network (also known as a mobile network, such as 3G, 4G, 5G, etc.).

[0100] Step 2: The underlying mobile terminal devices use local implicit feedback data to jointly train a denoising autoencoder model. The edge server in the middle layer uses a federated learning framework to aggregate the local models from the underlying mobile terminal devices. After multiple iterations, a global model is obtained to complete the popular video caching.

[0101] Step 3: Use the global model to predict the movies that the user is most likely to visit. After passing through the autoencoder model, the output is the probability of each movie being requested. Take the top N movies as the recommended movies for the user. In addition, since the autoencoder has the characteristic of exploring the potential characteristics of data, this embodiment also takes the movie list of similar users according to the collaborative filtering rules as the recommended movies for the user.

[0102] Step 4: Model all computing and communication resources spent in the federated training process as a comprehensive system cost problem, and further formulate the user computing resource allocation problem as a Markov decision process (MDP) to minimize the expected long-term system cost (reflecting the user's QoE). Under the premise of satisfying resource constraints, the DDQN-based resource allocation method is used to solve the optimal resource allocation strategy, effectively allocate the computing resources of each federated training client, and minimize the total cost of the federated learning process.

[0103] Wherein, step 1 specifically includes:

[0104] Each user within the coverage area of ​​ES is defined as a set C = {c1, c2, ..., c m}, where c i ∈C. ES can provide video services to users in different geographical locations within its coverage area, and can also access the cloud center through the backhaul link. The communication operation timeline between the mobile user and the SBS is described as the time slot ε={1,2,...,KG}.

[0105] In the communication model, this embodiment takes into account three key factors affecting system performance: uplink transmission delay of mobile devices transmitting local models to SBS, downlink transmission delay of SBS sending global models to mobile devices, and energy consumption of mobile devices transmitting data.

[0106] Within the coverage of the SBS, the mobile user communicates with the ES via a wireless transmission link. This embodiment assumes that the maximum total bandwidth provided by the SBS to the mobile device is When the mobile user has trained the local model after ξ local iterations and uploaded it to the ES, it communicates with the ES through the uplink transmission link. According to Shannon’s formula, the mobile user c i The maximum uplink transmission rate for transmitted data is:

[0107]

[0108] in, It is the SBS assigned to the mobile user c i The channel bandwidth meets the restriction is the channel power gain, Is a mobile device i The transmission power of N0 is the Gaussian white noise power.

[0109] In the εth round of communication, the uplink transmission delay is for:

[0110]

[0111] in, and denote the local model parameters and their sizes respectively.

[0112] When a mobile user sends a video request and the requested video is cached on the ES, the ES communicates with the mobile user through the downlink transmission link and transmits the cached video content to the mobile user terminal. In addition, the ES also sends the global model to the mobile user terminal through the downlink transmission link. i With ES e i The maximum downlink transmission rate for transmitting data is:

[0113]

[0114] in, Yes i The channel bandwidth, is the channel power gain, is the transmission power, and N0 represents the Gaussian white noise power.

[0115] In the εth round of communication, the downlink transmission delay is for:

[0116]

[0117] in, and denote the global model parameters and their sizes respectively.

[0118] The mobile user equipment consumes energy when transmitting data through the uplink. i Transmission energy consumption Defined as:

[0119]

[0120] Where γ1 is the transmission energy consumption coefficient.

[0121] Since SBS has sufficient resources, this embodiment assumes that ES e i The transmission energy consumption has little impact on the system, therefore, the model of this embodiment only considers the transmission energy consumption of the mobile device.

[0122] Most modern smart communication devices are equipped with powerful system-on-chips (SoCs) that can train lightweight neural network models. When training models locally on mobile user devices, computational delays and energy consumption are incurred.

[0123] For each client The CPU / GPU cycles and CPU / GPU cycles frequency of the sample data are and The CPU / GPU cycle frequency represents the local computing power. Assume that the CPU / GPU cycle frequency is divided into l levels, namely Mobile Userc i In the kth round of iteration, K L The computational delay caused by a round of local model training is:

[0124]

[0125] in, is the dataset size.

[0126] Considering that the SBS has sufficient computing resources, this embodiment assumes that the computing energy consumption of the SBS has little impact on the entire system. Therefore, this embodiment only considers the energy consumed by the mobile device for local training, which is defined as:

[0127]

[0128] Among them, γ2 is the computing energy consumption coefficient relative to the chip architecture.

[0129] Step 2 of this embodiment includes: using federated learning to jointly complete a model training goal, that is, minimizing the global model loss function:

[0130] w * = argmin F(w)

[0131] The process of federated learning includes:

[0132] Step 21: First, the user downloads the global model parameter w and performs local training. Each client has a training task, and the optimization goal of the local client is:

[0133]

[0134] Among them, F i (w) is the local loss function, To predict losses.

[0135] The input of the network is the user-item interaction matrix R M×N Let y∈R M×N , which indicates whether the user has an implicit feedback behavior for the video, is specifically defined as:

[0136]

[0137] Among them, u represents the user, i represents the video; y u,i =1 indicates that the user has implicit feedback behavior of requesting the video, y u,i =0 means that there is no request for the video, which does not mean that the user is not interested in the video, but may also mean that the user has not yet discovered the existence of video i. If the number of user-item matrix evaluations is not less than 4, the corresponding elements of the matrix are set to 1, otherwise they are all set to 0.

[0138] The input of the denoising autoencoder is not complete data, but partially destroyed input data. This enables the hidden layer to discover more robust features instead of learning overly simple features. This embodiment randomly sets the values ​​of some tensors to 0 through dropout.

[0139] Since the model trained in this embodiment is an autoencoder network model, the learning goal is whether the video will be requested in the future (1 if it will be requested, 0 if it will not be requested), so this is a binary classification problem. This embodiment adds a sigmoid activation function in the output layer to convert the output of the hidden layer into a probability between 0 and 1. The output layer is whether the user will request the video, that is, a binary classification task. Therefore, the loss function is a binary cross entropy loss function:

[0140]

[0141] in

[0142] When encoding, the autoencoder model of this embodiment not only has the user's interactive behavior on the video (user-item matrix), but also takes into account the user's characteristics, that is, the user characteristics are also encoded to obtain a hidden layer, which can make the semantics richer. The autoencoder model used in this embodiment is a denoising autoencoder, so this embodiment adds noise to the training data through dropout, but when restoring the data, the original clean data is restored, which is done to make the model more robust.

[0143] Step 22: After the user completes local training, the local model parameters will be uploaded To edge server

[0144] Step 23: After receiving the local model parameters, the edge server will aggregate them. Here, the parameter aggregation rule is:

[0145]

[0146] Step 24: After the edge server obtains the global model, it continues to distribute the global model parameters to the user, and the user will continue with step 21.

[0147] From the previous description, we can see that a complete round of FL communication process is:

[0148] 1. SBS initiates the training task, that is, sends the global model to the mobile device through the downlink transmission link;

[0149] 2. The mobile device is calculated by the local model;

[0150] 3. The mobile device uploads the trained local model to the SBS via the uplink transmission link.

[0151] A round of FL communication is completed. Since each mobile user c iThe time it takes to train local models varies depending on the computing resources. SBS starts model aggregation only after receiving all local models. Therefore, the delay caused by a round of FL communication is the sum of communication delay and computing delay:

[0152]

[0153] The total energy consumption generated by one round of FL communication is:

[0154]

[0155] This embodiment defines the total system cost as the weighted sum of computing and communication resources, that is, the weighted sum of time and energy consumption caused by computing and communication:

[0156]

[0157] How to reasonably allocate computing resources under energy consumption constraints to minimize the total system cost is a key issue. This embodiment describes this issue, i.e., the goal, as follows:

[0158]

[0159] C3:F(ω)<κ

[0160] Since the units of time and energy consumption are different, the cost of the entire system cannot be defined by simply adding the sum of time and energy consumption. Therefore, this embodiment defines the system cost as the sum of the time consumption ratio and the energy consumption ratio. max and Ω max is a constant.

[0161] C1 constrains the CPU / GPU frequency allocated to mobile users to be greater than the minimum value available and less than the maximum CPU / GPU frequency available. C2 constrains the total energy consumption to be no higher than the upper limit. C3 constrains the global loss function to be less than κ, ensuring the training quality.

[0162] Step 3 of this embodiment includes:

[0163] After the training is completed, each client uses local data to recommend popular content to users based on the global model and uploads it to the server. When each client makes a prediction, the input is the existing training set, and the training set data is not destroyed, that is, no noise is added. After passing through the autoencoder model, the output is the probability of each movie being requested. In this embodiment, the top N movies are taken as recommended movies for the user. In addition, since the autoencoder has the characteristic of exploring the potential characteristics of the data, this embodiment also takes the movie list of similar users as the recommended movies for the user according to the collaborative filtering rules. The specific process is that this embodiment uses the hidden layer of the autoencoder and uses cosine similarity to find the top H users similar to the active user, and takes the movie list they have clicked as a recommendation. Active users are defined as users who have more interactive behaviors with videos. The cosine similarity formula is as follows:

[0164]

[0165] The movies recommended by the autoencoder and the movies recommended by similar users are combined, and the top-N movies are selected as the movies recommended by the client based on the number of recommendations for each movie. The server sorts the recommendation results of each client according to the number of recommendations for each movie, and caches the top N movies based on the cache capacity.

[0166] Step 4 of this embodiment includes:

[0167] The purpose of this embodiment is to minimize the system cost caused by federated learning in an edge cloud system with energy constraints. This embodiment models this problem as a Markov decision process (MDP) problem. The state space and action space are discrete. Next, this embodiment describes the state space S, action space A, and reward function R in the DDQN model respectively.

[0168] State space S: The network bandwidth in the mobile edge network scenario of this embodiment is dynamic, which conforms to the rules of the actual scenario. However, the network will not change dramatically in a short period of time, such as tens of seconds. According to the above optimization formula, there are many factors that affect the system cost, such as delay, consumed energy and network bandwidth, and the network bandwidth is related to the historical bandwidth. Therefore, this embodiment sets the current state space of the DDQN model to where Π is the Cartesian product, and represents the state vector combination of mobile user c at time t, where They represent the delay, energy and bandwidth of user c at the current time t. Since S is a state space set, s t ∈S.

[0169] Action Space A: The action of the DDQN agent is to allocate CPU resources. CPU frequency θ c For discrete type, is the CPU frequency allocation set. The action in round t is defined as a t , In this embodiment, the action space is defined as A, then a t ∈A.

[0170] Reward function R: Since the goal of this embodiment is to minimize the system cost under energy consumption constraints, this embodiment defines an immediate reward function that is inversely proportional to delay and energy consumption: t =-σT(t)-ιΩ(t), where σ and ι are two positive constants ranging from 0 to 1. Since different federated learning tasks have different preferences for learning time and energy efficiency, this embodiment defines two non-negative parameters σ and ι to adjust the preferences in the objective function.

[0171] This embodiment uses the discounted return as the total return R of all rewards from the current moment to the end of this episode. t :

[0172]

[0173] Among them, γ is the loss factor, γ∈(0,1].

[0174] What actions are taken in the process involves the policy. In this embodiment, given a certain policy π, a=π(s), which means the input state s and the output action a. (This policy π summarizes how to choose behavior in each state. Therefore, the policy π is something like a set.) π(a|s)=P(A t =a|S t =s) represents the probability of the agent taking an action a when a state s is input. The expected return of a state is called the value of the state, and the values ​​of all states constitute the value function. When following the strategy π, this embodiment gives the expected long-term training reward state-action value function as:

[0175]

[0176] The Bellman equation can be derived as follows:

[0177] Q π (s,a)=E π [r t +γR t+1 |S t =s,A t =a]

[0178] =E π [r t +γQ π (S t+1 ,A t+1 )|S t =s,A t =a]

[0179] This state-action value function is the Q-value function, which represents the expected reward that an agent can obtain by performing an action a according to a specific strategy π for the current state s, that is, the maximum possible benefit of the entire episode.

[0180] The optimal strategy π ＊ will be a policy that maximizes the state-action value function, obtained by the following optimal objective function:

[0181]

[0182] Therefore, the optimal state-action value function Q * (s,a) is:

[0183]

[0184] That is, the optimal Q value of the current state and behavior is equal to the immediate reward r t Add the maximum possible Q value for the next state.

[0185] Deep reinforcement learning (DRL) is a popular algorithm for solving MDP problems, especially when the state space and action space are very large. Since the action of allocating computing resources is discrete, this embodiment selects the value-based DQN algorithm. In past studies, it was found that DQN overestimates the action value, and doubledeep Q-network (DDQN) can solve the problem of overestimation of action value. Therefore, this embodiment selects the DDQN method, which changes the state-action value function in DQN to:

[0186]

[0187] Among them, Q' and Q network are two neural networks with the same structure.

[0188] In addition, considering the limited available traces of FL, DDQN reuses data more effectively than policy gradient methods and actor-critic methods. This section details the neural network training function Q(s,a;ω) of the DDQN-Based Agent to approximate Q* (s,a) Find the optimal strategy π ＊ The target value of Q(s,a;ω) is The estimated value is Q(s t ,a;ω). Therefore, the loss function Loss=MSE(y t ,Q(s t ,a;ω)). Among them, the pseudo code of the DDQN resource allocation algorithm and the video caching algorithm based on federated learning is as follows Figure 6 and Figure 7 shown.

[0189] The flow chart of DDQN resource allocation algorithm is as follows Figure 8 As shown, the following steps are included:

[0190] S1: Get the experience replay memory pool size D, initial network weight parameters, regularization coefficient, learning rate, discount factor, maximum resource amount, Q network, and target network;

[0191] S2: Randomly initialize the weights of the Q network and the target network;

[0192] S3: observe the current state s;

[0193] S4: Select actions based on the greedy strategy;

[0194] S5: store the status of the conversion;

[0195] S6: Determine whether the step exceeds D. If so, perform network learning and update parameters, and clear the playback memory pool size D;

[0196] S7: Determine whether the accuracy of FL reaches a or the remaining resources are less than 0. If so, terminate;

[0197] S8: Return to S3 and continue execution.

[0198] The video caching algorithm based on federated learning includes the following steps:

[0199] S1: The client and server establish communication;

[0200] S2: Each client downloads the global model from the server and performs local training;

[0201] S3: All clients upload their local models to the server;

[0202] S4: The server updates the global model;

[0203] S5: Input the current state into the DDQN model and output the CPU frequency assigned to each client;

[0204] S6: Send the allocated CPU frequency to the client;

[0205] S7: Return to S1 and continue execution.

[0206] This embodiment performs a simulation experiment on the cache method of the present invention, and the results are as follows:

[0207] Figure 3 A schematic diagram comparing the cache hit rate of the cache method of the present invention and the other four algorithms when the cache capacity is 50-500 is described. The cache hit rate results increase with the increase of cache size, and the curve shows an upward trend. This is because the larger the cache space, the more it can store a few or less popular videos for different user requests. Oracle shows the best cache results because Oracle knows all future user requests in advance. FRPVC and c-ε-greedy both learn from past implicit feedback data, so their performance is better than TS and Random. The cache performance of FRPVC is closest to Oracle because FRPVC uses a denoising autoencoder to learn damaged user implicit feedback data and make video recommendations. In addition, FRPVC also uses the collaborative filtering and data feature extraction characteristics of the autoencoder to use videos requested by similar users for video recommendations. Therefore, the cache hit rate of FRPVC is very high. Since videos are randomly selected from all movies for caching, the cache hit rate of Random is the worst.

[0208] Figure 4 The schematic diagram comparing the average system cost of the caching method of the present invention with the other three algorithms is described. For intuitive comparison, we selected two comparison algorithms, namely Heuristic and Static. In Heuristic, since the bandwidth information of the previous round of the mobile device can be known for each iteration of federated learning, the CPU cycle frequency of the current round can be determined based on this information. The Static algorithm assumes that the network is static and uses a uniform CPU cycle frequency for each iterative training. The CPU cycle frequency used in the Static algorithm is determined by taking the average value of the data in the network bandwidth data set. It can be found that the DRL method achieves the best performance.

[0209] Specifically, the average system cost of the DDQN method is 0.24, while the average system costs of the Heuristic and Static methods are 0.67 and 0.73, respectively. This is due to the learning ability of the DDQN method, which can gain experience from previous episodes and allocate the most appropriate CPU frequency. Although the Heuristic algorithm can gain some experience from the bandwidth information in the previous round, it cannot guarantee the lowest long-term average system energy consumption. Static always uses the same CPU frequency in the FL training round, resulting in the highest system energy consumption and time.

[0210] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0211] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A low-cost video edge caching method based on federated deep reinforcement learning, characterized in that: The method comprises: Step 1: Establish a network model, including the top layer, middle layer and bottom layer; The top layer is a centralized cloud computing center; the middle layer includes multiple base stations, each of which is equipped with an edge server; the bottom layer includes multiple mobile terminals; the centralized cloud computing center is connected to the edge server in communication, and the mobile terminal is connected to the edge server in communication; Step 2: The mobile terminal uses local implicit feedback data to jointly train the denoising autoencoder model, and the edge server in the middle layer applies a federated learning framework to aggregate the local models from the underlying mobile terminal devices, and obtains a global model after multiple iterations to complete the popular video caching; Step 3: Use the global model to predict the movies that the user is most likely to visit; Step 4: Model all computing and communication resources spent in the federated training process as a comprehensive system cost problem, and further formulate the user computing resource allocation problem as a Markov decision process to minimize the expected long-term system cost; train the DDQN model to allocate computing resources to each mobile terminal while satisfying resource constraints.

2. The video edge caching method according to claim 1, characterized in that: The step 1 comprises: The users within the coverage area of ​​each edge server are defined as a set C = {c1, c2, ..., c m }, where c i ∈C; the edge server provides video services to users within the coverage area and accesses the centralized cloud computing center through the backhaul link; the communication operation timeline between the mobile terminal and the base station is described as a time slot ε={1,2,...,K G }; Define the key factors affecting system performance: uplink transmission delay of the mobile terminal transmitting the local model to the base station, downlink transmission delay of the base station sending the global model to the mobile device, and energy consumption of the mobile terminal transmitting data; The uplink transmission delay is expressed as: in, and Represent the local model parameters and their sizes respectively; is the base station assigned to mobile user c i The channel bandwidth meets the restriction is the channel power gain, Is a mobile device i The transmission power, N0 represents the Gaussian white noise power; The downlink transmission delay is expressed as: in, and denote the global model parameters and their sizes, respectively. It is the edge server i The channel bandwidth, is the channel power gain, is the transmission power, N0 represents the Gaussian white noise power; The energy consumption of transmitting data by the mobile terminal is expressed as: Among them, γ1 is the transmission energy consumption coefficient; Mobile terminalc i In the kth round of iteration, K L The computational delay caused by a round of local model training is: in, is the dataset size, Indicates mobile terminal c i The CPU / GPU cycle frequency at which the sample data is executed, Indicates mobile terminal c i The number of CPU / GPU cycles to execute the sample data; The energy consumed by the mobile terminal for local training is defined as: Among them, γ2 is the computing energy consumption coefficient relative to the chip architecture.

3. The video edge caching method according to claim 2, characterized in that: The step 2 includes: using federated learning to jointly complete a model training goal, namely, minimizing the global model loss function: w * =argmin F(w) The process of federated learning includes: Step 21: First, the user downloads the global model parameter w and performs local training. Each client has a training task, and the optimization goal of the local client is: Among them, F i (w) is the local loss function, To predict losses; The input of the network is the user-item interaction matrix R M×N , let y∈R M×N , which indicates whether the user has an implicit feedback behavior for the video, is specifically defined as: Among them, u represents the user, i represents the video; y u,i =1 indicates that the user has implicit feedback behavior of requesting the video, y u,i =0 means that there is no request for the video, which does not mean that the user is not interested in the video, but may also mean that the user has not yet discovered the existence of video i; Step 22: After the user completes local training, the local model parameters will be uploaded To the edge server; Step 23: After receiving the local model parameters, the edge server will aggregate them. The parameter aggregation rules are as follows: Step 24: After the edge server obtains the global model, it continues to distribute the global model parameters to the user, and the user will continue with step 21.

4. The video edge caching method according to claim 3, characterized in that: The step 3 comprises: Each mobile terminal uses the trained global model to recommend content, and the output is the probability of each movie being requested and uploaded to the server; Using the hidden layer of the autoencoder, we use similarity to obtain the top H users who are similar to the active user, and take the movie lists they have clicked as recommendations; The movies recommended by the autoencoder and the movies recommended by similar users are merged, and the top N movies are taken as the recommended movies for the mobile terminal according to the number of recommendations of each movie and cached.

5. The video edge caching method according to claim 4, characterized in that: The step 4 includes: defining a state space S, an action space A, and a reward function R in the DDQN model; The state in the state space S is represented as: t ∈S, where Π is the Cartesian product, and represents the state vector combination of mobile user c at time t, where They represent the delay, energy and bandwidth of user c at the current time t respectively; The t-th round action in the action space A is represented by a t ∈A, in Indicates the CPU frequency allocation set; The reward function R defines an immediate reward function that is inversely proportional to the delay and energy consumption: t = -σT(t)-ιΩ(t), where σ and ι are two positive constants ranging from 0 to 1, T(t) and Ω(t) represent the delay and energy consumption caused by computing and communication respectively; Use the discounted return as the total return R of all rewards from the current moment to the end of this episode t : Among them, γ is the loss factor, γ∈(0,1].

6. The video edge caching method according to claim 2, characterized in that: The delay caused by a round of federated learning is the sum of communication delay and computation delay, expressed as: The total energy consumption of one round of federated learning is: The total system cost is defined as the weighted sum of the time and energy consumption due to computation and communication, expressed as: The problem of reasonably allocating computing resources under energy consumption constraints to minimize the total system cost is described as follows: C3:F(ω)<κ Among them, T max and Ω max is a constant; C1 constrains the CPU / GPU frequency allocated to the mobile terminal; C2 constrains the total energy consumption not to be higher than the upper limit; C3 constrains the global loss function to be less than κ.

7. The video edge caching method according to claim 1, characterized in that: A round of federated learning includes: (1) The base station initiates a training task and sends the global model to the mobile terminal via a downlink transmission link; (2) The mobile terminal is trained with a local model; (3) The mobile terminal uploads the trained local model to the base station via an uplink transmission link.

8. The video edge caching method according to claim 4, characterized in that: The step 3 uses cosine similarity to obtain the top H users similar to the active user.

9. The video edge caching method according to claim 4, characterized in that: The state-action value function in the DDQN model is: Among them, Q' and Q network are two neural networks with the same structure, r t Indicates the immediate reward of the current state.

10. The video edge caching method according to claim 1, characterized in that: The mobile terminals include: smart phones, smart wearable devices, smart tablets and portable computers.

Images