A collaborative caching approach based on quantized federated reinforcement learning in fog wireless access networks

Abstract

The present invention discloses a collaborative caching method based on quantized federated reinforcement learning in a fog wireless access network, including: 1. Initializing the local cache content of the cache node according to the global content popularity, initializing the local model weight parameter of the cache node, the number of local model iterations in a single cycle and the total model training cycle; 2. Each cache node makes content cache decisions and local model training according to the received user request information; 3. Calculate the cache hit rate and user request delay; 4. At the end of a training cycle, perform network pruning on the local model of each cache node; 5. Perform quantized compression and weight sharing processing on the local model update amount; 6. Upload the local model update amount to the cloud center for global model aggregation to obtain a global model, and then distribute the global model to each cache node to enter the next training cycle. The present invention improves the cache hit rate and reduces the network load pressure.

Description

Technical Field

[0001] The present invention belongs to the field of collaborative caching of edge networks in mobile communication systems, and in particular to a collaborative caching method based on quantized federated reinforcement learning in a fog wireless access network. Background Art

[0002] With the widespread popularity of various smart devices and the rapid development of multimedia application technology, the surge in data traffic has caused many problems in wireless networks, such as network congestion, server overload, etc. In the fog wireless access network, cache nodes with computing, caching and signal processing functions are equipped at the edge of the network to store the most popular content for users in the cache nodes. Since the cache nodes storing content are closer to the users who request the content, users can directly obtain the content from the cache nodes, which reduces the delay of user requesting content and also reduces the traffic load pressure of the fronthaul link.

[0003] In recent years, learning algorithms have been widely used to optimize content caching problems by learning key attribute characteristics of fog wireless access networks, such as user request behavior, content popularity, and user mobility distribution. However, most centralized learning algorithms are prone to cache over-utilization problems, which seriously consume network communication resources. At the same time, centralized learning algorithms require users to upload data to the cloud center server, and user data security will also be threatened. On the other hand, although distributed learning methods do not need to transmit user data, they require a large amount of cache and action space, such as the dimensionality trap problem of reinforcement learning, which is a huge challenge for cache nodes with limited storage and computing resources in fog wireless access networks. In order to protect user data privacy and promote collaborative training of models between distributed devices, federated learning can meet the above requirements well through model deployment training. However, federated learning protects user data privacy through model transmission at the cost of huge communication resource consumption. Therefore, on the one hand, it is necessary to strengthen multi-model collaborative training, improve cache hit rate, and reduce user request latency. On the other hand, it is also necessary to protect user data privacy and reduce the huge communication resource consumption caused by model transmission in federated learning while ensuring that the model training accuracy remains unchanged or the loss is very small. Summary of the invention

[0004] The purpose of the present invention is to provide a collaborative caching method based on quantized federated reinforcement learning in a fog wireless access network to solve the technical problems of low content cache hit rate, high user content request latency, poor protection of user privacy, and excessive consumption of communication resources.

[0005] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0006] A collaborative caching method based on quantized federated reinforcement learning in a fog wireless access network comprises the following steps:

[0007] Step 1: According to the global content popularity P f , initialize the local content cache status of the cache node The weight parameter θ of the training model n , the number of local model iterations in a single cycle is X and the total model training cycle is Y;

[0008] Step 2: At the tth iteration, the state space of cache node n is in, Indicates the content cache status in cache node n. Represents a collection of cache nodes, Represents the collection of all request contents; Indicates the content request status of cache node n, represents the user set in cache node n; the action space of cache node n is in, Indicates that the request content can be processed in the local cache node n, otherwise, in, Indicates that the requested content can be processed from the neighboring cache node l, otherwise, Indicates that the request content can be processed from the cloud center, otherwise, in, Indicates that the cache content j in cache node n will be replaced by the requested content, otherwise, Cache node n collects user content request information and caches it based on the local content cache status. Make a response decision; the local model of cache node n is It is the local model parameter of cache node n, and performs local model update according to user request information;

[0009] Step 3: Based on step 2, cache node n makes a response decision, that is, selects a server from the local cache node, neighboring cache nodes and cloud center to process the user's request content, and then calculates the cache hit rate h n (t) and user content request delay and Indicates the request latency caused by the user directly obtaining the requested content from the local cache node. Indicates the request latency caused by the user needing to obtain its request content from a neighboring cache node. Indicates the request delay caused by the user needing to obtain its request content from the cloud center server;

[0010] Step 4: After the cache node performs X iterative updates of the local content, it calculates the update amount of the local model parameters and performs network pruning on the local model according to the parameter update amount of each layer of the model;

[0011] Step 5: The cache node performs quantization compression and weight sharing processing on the local model parameter update amount after network pruning, and uploads the processed local model parameter update amount to the cloud center server;

[0012] Step 6: The cloud center server globally aggregates all received local model parameter updates to obtain a global model. The global model is distributed to each cache node and enters the next cycle.

[0013] Furthermore, step 2 specifically includes the following steps:

[0014] Step 2.1: Cache the state space of node n For the arriving user request, cache node n selects from the action space Select an action to handle the user request;

[0015] Step 2.2: If Then the cache replacement update action will occur, and the cached content j in cache node n will be replaced by the requested content f;

[0016] Step 2.3: After the cache replacement operation in step 2.2, the cache state of cache node n will be updated to obtain a new state space Cache node n will iteratively update the local cache model parameters to obtain is the learning rate, is the local model of cache node n Parameters gradient.

[0017] Furthermore, step 3 calculates the cache hit rate h n (t) and user content request delay and The specific steps include:

[0018] Step 3.1: Based on the cache decision made by the user, in time slot t, the request delay caused by the user directly obtaining its request content from the local cache node is:

[0019]

[0020] Where N represents the number of cache nodes, p n,f represents the probability distribution of content f being requested by users in cache node n, Indicates that the user can obtain the requested content f from the local node n. Indicates that the user cannot obtain the requested content f, d from the local node n a It represents the transmission delay caused by transmitting a single content from the cache node to the user;

[0021] Step 3.2: Based on the cache decision made by the user, in time slot t, the request delay caused by the user needing to obtain its request content from the neighboring cache nodes is:

[0022]

[0023] in, It means that the user can obtain the requested content f from the neighboring cache node l. Indicates that the user cannot obtain the requested content f from the neighboring cache node l, d b It represents the transmission delay caused by transmitting a single content between cache nodes;

[0024] Step 3.3: Based on the cache decision made by the user, in time slot t, the request delay caused by the user needing to obtain its request content from the cloud center server is:

[0025]

[0026] in, Indicates that the user needs to obtain its request content from the cloud center server. Indicates that the user does not need to obtain the requested content from the cloud center server. c Indicates the transmission delay caused by transmitting a single content from the cloud center server to the cache node;

[0027] Step 3.4: In time slot t, for a given cache state in node n, calculate the total content request delay of the user: Cache hit rate:

[0028]

[0029] Furthermore, step 4 specifically includes the following steps:

[0030] Step 4.1: In the yth cycle, after X local iterations, the model parameters of the local model of cache node n are reduced from L represents the number of layers of the local model;

[0031] Step 4.2: Cache node n calculates the parameter change of layer i as This formula means that the absolute value of the change of the i-th layer parameter is taken first, and then the average value of the absolute value of the change of the i-th layer parameter is taken; after calculation, the set of changes of all layers of the local model of cache node n is obtained as

[0032] Step 4.3: Set the threshold χ so that the layer parameter change The layer parameters are uploaded to the cloud center server to participate in the global model aggregation update, and the remaining layer parameters do not participate in the global model update.

[0033] Furthermore, in step 5, the cache node performs quantization compression and weight sharing processing on the local model parameter update amount after network pruning, which specifically includes the following steps:

[0034] Step 5.1: Use the K-means clustering algorithm to cluster the changes in layer parameters of each layer and divide the n original parameters into k clusters. n>>k, the clustering goal is to minimize the intra-cluster sum of squares, expressed as:

[0035]

[0036] The centroid of each class in the K-means clustering algorithm is linearly initialized, so that the selection of the centroid position is not affected by the weight distribution. The centroid of the K-means clustering is the shared weight, and the original parameters classified into the same class will uniformly take the same weight;

[0037] Step 5.2: For k classes, log2(k) bits are required to encode the index; for a network with n weight parameters, one weight parameter upload requires b bits. Limiting n weight parameters to only k shared weights will result in a compression ratio r of:

[0038] Step 5.3: The cache node performs quantization compression and weight sharing on the local model parameter update. This process is represented by θ y Represents the global model parameters received by node n at the beginning of the yth cycle. The cache node will Upload to the cloud center server.

[0039] Furthermore, in step 6, the cloud center server globally aggregates all received local model parameter updates to obtain a global model, and the global model is distributed to each cache node to enter the next cycle, which specifically includes the following steps:

[0040] Step 6.1: After X local update trainings in the yth cycle, the cloud center server collects the weight parameter quantization update amount from each cache node θ y represents the global model parameters received by node n at the beginning of the yth cycle;

[0041] Step 6.2: Calculate and update the weight parameters of the global model:

[0042]

[0043] Among them, θ y+1 is the updated global model weight parameter, is the local data set of cache node n;

[0044] Step 6.3: The cloud server sends the global weight parameter to each cache node and Enter the next cycle.

[0045] The collaborative caching method based on quantized federated reinforcement learning in the fog wireless access network of the present invention has the following advantages:

[0046] 1. In a single cache node, the present invention uses deep reinforcement learning to learn the dynamic network environment and time-varying user requests, so as to make optimal cache decisions, improve cache hit rate, and reduce user content request delays.

[0047] 2. In the present invention, horizontal federated reinforcement learning between multiple cache nodes can strengthen the mutual cooperation of model training between multiple cache nodes through the training method of shared models, and ensure that user data always stays in the local node without being shared to the cloud center server, thereby protecting user data privacy well.

[0048] 3. During the model transmission process of federated learning, the present invention performs network pruning on the local model participating in the global update, quantizes and compresses the local model parameter update amount and performs weight sharing processing, thereby compressing the parameter amount of the uploaded local model, reducing the network transmission load and improving the network operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of a collaborative caching method based on quantized federated reinforcement learning in a fog wireless access network of the present invention;

[0050] Figure 2 The figure is a simulation result comparing the cache hit rate of the present invention and the traditional edge cache strategy. DETAILED DESCRIPTION

[0051] In order to better understand the purpose, structure and function of the present invention, the following further describes in detail a collaborative caching method based on quantized federated reinforcement learning in a fog wireless access network of the present invention in conjunction with the accompanying drawings.

[0052] like Figure 1 The flowchart of the collaborative caching method based on quantized federated reinforcement learning in the fog wireless access network of the present invention is shown, which includes the following steps:

[0053] S1: Calculate the global content popularity P according to the Mandelbrot-Zipf distribution f , initialize the local cache content status of the cache node The weight parameter θ of the training model n , the number of local model iterations in a single cycle is X and the total model training cycle is Y

[0054] S2: At iteration t, the state space of cache node n is in, Indicates the content cache status in cache node n. Represents a collection of cache nodes, Represents the collection of all request contents; Indicates the content request status of cache node n, represents the user set in cache node n; the action space of cache node n is in, Indicates that the request content can be processed in the local cache node n, otherwise, in, Indicates that the requested content can be processed from the neighboring cache node l, otherwise, Indicates that the request content can be processed from the cloud center, otherwise, in, Indicates that the cache content j in cache node n will be replaced by the requested content, otherwise, The cache node collects the user's content request information and caches the content based on the local content status. Make a response decision; the local model of the cache node is is the local model parameter of cache node n, and performs local model update according to user request information; specifically includes:

[0055] S2-1: cache state space of node n For the arriving user request, cache node n selects from the action space Select an action to handle the user request.

[0056] S2-2: If Then the cache replacement update action will occur, and the cached content j in cache node n will be replaced by the requested content f. S2-3: After the cache replacement operation in step 2.2, the cache state of cache node n will be updated to obtain a new state space. Cache node n will iteratively update the local cache model is the learning rate, is the local model of cache node n Parameters gradient.

[0057] S3: Based on step 2, the cache node makes a response decision, that is, selects a server from the local cache node, neighboring cache nodes and cloud center to process the user's request content, and then calculates the cache hit rate h n (t) and user content request delay and Indicates the request latency caused by the user directly obtaining the requested content from the local cache node. Indicates the request latency caused by the user needing to obtain its request content from a neighboring cache node. Indicates the request delay caused by the user needing to obtain its request content from the cloud center server; specifically includes:

[0058] S3-1: According to the caching decision made by the user, in time slot t, the total request delay for the user to obtain content from the local node is:

[0059]

[0060] Where N represents the number of cache nodes, p n,f represents the probability distribution of content f being requested by users in cache node n, Indicates that the user can obtain the requested content f from the local node n. Indicates that the user cannot obtain the requested content f, d from the local node n a Indicates the transmission delay caused by transmitting a single content from a cache node to a user.

[0061] S3-2: According to the cache decision made by the user, in time slot t, the total request delay for the user to obtain content from neighboring nodes is:

[0062]

[0063] in, Indicates that the user can obtain the requested content f from the neighboring cache node. Indicates that the user cannot obtain the requested content f, d from the neighboring cache nodes b Indicates the transmission delay caused by transmitting a single content between cache nodes.

[0064] S3-3: According to the cache decision made by the user, in time slot t, the total request delay for the user to obtain content from the cloud center server is:

[0065]

[0066] in, Indicates that the user needs to obtain its request content from the cloud center server. Indicates that the user does not need to obtain the request content from the cloud server. c Indicates the transmission delay caused by transmitting a single content from the cloud center server to the cache node.

[0067] S3-4: In time slot t, for a given cache state in node n, calculate the total content request delay of the user: Cache hit rate:

[0068]

[0069] S4: After the cache node has iterated and updated the local content X times, it calculates the update amount of the local model parameters and performs network pruning on the local model according to the update amount of the parameters of each layer of the model. Specifically, it includes:

[0070] S4-1: In the yth cycle, after X local iterations, the model parameters of the local model of cache node n change from becomes L represents the number of layers of the local model.

[0071] S4-2: Cache node n calculates the parameter change of layer i as This formula means that the absolute value of the change of the parameters of the i-th layer is taken first, and then the average value of the absolute value of the change of the parameters of the i-th layer is taken, and the set of changes of all layers of the local model of cache node n is obtained:

[0072] S4-3: Set an appropriate threshold χ so that the layer parameter change The layer parameters are uploaded to the cloud center server to participate in the global model aggregation update, and the remaining layer parameters do not participate in the global model update.

[0073] S5: The cache node performs quantization compression and weight sharing processing on the local model parameter update amount after network pruning, and uploads the processed local model parameter update amount to the cloud center server. Specifically including:

[0074] S5-1: Use K-means clustering algorithm to cluster the changes in layer parameters of each layer and divide n original parameters into k clusters. n>>k, the clustering goal is to minimize the intra-cluster sum of squares, which can be expressed as:

[0075]

[0076] The centroid position of each class in the K-means clustering algorithm is linearly initialized, so that the selection of the centroid position is not affected by the weight distribution. The centroid of the K-means clustering is the shared weight, and the original parameters classified into the same class will uniformly take the same weight.

[0077] S5-2: For k classes, log2(k) bits are required to encode the index. For a network with n weight parameters, one weight parameter upload requires b bits. Limiting n weight parameters to only k shared weights will result in a compression ratio r of:

[0078] S5-3: The cache node performs quantization compression and weight sharing on the local model parameter update. This process represents θ y Represents the global model parameters received by node n at the beginning of the yth cycle training, and then the cache node will Upload to the cloud center server.

[0079] S6: The cloud center server globally aggregates and updates all received local model parameter quantization updates to obtain a global model. Then, the global model is distributed to each cache node and enters the next training cycle. Specifically, it includes:

[0080] S6-1: After X local update trainings in the yth cycle, the cloud center server collects the weight parameter quantization update amount from each cache node θ y Represents the global model parameters received by node n at the beginning of the yth cycle training;

[0081] S6-2: Calculate and update the weight parameters of the global model:

[0082]

[0083] Among them, θ y+1 is the updated global model weight parameter, is the local data set of cache node n;

[0084] S6-3: The cloud server sends the global weight parameter to each cache node and Enter the next cycle.

[0085] By the attached Figure 2The simulation results show that compared with four traditional caching methods, namely federated reinforcement learning (FRL), deep reinforcement learning (DRL), least recently used (LRU) and least frequently used (LFU), the network operation efficiency of the quantized federated reinforcement learning method described in the present invention is significantly higher, and the cache hit rate performance is significantly better. The quantized federated reinforcement learning method is specifically based on the dueling deep Q network (Dueling DQN) and horizontal federated learning (HFL), and performs network pruning, quantization compression and weight sharing processing on the shared training model.

[0086] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A collaborative caching method based on quantized federated reinforcement learning in fog wireless access network, characterized in that: The following steps are involved: Step 1: According to the global content popularity P f , initialize the local content cache status of the cache node The weight parameter θ of the training model n , the number of local model iterations in a single cycle is X and the total model training cycle is Y; Step 2: At the tth iteration, the state space of cache node n is in, Indicates the content cache status in cache node n. Represents a collection of cache nodes, Represents the collection of all request contents; Indicates the content request status of cache node n, represents the user set in cache node n; the action space of cache node n is in, Indicates that the request content can be processed in the local cache node n, otherwise, in, Indicates that the requested content can be processed from the neighboring cache node l, otherwise, Indicates that the request content can be processed from the cloud center, otherwise, in, Indicates that the cache content j in cache node n will be replaced by the requested content, otherwise, Cache node n collects user content request information and caches it based on the local content cache status. Make a response decision; the local model of cache node n is It is the local model parameter of cache node n, and performs local model update according to user request information; Step 3: Based on step 2, cache node n makes a response decision, that is, selects a server from the local cache node, neighboring cache nodes and cloud center to process the user's request content, and then calculates the cache hit rate h n (t) and user content request delay and Indicates the request latency caused by the user directly obtaining the requested content from the local cache node. Indicates the request latency caused by the user needing to obtain its request content from a neighboring cache node. Indicates the request delay caused by the user needing to obtain its request content from the cloud center server; Step 4: After the cache node performs X iterative updates of the local content, it calculates the update amount of the local model parameters and performs network pruning on the local model according to the parameter update amount of each layer of the model; Step 5: The cache node performs quantization compression and weight sharing processing on the local model parameter update amount after network pruning, and uploads the processed local model parameter update amount to the cloud center server; Step 6: The cloud center server globally aggregates all received local model parameter updates to obtain a global model, which is then distributed to each cache node and enters the next cycle. The step 2 specifically includes the following steps: Step 2.1: Cache the state space of node n For the arriving user request, cache node n selects from the action space Select an action to handle the user request; Step 2.2: If Then the cache replacement update action will occur, and the cached content j in cache node n will be replaced by the requested content f; Step 2.3: After the cache replacement operation in step 2.2, the cache state of cache node n will be updated to obtain a new state space Cache node n will iteratively update the local cache model parameters to obtain is the learning rate, is the local model of cache node n Parameters The gradient of Step 3 calculates the cache hit rate h n (t) and user content request delay and The specific steps include: Step 3.1: Based on the cache decision made by the user, in time slot t, the request delay caused by the user directly obtaining its request content from the local cache node is: Where N represents the number of cache nodes, p n,f represents the probability distribution of content f being requested by users in cache node n, Indicates that the user can obtain the requested content f from the local node n. Indicates that the user cannot obtain the requested content f, d from the local node n a It represents the transmission delay caused by transmitting a single content from the cache node to the user; Step 3.2: Based on the cache decision made by the user, in time slot t, the request delay caused by the user needing to obtain its request content from the neighboring cache nodes is: in, It means that the user can obtain the requested content f from the neighboring cache node l. Indicates that the user cannot obtain the requested content f from the neighboring cache node l, d b It represents the transmission delay caused by transmitting a single content between cache nodes; Step 3.3: Based on the cache decision made by the user, in time slot t, the request delay caused by the user needing to obtain its request content from the cloud center server is: in, Indicates that the user needs to obtain its request content from the cloud center server. Indicates that the user does not need to obtain the requested content from the cloud center server. c Indicates the transmission delay caused by transmitting a single content from the cloud center server to the cache node; Step 3.4: In time slot t, for a given cache state in node n, calculate the total content request delay of the user: Cache hit rate: The step 4 specifically comprises the following steps: Step 4.1: In the yth cycle, after X local iterations, the model parameters of the local model of cache node n are reduced from becomes L represents the number of layers of the local model; Step 4.2: Cache node n calculates the parameter change of layer i as It means that the absolute value of the change of the parameters of the i-th layer is taken first, and then the average value of the absolute value of the change of the parameters of the i-th layer is taken; after calculation, the set of changes of all layers of the local model of cache node n is obtained as Step 4.3: Set the threshold χ so that the layer parameter change The layer parameters of are uploaded to the cloud center server to participate in the global model aggregation update, and the remaining layer parameters do not participate in the global model update; In step 5, the cache node performs quantization compression and weight sharing processing on the local model parameter update amount after network pruning, which specifically includes the following steps: Step 5.1: Use the K-means clustering algorithm to cluster the changes in layer parameters of each layer and divide the n original parameters into k clusters. The goal of clustering is to minimize the intra-cluster sum of squares, expressed as: The centroid of each class in the K-means clustering algorithm is linearly initialized, so that the selection of the centroid position is not affected by the weight distribution. The centroid of the K-means clustering is the shared weight, and the original parameters classified into the same class will uniformly take the same weight; Step 5.2: For k classes, log2(k) bits are required to encode the index; for a network with n weight parameters, one weight parameter upload requires b bits. Limiting n weight parameters to only k shared weights will result in a compression ratio r of: Step 5.3: The cache node performs quantization compression and weight sharing on the local model parameter update. θ y Represents the global model parameters received by node n at the beginning of the yth cycle. The cache node will Upload to the cloud center server; In step 6, the cloud center server globally aggregates all received local model parameter updates to obtain a global model, which is distributed to each cache node and enters the next cycle, which specifically includes the following steps: Step 6.1: After X local update trainings in the yth cycle, the cloud center server collects the weight parameter quantization update amount from each cache node θ y represents the global model parameters received by node n at the beginning of the yth cycle; Step 6.2: Calculate and update the weight parameters of the global model: Among them, θ y+1 is the updated global model weight parameter, is the local data set of cache node n; Step 6.3: The cloud server sends the global weight parameter to each cache node and Enter the next cycle.

Images