Wireless edge cache dynamic optimization strategy based on context dobby and deep reinforcement learning

Abstract

The invention discloses a wireless edge cache dynamic optimization strategy based on a context dobby and deep reinforcement learning, and belongs to the technical field of artificial intelligence and edge computing crossing. The method comprises the following steps: (1) in an intelligent traffic scene, a slave base station sends a cache state index set sk of cached contents in a slave base station cache to a master base station; (2) through the DRL, the main base station can carry out reward according to a cache state value sk and an expected discount reward function Q (sk, ak; (2) the slave base station determines an action ak to be executed according to the number [theta] k, so that an optimal cache replacement strategy # imgabs0 # is obtained; (3) the slave base station updates a cached content set according to # imgabs1 #; and (4) extracting and analyzing the feature information of the user and the cache information from the base station by using a CMAB algorithm, and calculating a prediction reward UCB of each content. And selecting the request queue corresponding to the content with the maximum UCB value for multicast. And after receiving the actual reward fed back by the user, updating the model parameters to dynamically adjust the multicast content. The method ensures that the user obtains the latest content, improves the real-time data processing capability and avoids providing old data.

Description

Technical Field

[0001] The present invention relates to the cross-technical field of artificial intelligence and edge computing, and specifically to a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning. Background Art

[0002] Contextual Multi-Armed Bandit (CMAB) is an online learning algorithm used to gradually optimize decisions in an environment with contextual information. Compared with the traditional multi-armed bandit model, CMAB technology enables the decision-making process to dynamically adapt to changes in the environment by introducing context (such as user behavior, environmental status, etc., in this invention, the content information age, request queue size). In each round of decision-making, the algorithm not only selects the optimal action based on the historical performance of the current optional actions (i.e., different arms), but also considers contextual information, and learns and adjusts based on the rewards obtained. The core challenge of CMAB is to balance exploration and utilization: it is necessary to explore uncertain actions to obtain more information, and to use existing knowledge to maximize immediate and long-term returns.

[0003] Deep reinforcement learning (DRL) combines reinforcement learning and deep learning, using reinforcement learning to define problems and optimization goals, and deep learning to solve problems such as state representation, policy representation, and value function modeling. The deep Q network (DQN) is an algorithm that combines the Q-Learning algorithm and the deep neural network (DNN), that is, a Q-Learning algorithm based on deep learning. Q-Learning can perceive the reward function based on parameters such as the current state of the content and the age of the information, and achieve the optimal caching strategy by continuously updating the Q value. This end-to-end optimization can better adapt to different scenario conditions and is more conducive to improving the global performance of the system to adapt to changing network conditions and a large number of user requests.

[0004] The high demand and high expectations of users for smart transportation systems highlight the necessity of regularly updating the cache content of edge devices. As the management requirements of smart devices and mobile Internet services in smart transportation systems increase, traditional caching faces great limitations in popular content during low traffic periods, making it difficult to ensure the safety, operational efficiency, and comfort of the transportation system.

[0005] The present invention combines contextual multi-arm machines with deep reinforcement learning algorithms to optimize cache strategies and multicast scheduling in edge computing environments to improve network performance and resource utilization, update cache content in a timely manner, and take into account the "age" factor of different content or data, that is, the freshness of the content. This ensures that users obtain the latest content, improves real-time data processing capabilities, and avoids providing outdated data and network load. Summary of the invention

[0006] The purpose of the present invention is to provide a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning to optimize the classification, caching and replacement of information content, minimize the update cost and network delay, and achieve efficient and accurate caching.

[0007] To achieve the above-mentioned object, the present invention provides the following technical solutions: a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning, wherein the network architecture of the wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning is a wireless edge cache network composed of one master base station and one slave base station; a deep Q network is deployed in the master base station, and a lightweight optimization algorithm driven by a CMAB algorithm is deployed in the slave base station; the specific steps of the wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning are as follows:

[0008] S1: In the wireless edge cache network, the slave base station sends a cache request state s of the k-period system to the master base station. k ;

[0009] S2: After receiving the feedback from the slave base station, the master base station determines the cache replacement strategy Through the DQN algorithm in deep reinforcement learning, the main base station calculates the state value s of the current k period. k and the expected discounted reward function Q(s k ,a k θ k ) determines the best action to be performed a k , which is the optimal cache replacement strategy

[0010] S3: Replacement strategy based on optimal cache content The slave base station caches the content set and the state value s of the k-period k Make updates;

[0011] S4: The main base station uses the CMAB algorithm to extract and analyze the characteristic information of users and cache information, and calculates the predicted reward UCB for each content; selects the request queue corresponding to the content with the largest UCB value for multicast, that is, satisfies The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, and s i is the uncertainty measure of content i; the actual reward for receiving user feedback After that, the model parameters are updated to dynamically adjust the multicast content.

[0012] Preferably, the S1 specifically includes:

[0013] The wireless edge cache network comprises a content library C={1,2,…,N} having N fresh contents, all of which are accessed by the primary base station;

[0014] The system status sent by the slave base station to the master base station The index set C of the cached content in the base station cache with a size of M (M<N) at time t t , the information age of the multicast content u at time t The user request queue Q for content i at time t i Number of pending requests Content index j to be downloaded, reward r for period k-1 k-1 .

[0015] Preferably, in S2, a deep reinforcement learning algorithm is deployed in the primary base station to generate the optimal cache replacement strategy. Here are the steps:

[0016] F1: The master base station receives status from the slave base station

[0017] F2: The master base station uses ε k -greedy strategy chooses action a k , that is, with probability ε k Random from Choose an action in the k ,a k θ k ) maximizes the action a k ;

[0018] F3: Set the selected action to a k Send to the slave base station, that is, send the optimal content replacement strategy

[0019] Preferably, in S3, the slave base station performs the cached content set and the state value s of the k-period. k The specific update is as follows: after the slave base station downloads the uncached content from the master base station according to the content replacement strategy, it will replace a content in the cached content set at time t; at the same time, the age of the content u at time t in the system is updated. The reward r in period k k .

[0020] Preferably, the S4 specifically comprises the following steps:

[0021] H1: The user request queue Q for content i at time t extracted from the base station i Number of pending requests Information age of multicast content i at time t Content feature vector And calculate the predicted reward for each content where θ i is the weight parameter, sim(u t ,u j ) is the requesting user u at time t t With historical request user u j The similarity of For history request user u j Feedback reward for content i;

[0022] H2: The slave base station uses the Lin-UCB strategy to select content a t , select the content with the largest UCB, that is, satisfying The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, and s i is the uncertainty measure of content i;

[0023] H3: Multicast selected content a t Corresponding to the request queue, users provide feedback based on the received content; the collected user feedback is used as the actual reward and quantified into a ternary indicator Subsequently, the system parameters are updated to optimize and adjust the multicast system.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) This paper proposes a two-stage optimization strategy, where each stage focuses on different system components and goals. The first stage uses the CMAB method to schedule multicast in the slave base station, and the second stage uses deep reinforcement learning to determine the content replacement strategy in the master base station.

[0026] (2) The present invention deploys a lightweight optimization algorithm driven by the CMAB algorithm at the slave base station, dynamically balances exploration and utilization based on context information, makes optimal multicast transmission decisions, and makes timely adjustments through the user feedback mechanism, taking into account both system performance and content distribution flexibility. The invention improves network performance and resource utilization, and updates cache content in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A flow chart of a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning of the present invention;

[0028] Figure 2 This is a model diagram of the wireless edge cache dynamic optimization strategy system based on contextual multi-arm machine and deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The experimental methods described in the following embodiments are all conventional methods unless otherwise specified.

[0030] The present invention provides a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning, which comprehensively considers cache service, information age, update cost and delay to jointly measure the cacheable value of information content. It is applied to a wireless edge cache system with limited base station (BS) cache capacity, access rate and average queuing delay of user requests. A combined framework of slave base station and master base station is proposed, and relevant optimization algorithms are deployed to solve the joint problem of optimal content update of information age and multicast scheduling in smart transportation systems.

[0031] like Figure 1 As shown, a flow chart of a wireless edge cache dynamic optimization strategy based on a contextual multi-arm machine and deep reinforcement learning is provided in an embodiment of the present invention. The method includes the following steps:

[0032] Step 1: Send the cache request status s of the k-period system from the base station to the master base station k , including the index set C of the cached content from the base station cache t , the age of the multicast content u at time t The user request queue Q for content i at time t i Number of pending requests The reward r in period k-1 k-1 ;

[0033] Step 2: The master base station decides the cache replacement strategy after receiving feedback from the slave base station Through the deep reinforcement learning (DRL) algorithm, the main base station can calculate the state value s of the current k period. k and the expected discounted reward function Q(s k ,a k θ k ) determines the best action to be performed ak , which is the optimal cache replacement strategy

[0034] Step 3: Replace the content based on the optimal cache From the base station, the cached content set and the state value s of k period k Make updates;

[0035] Step 4: Use the CMAB algorithm from the base station to extract and analyze the characteristic information of users and cache information, and calculate the upper confidence bound (UCB) of the predicted reward for each content. Select the request queue corresponding to the content with the largest UCB value for multicast, that is, satisfy The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, and s i is the uncertainty measure of content i. The actual reward for receiving user feedback After that, the model parameters are updated to dynamically adjust the multicast content.

[0036] In an embodiment, Figure 2 As shown, the network architecture considered in the present invention in step 1 is a wireless edge cache network consisting of a master base station and a slave base station, which is applied to a smart transportation system. The network includes a content library C = {1, 2, ..., N} with N fresh contents, all of which can be accessed by the master base station. The system status sent by the slave base station to the master base station The index set C of the cached content in the base station cache with a size of M (M<N) at time t t , the information age of the multicast content u at time t The user request queue Q for content i at time t i Number of pending requests Content index j to be downloaded, reward r for period k-1 k-1 . It can be expressed as:

[0037]

[0038] where we define a binary random variable Indicates whether the content is multicast during the time slot and Indicates that the content has been multicast. It enters the request queue Q at time slot t i The number of requests, B i is the request queue Q i We assume that B i Large enough to prevent any requests from being lost.

[0039] In step 2, the deep reinforcement learning algorithm described in the present invention is used to generate the optimal cache content replacement strategy, which is divided into three stages, specifically including:

[0040] Step 2-1: The master base station receives the status from the slave base station

[0041] Step 2-2: The primary base station uses ε k -greedy strategy chooses action a k , that is, with probability ε k Random from Choose an action, otherwise choose to make Q(s k ,a k θ k ) maximizes the action a k , that is, a k = arg min a Q (k) (s k ,a;θ k );

[0042] Use the Q-Learning algorithm to calculate each state s k Next action k The total expected discounted reward, i.e. the state-action value function, is expressed as:

[0043]

[0044] The estimate of the optimal Q-value function at epoch k is updated as follows, where ζ is the learning rate:

[0045]

[0046] As training progresses, the exploration probability ε k It will gradually decrease, so that the main base station will choose the optimal cache replacement action that can minimize the average information age of the content.

[0047] Step 2-3: Select the action a k Sent to the slave base station, that is, sending the optimal content replacement strategy

[0048] In the embodiment, in step 3, after the slave base station downloads the uncached content from the master base station according to the content replacement strategy, it will replace a content in the cached content set at time t. In order to better analyze the cache value of each data content, it is necessary to calculate the average information age of each request delivery, which can be expressed as:

[0049]

[0050]

[0051] Among them, the binary random variable is whether content i has been updated (or downloaded) at time t, and Indicates that content i has been updated (or downloaded) at time t; Defined as the request queue Q at the end of time t i The information age of content i received by all pending requests in represents the average information age of the content delivered per request in order to minimize the weight. Assume that In the case of Indicates the action to be taken. means that content i is discarded after multicast, while This means that content i will replace the existing content j∈C t Therefore, content set C t The evolution of can be characterized as:

[0052]

[0053] At the same time, update the information age of content u at time t in the system The reward r in period k k The specific update formula is as follows:

[0054]

[0055]

[0056] Among them, if That is, content i is not updated (or downloaded) at time t, and the information age is The present invention imposes a limit on the update rate of the slave base station, making its upper limit a constant D, as shown below:

[0057]

[0058] In addition, the present invention also limits the user request queue Q i The average delay of pending requests in , which is the upper bound of the average queue length, is as follows, where d i Indicates the average waiting time threshold in the content i request queue:

[0059]

[0060] In step 4, a lightweight optimization algorithm driven by the CMAB algorithm is deployed at the secondary base station to arrange multicast transmission. The specific steps include:

[0061] Step 4-1: Extract content feature vector from base station The user request queue Q for content i at time t i Number of pending requests Information age of multicast content i at time t Calculate the expected value of the predicted reward for each content where θ i is the weight parameter, sim(u t ,u j ) is the requesting user u at time t t With historical request user u j The similarity reflects the semantic similarity between the current content and the user's historical request information. It can be calculated based on the user's interests, preferences, behaviors, geographic location and other information, so that the system can combine the historical behaviors of similar users to recommend content more accurately. For history request user u j Feedback reward for content i.

[0062] To ensure that the contribution of each element in the feature vector to the model is balanced, the results are normalized so that the predicted reward value is scaled to the [-1,1] interval and the normalized expected value of the predicted reward is It is expressed as:

[0063]

[0064]

[0065]

[0066] Step 4-2: Calculate from the base station in is the uncertainty measure of content i, Σ i is the covariance matrix of content i and is initialized to the identity matrix Ι.

[0067] β is the exploration coefficient, which is used to adjust the balance between exploration and utilization. Initially, β can be set to 0.5, and then gradually adjusted according to the actual performance of the model. By testing different β values ​​and observing different system performances, we can gradually approach the optimal value. Use Lin-UCB strategy to select content That is, select the content with the largest UCB;

[0068] Step 4-3: Multicast the selected content t Corresponding to the request queue, users provide feedback based on the received content, such as collecting the completion, likes, and comment rates of the multicast content. The collected user feedback is used as the actual reward and quantified into a ternary indicator and Indicates the user's response to the multicast content a tSatisfaction, such as user completion rate, like rate or comment rate. Then, the covariance matrix in the system is updated Weight vector The specific update formula is as follows:

[0069]

[0070]

[0071] The entire program is executed in a loop in the above order. The slave base station is responsible for real-time decision-making, and the master base station is responsible for obtaining the long-term optimal cache replacement strategy through DRL learning. The two work together to gradually optimize the cache replacement strategy until the DRL model parameter θ k Convergence. DRL training will repeat the following steps until convergence:

[0072] 1) will (s k ,a k ,r k ,s k+1 ) That is, the current state, action, reward, and next state are stored in the experience pool Β.

[0073] 2) Randomly extract a small batch of samples from the experience pool Β and calculate the target value of each sample i And use gradient descent method to update the Q network parameters θ k , to minimize y i and Q(s i ,a i θ i ), the model parameter θ k The gradient of θ The update formula is expressed as:

[0074] L(θ k )=θ k+1 +ζ[y k -Q(s k ,a k θ k )]δ θ Q(s k ,a k θ k )

[0075] 3) In order to improve the stability of DRL training and avoid divergence in the training process, the master base station is updated every k training time steps, i.e., k t moments Used to calculate the target value.

[0076] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning, characterized by: The network architecture of the wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning is a wireless edge cache network composed of one master base station and one slave base station; a deep Q network is deployed in the master base station, and a lightweight optimization algorithm driven by the CMAB algorithm is deployed in the slave base station; the specific steps of the wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning are as follows: S1: In the wireless edge cache network, the slave base station sends a cache request state s of the k-period system to the master base station. k ; S2: After receiving the feedback from the slave base station, the master base station determines the cache replacement strategy Through the DQN algorithm in deep reinforcement learning, the main base station calculates the state value s of the current k period. k and the expected discounted reward function Q(s k ,a k θ k ) determines the best action to be performed a k , which is the optimal cache replacement strategy S3: Replacement strategy based on optimal cache content The slave base station caches the content set and the state value s of the k-period k Make updates; S4: The main base station uses the CMAB algorithm to extract and analyze the characteristic information of users and cache information, and calculates the predicted reward UCB for each content; selects the request queue corresponding to the content with the largest UCB value for multicast, that is, satisfies The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, and s i is the uncertainty measure of content i; the actual reward for receiving user feedback After that, the model parameters are updated to dynamically adjust the multicast content.

2. According to claim 1, a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning is characterized by: The S1 is specifically: The wireless edge cache network comprises a content library C={1,2,…,N} having N fresh contents, all of which are accessed by the primary base station; The system status sent by the slave base station to the master base station The index set C of the cached content in the base station cache with a size of M (M<N) at time t t , the information age of the multicast content u at time t The user request queue Q for content i at time t i Number of pending requests Content index j to be downloaded, reward r for period k-1 k-1 .

3. According to claim 1, a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning is characterized in that: In S2, a deep reinforcement learning algorithm is deployed in the primary base station to generate the optimal cache replacement strategy. Here are the steps: F1: The master base station receives status from the slave base station F2: The master base station uses ε k -greedy strategy chooses action a k , that is, with probability ε k Random from Choose an action in the k ,a k θ k ) maximizes the action a k ; F3: Set the selected action to a k Send to the slave base station, that is, send the optimal content replacement strategy 4. According to claim 1, a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning is characterized in that: In S3, the slave base station calculates the cached content set and the state value s of the k-period. k The specific update is as follows: after the slave base station downloads the uncached content from the master base station according to the content replacement strategy, it will replace a content in the cached content set at time t; at the same time, the age of the content u at time t in the system is updated. The reward r in period k k .

5. According to claim 1, a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning is characterized in that: The S4 specifically includes the following steps: H1: The user request queue Q for content i at time t extracted from the base station i Number of pending requests Information age of multicast content i at time t Content feature vector And calculate the predicted reward for each content where θ i is the weight parameter, sim(u t ,u j ) is the requesting user u at time t t With historical request user u j The similarity of For history request user u j Feedback reward for content i; H2: The slave base station uses the Lin-UCB strategy to select content a t , select the content with the largest UCB, that is, satisfying The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, and s i is the uncertainty measure of content i; H3: Multicast selected content a t Corresponding to the request queue, users provide feedback based on the received content; the collected user feedback is used as the actual reward and quantified into a ternary indicator Subsequently, the system parameters are updated to optimize and adjust the multicast system.

Images