A dynamic optimization strategy for wireless edge caching based on contextual multi-arm machine and deep reinforcement learning

Abstract

The present invention discloses a wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning, which belongs to the field of intersection technology of artificial intelligence and edge computing. The present invention steps: ① In the intelligent transportation scenario, the slave base station sends the cache status index set s of the cached content in the slave base station cache to the master base station. k ;②Through DRL, the master base station can cache the state value s according to the current k period k and the expected discounted reward function Q(s k ,a k θ k ) decides the action to be performed a k , thereby obtaining the optimal cache replacement strategy. ③ The base station updates the cached content set. ④ The base station uses the CMAB algorithm to extract and analyze user and cached information characteristics to calculate the predicted reward (UCB) for each content. The request queue corresponding to the content that maximizes the UCB value is selected for multicast. After receiving actual reward feedback from users, the model parameters are updated to dynamically adjust the multicast content. This method ensures that users receive the latest content, improves real-time data processing capabilities, and avoids serving stale data.

Description

Technical Field

[0001] The present invention relates to the field of the intersection of artificial intelligence and edge computing, and specifically to a dynamic optimization strategy for wireless edge caching based on contextual multi-arm machines and deep reinforcement learning. Background Art

[0002] Contextual Multi-Armed Bandit (CMAB) is an online learning algorithm used to gradually optimize decisions in an environment with contextual information. Compared with the traditional multi-armed bandit model, CMAB technology enables the decision-making process to dynamically adapt to changes in the environment by introducing context (such as user behavior, environmental status, etc., in this invention, the age of content information, request queue size). In each round of decision-making, the algorithm not only selects the optimal action based on the historical performance of the current optional actions (i.e., different arms), but also considers contextual information, and learns and adjusts based on the rewards obtained. The core challenge of CMAB is to balance exploration and utilization: it is necessary to explore uncertain actions to obtain more information, and to use existing knowledge to maximize immediate and long-term returns.

[0003] Deep reinforcement learning (DRL) combines reinforcement learning and deep learning, using reinforcement learning to define problems and optimization objectives, and deep learning to solve problems such as state representation, policy representation, and value function modeling. The deep Q network (DQN) is an algorithm that combines the Q-Learning algorithm and a deep neural network (DNN), i.e., a Q-Learning algorithm based on deep learning. Q-Learning can perceive the reward function based on parameters such as the current state of the content and the age of the information, and achieve the optimal caching strategy by continuously updating the Q value. This end-to-end optimization can better adapt to different scenarios and conditions, and can help improve the overall system performance to adapt to changing network conditions and a large number of user requests.

[0004] High user demand and expectations for intelligent transportation systems highlight the need for regularly updating cached content on edge devices. As the need to manage smart devices and mobile internet services within intelligent transportation systems grows, traditional caching faces significant limitations in capturing popular content during low-traffic periods, making it difficult to ensure safety, operational efficiency, and comfort within the transportation system.

[0005] This invention combines contextual multi-arm machines with deep reinforcement learning algorithms to optimize caching strategies and multicast scheduling in edge computing environments. This improves network performance and resource utilization, allowing for timely updates of cached content while also factoring in the "age" of different content or data, i.e., its freshness. This ensures that users receive the latest content, improves real-time data processing capabilities, and avoids serving stale data and network overload. Summary of the Invention

[0006] The purpose of the present invention is to provide a dynamic optimization strategy for wireless edge caching based on contextual multi-arm machines and deep reinforcement learning to optimize the classification, caching and replacement of information content, minimize update costs and network latency, and achieve efficient and accurate caching.

[0007] To achieve the above objectives, the present invention provides the following technical solutions: a dynamic optimization strategy for wireless edge cache based on contextual multi-arm machine and deep reinforcement learning, wherein the network architecture of the dynamic optimization strategy for wireless edge cache based on contextual multi-arm machine and deep reinforcement learning is a wireless edge cache network consisting of one master base station and one slave base station; a deep Q network is deployed in the master base station, and a lightweight optimization algorithm driven by the CMAB algorithm is deployed in the slave base station; the specific steps of the dynamic optimization strategy for wireless edge cache based on contextual multi-arm machine and deep reinforcement learning are as follows:

[0008] S1: In the wireless edge cache network, the slave base station sends the cache request status s of the k-period system to the master base station. k ;

[0009] S2: After receiving the feedback from the slave base station, the master base station decides the cache replacement strategy Through the DQN algorithm in deep reinforcement learning, the master base station is based on the state value s of the current k period. k and the expected discounted reward function Q(s k ,a k θ k ) determines the optimal action a to be performed k , that is, the optimal cache replacement strategy

[0010] S3: Replacement strategy based on optimal cache content The slave base station caches the content set and the state value s of the k-period k Make updates;

[0011] S4: The main base station uses the CMAB algorithm to extract and analyze the characteristic information of users and cache information, and calculates the predicted reward UCB for each content; selects the request queue corresponding to the content with the largest UCB value for multicast, that is, satisfies The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, and s i is the uncertainty measure of content i; the actual reward for receiving user feedback After that, the model parameters are updated to dynamically adjust the multicast content.

[0012] Preferably, the S1 specifically includes:

[0013] The wireless edge cache network includes a content library C = {1, 2, ..., N} with N fresh contents, all of which are accessed by the primary base station;

[0014] The system status sent by the secondary base station to the primary base station The index set C of cached content in the base station cache with a size of M (M<N) at time t t , the information age of the multicast content u at time t User request queue Q for content i at time t i Number of pending requests The content index j to be downloaded, the reward r of period k-1 k-1 .

[0015] Preferably, in S2, a deep reinforcement learning algorithm is deployed in the primary base station to generate the optimal cache replacement strategy. Here are the steps:

[0016] F1: The master base station receives status from the slave base station

[0017] F2: The primary base station uses ε k -greedy strategy chooses action a k , that is, with probability ε k Random from Choose an action, otherwise, choose to make Q(s k ,a k θ k ) Maximize action a k ;

[0018] F3: Set the selected action to a k Send to the slave base station, that is, send the optimal content replacement strategy

[0019] Preferably, in said S3, said slave base station calculates the cached content set and the state value s of k period k The update is specifically as follows: after the slave base station downloads the uncached content from the master base station according to the content replacement strategy, it will replace a content in the cached content set at time t; at the same time, the age of the content u at time t in the system is updated. The reward r in period k k .

[0020] Preferably, the S4 specifically includes the following steps:

[0021] H1: The user request queue Q for content i at time t extracted from the base station i Number of pending requests Information age of multicast content i at time t Content feature vector And calculate the predicted reward for each content where θ i is the weight parameter, sim(u t ,u j ) is the requesting user u at time t t With historical request user u j The similarity of For history request user u j Feedback reward for content i;

[0022] H2: The slave base station uses the Lin-UCB strategy to select content a t , select the content with the largest UCB, that is, satisfy The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, s i is the uncertainty measure of content i;

[0023] H3: Multicast selected content a t Corresponding to the request queue, users provide feedback based on the received content; the collected user feedback is used as the actual reward and quantified into a ternary indicator Subsequently, the system parameters are updated and the multicast system is optimized and adjusted.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) This paper proposes a two-stage optimization strategy, where each stage focuses on different system components and objectives. The first stage uses the CMAB method to schedule multicast at the slave base station, and the second stage uses deep reinforcement learning to determine the content replacement strategy at the master base station.

[0026] (2) This invention deploys a lightweight optimization algorithm driven by the CMAB algorithm at the slave base station. This algorithm dynamically balances exploration and utilization of contextual information to make optimal multicast transmission decisions. This algorithm then makes timely adjustments through user feedback, balancing system performance and content distribution flexibility. This invention improves network performance and resource utilization, and allows for timely updates of cached content. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning of the present invention;

[0028] Figure 2 This is a model diagram of the wireless edge cache dynamic optimization strategy system based on contextual multi-arm machine and deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The experimental methods described in the following examples are all conventional methods unless otherwise specified.

[0030] The present invention provides a dynamic optimization strategy for wireless edge caching based on contextual multi-arm machines and deep reinforcement learning. This strategy comprehensively considers cache service, information age, update cost, and latency to jointly measure the cacheability of information content. This strategy is applied to wireless edge caching systems with limited base station (BS) cache capacity, access rate, and average queuing delay of user requests. A combined framework of slave and master base stations is proposed, and relevant optimization algorithms are deployed to solve the joint problem of optimal content update and multicast scheduling for information age in intelligent transportation systems.

[0031] like Figure 1 As shown, an embodiment of the present invention provides a flow chart of a wireless edge cache dynamic optimization strategy based on a contextual multi-arm machine and deep reinforcement learning, the method comprising the following steps:

[0032] Step 1: Send the cache request status s of the k-period system from the base station to the master base station k , including the index set C of cached content from the base station cache t , the age of the multicast content u at time t User request queue Q for content i at time t i Number of pending requests The reward r in period k-1 k-1 ;

[0033] Step 2: The master base station decides the cache replacement strategy after receiving feedback from the slave base station Through the deep reinforcement learning (DRL) algorithm, the master base station can calculate the state value s of the current k period. k and the expected discounted reward function Q(s k ,a k θ k ) determines the optimal action a to be performedk , that is, the optimal cache replacement strategy

[0034] Step 3: Replace the content based on the optimal cache strategy From the base station, the cached content set and the state value s of k period k Make updates;

[0035] Step 4: Use the CMAB algorithm from the base station to extract and analyze the characteristic information of users and cache information, and calculate the upper confidence bound (UCB) of the predicted reward for each content. Select the request queue corresponding to the content with the largest UCB value for multicast, which satisfies The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, and s i is the uncertainty measure of content i. The actual reward for receiving user feedback After that, the model parameters are updated to dynamically adjust the multicast content.

[0036] In an embodiment, Figure 2 As shown in step 1, the network architecture considered in the present invention is a wireless edge cache network consisting of a master base station and a slave base station, which is applied to the intelligent transportation system. The network includes a content library C = {1, 2, ..., N} with N fresh contents, all of which can be accessed by the master base station. The system status sent by the slave base station to the master base station The index set C of cached content in the base station cache with a size of M (M<N) at time t t , the information age of the multicast content u at time t User request queue Q for content i at time t i Number of pending requests The content index j to be downloaded, the reward r of period k-1 k-1 . It can be expressed as:

[0037]

[0038] where we define a binary random variable Indicates whether the content is multicast during the time slot and Indicates that the content has been multicast. Entering the request queue Q at time slot t i The number of requests, B i is the request queue Q i We assume that B i Large enough to prevent any requests from being lost.

[0039] In step 2, the deep reinforcement learning algorithm described in the present invention is used to generate the optimal cache content replacement strategy, which is divided into three stages, specifically including:

[0040] Step 2-1: The master base station receives status from the slave base station

[0041] Step 2-2: The primary base station uses ε k -greedy strategy chooses action a k , that is, with probability ε k Random from Choose an action, otherwise choose to make Q(s k ,a k θ k ) Maximize action a k , that is, a k =arg min a Q (k) (s k ,a;θ k );

[0042] Use Q-Learning algorithm to calculate each state s k Next action a k The total expected discounted reward, i.e. the state-action value function, is expressed as:

[0043]

[0044] The estimate of the optimal Q-value function at epoch k is updated as follows, where ζ is the learning rate:

[0045]

[0046] As training progresses, the exploration probability ε k It will gradually decrease, making the main base station choose more optimal cache replacement actions that can minimize the average information age of the content.

[0047] Step 2-3: Select the action a k Sent to the slave base station, that is, sending the optimal content replacement strategy

[0048] In this embodiment, in step 3, after the secondary base station downloads the uncached content from the primary base station according to the content replacement policy, it will replace a content in the cached content set at time t. To better analyze the cache value of each data content, it is necessary to calculate the average information age of each request delivery, which can be expressed as:

[0049]

[0050]

[0051] Among them, the binary random variable is whether content i has been updated (or downloaded) at time t, and Indicates that content i has been updated (or downloaded) at time t; Defined as the request queue Q at the end of time t i The information age of content i received in all pending requests; represents the average age of information delivered per request in order to minimize the weight. In the case of Indicates the action to be taken. means that content i is discarded after multicast, while It means that content i will replace the existing content j∈C t Therefore, content set C t The evolution of can be characterized as follows:

[0052]

[0053] At the same time, update the information age of content u at time t in the system The reward r in period k k The specific update formula is as follows:

[0054]

[0055]

[0056] Among them, if That is, at time t, content i is not updated (or downloaded), and the information age is The present invention imposes a limit on the update rate of the slave base station, with its upper limit being a constant D, as shown below:

[0057]

[0058] In addition, the present invention also limits the user request queue Q i The average delay of pending requests in , which is the upper bound of the average queue length, is as follows, where d i Indicates the average waiting time threshold in the request queue for content i:

[0059]

[0060] In step 4, a lightweight optimization algorithm driven by the CMAB algorithm is deployed at the secondary base station to schedule multicast transmission. The specific steps include:

[0061] Step 4-1: Extract content feature vector from base station The user request queue Q for content i at time t i Number of pending requests Information age of multicast content i at time t Calculate the expected value of the predicted reward for each content where θ i is the weight parameter, sim(u t ,u j ) is the requesting user u at time t t With historical request user u j The similarity reflects the semantic similarity between the current content and the user's historical request information. It can be calculated based on the user's interests, preferences, behaviors, geographic location and other information, so that the system can combine the historical behaviors of similar users to recommend content more accurately. For history request user u j Feedback reward for content i.

[0062] To ensure that the contribution of each element in the feature vector to the model is balanced, the results are normalized so that the predicted reward value is scaled to the [-1,1] interval and the normalized expected value of the predicted reward is Expressed as:

[0063]

[0064]

[0065]

[0066] Step 4-2: Calculate from the base station in is the uncertainty measure of content i, Σ i is the covariance matrix of content i and is initialized to the identity matrix Ι.

[0067] β is the exploration coefficient, which is used to adjust the balance between exploration and exploitation. Initially, β can be set to 0.5, and then gradually adjusted according to the actual performance of the model. By experimenting with different β values ​​and observing different system performances, we can gradually approach the optimal value. Use the Lin-UCB strategy to select content That is, select the content with the largest UCB;

[0068] Step 4-3: Multicast the selected content t Corresponding to the request queue, users provide feedback based on the received content, such as the completion rate, likes, and comment rate of the multicast content. The collected user feedback is used as the actual reward and quantified into a ternary indicator. and Indicates the user's response to the multicast content a tSatisfaction, such as user completion rate, like rate or comment rate. Then, update the covariance matrix in the system Weight vector The specific update formula is as follows:

[0069]

[0070]

[0071] The entire program is executed in a loop according to the above sequence. The slave base station is responsible for real-time decision-making, and the master base station is responsible for obtaining the long-term optimal cache replacement strategy through DRL learning. The two work together to gradually optimize the cache replacement strategy until the DRL model parameter θ k Convergence. DRL training will repeat the following steps until convergence:

[0072] 1) will (s k ,a k ,r k ,s k+1 ) That is, the current state, action, reward, and next state are stored in the experience pool B.

[0073] 2) Randomly extract a small batch of samples from the experience pool Β and calculate the target value of each sample i And use gradient descent method to update the Q network parameters θ k , to minimize y i and Q(s i ,a i θ i ), the gap between the model parameters θ k The gradient of θ The update formula is expressed as:

[0074] L(θ k )=θ k+1 +ζ[y k -Q(s k ,a k θ k )]δ θ Q(s k ,a k θ k )

[0075] 3) In order to improve the stability of DRL training and avoid divergence of the training process, the main base station is updated every k training time steps, that is, k t moments Used to calculate target values.

[0076] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A dynamic optimization strategy for wireless edge caching based on contextual multi-arm machines and deep reinforcement learning, characterized by: The network architecture of the wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning is a wireless edge cache network consisting of one master base station and one slave base station; a deep Q network is deployed on the master base station, and a lightweight optimization algorithm driven by the CMAB algorithm is deployed on the slave base station; the specific steps of the wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning are as follows: S1: In the wireless edge cache network, the slave base station sends the cache request status s of the k-period system to the master base station. k ; S2: After receiving the feedback from the slave base station, the master base station decides the cache replacement strategy Through the DQN algorithm in deep reinforcement learning, the master base station is based on the state value s of the current k period. k and the expected discounted reward function Q(s k ,a k θ k ) determines the optimal action a to be performed k , that is, the optimal cache replacement strategy S3: Replacement strategy based on optimal cache content The slave base station caches the content set and the state value s of the k-period k Make updates; S4: The main base station uses the CMAB algorithm to extract and analyze the characteristic information of users and cache information, and calculates the predicted reward UCB for each content; selects the request queue corresponding to the content with the largest UCB value for multicast, that is, satisfies The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, and s i is the uncertainty measure of content i; the actual reward for receiving user feedback After that, the model parameters are updated to dynamically adjust the multicast content.

2. The wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning according to claim 1, characterized in that: The S1 specifically includes: The wireless edge cache network includes a content library C = {1, 2, ..., N} with N fresh contents, all of which are accessed by the primary base station; The system status sent by the secondary base station to the primary base station The index set C of cached content in the base station cache with a size of M (M<N) at time t t , the information age of the multicast content u at time t User request queue Q for content i at time t i Number of pending requests The content index j to be downloaded, the reward r of period k-1 k-1 .

3. The wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning according to claim 1, characterized in that: In S2, a deep reinforcement learning algorithm is deployed in the primary base station to generate the optimal cache replacement strategy. Here are the steps: F1: The master base station receives status from the slave base station F2: The primary base station uses ε k -greedy strategy chooses action a k , that is, with probability ε k Random from Choose an action, otherwise, choose to make Q(s k ,a k θ k ) Maximize action a k ; F3: Set the selected action to a k Send to the slave base station, that is, send the optimal content replacement strategy 4. The wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning according to claim 1, characterized in that: In S3, the slave base station calculates the cached content set and the state value s of k periods. k The update is specifically as follows: after the slave base station downloads the uncached content from the master base station according to the content replacement strategy, it will replace a content in the cached content set at time t; at the same time, the age of the content u at time t in the system is updated. The reward r in period k k .

5. The wireless edge cache dynamic optimization strategy based on contextual multi-arm machine and deep reinforcement learning according to claim 1, characterized in that: The S4 specifically includes the following steps: H1: The user request queue Q for content i at time t extracted from the base station i Number of pending requests Information age of multicast content i at time t Content feature vector And calculate the predicted reward for each content where θ i is the weight parameter, sim(u t ,u j ) is the requesting user u at time t t With historical request user u j The similarity of For history request user u j Feedback reward for content i; H2: The slave base station uses the Lin-UCB strategy to select content a t , select the content with the largest UCB, that is, satisfy The content index of is the normalized expected value of the predicted reward, β is the exploration coefficient, s i is the uncertainty measure of content i; H3: Multicast selected content a t Corresponding to the request queue, users provide feedback based on the received content; the collected user feedback is used as the actual reward and quantified into a ternary indicator Subsequently, the system parameters are updated and the multicast system is optimized and adjusted.

Images