An Edge Collaborative Caching Method and Device Based on Deep Reinforcement Learning

Abstract

The present invention provides an edge collaborative caching method and device based on deep reinforcement learning, and the steps of the method include: the edge node calls data based on a data request, divides the data into multiple data blocks, and marks them; constructs a first state vector, inputs it into a gate loop unit, obtains a first hidden state vector, inputs it into a target network pre-trained by the edge node, and outputs a first action; based on the first action, changes the data block stored in the edge node, and constructs a second state vector based on the edge node of the changed data block and another called data block, and further obtains a second hidden state vector based on the gate loop unit; inputs the second hidden state vector into the value network of the central node to obtain an estimated action, outputs an estimated value corresponding to the estimated action, and calculates a target value based on the estimated value; obtains a loss function value based on the target value, updates the network parameters in the value network, and synchronizes the network parameters in the value network to the target network.

Description

Technical Field

[0001] The present invention relates to the field of distributed processing technology, and in particular to an edge collaborative caching method and device based on deep reinforcement learning. Background Art

[0002] With the advent of the 5G era, an increasing number of smart devices are being added to the Industrial Internet. These numerous terminal devices, deployed in diverse locations, generate a massive influx of data access requests, significantly impacting not only network communications but also the cloud platforms that provide the data. Edge computing can help address the challenges posed by massive data access. Edge computing involves deploying platforms with computing, storage, and network communication capabilities close to end users, providing local services to selected end users. This provides faster response times and reduces pressure on the cloud. Caching data in edge nodes close to end users reduces data access latency and backhaul traffic to the cloud. Limited by the capacity and processing power of edge nodes, we need to select appropriate cache content and replacement strategies for edge nodes to achieve higher efficiency.

[0003] At the same time, most existing edge caching methods only consider the caching strategy of a single node, lacking collaboration between different edge nodes within the same region. In fact, edge nodes within the same region can also access each other's cached data, which can also reduce latency and data consumption compared to obtaining data from the cloud. This requires selecting appropriate methods to achieve collaboration between edge nodes, fully utilizing their caching capabilities and improving overall efficiency. However, existing technologies lack methods for achieving edge node collaboration and thus improving their caching capabilities. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides an edge collaborative caching method based on deep reinforcement learning to eliminate or improve one or more defects in the prior art.

[0005] One aspect of the present invention provides an edge collaborative caching method based on deep reinforcement learning. The method is applied to a distributed system including a central node and multiple edge nodes. The method comprises the following steps:

[0006] The edge node receives the data request from the terminal device, calls the data for the terminal device based on the data request, divides the data into multiple data blocks based on the data size, and marks each data block;

[0007] Construct a first state vector based on the data block stored in the current edge node and one of the called data blocks, input the first state vector into the gate recurrent unit to obtain a first hidden state vector, input the first hidden state vector into the target network pre-trained by the edge node, and output a first action;

[0008] Based on the first action, a data block stored in the edge node is changed, and a second state vector is constructed based on the edge node of the changed data block and another called data block, and a second hidden state vector is further obtained based on the gate recurrent unit;

[0009] Input the second hidden state vector into the value network of the central node to obtain an estimated action, output an estimated value corresponding to the estimated action, and calculate the target value based on the estimated value;

[0010] The loss function value is calculated based on the target value, the network parameters in the value network are updated based on the loss function value, and the network parameters in the value network are synchronized to the target network of each edge node.

[0011] Adopting the above scheme, this scheme is based on the distributed system that can receive data from each edge node, thereby obtaining the actions taken by the edge node and the status after taking the action, and transmitting the status of each edge node after taking the action to the central node, training the value network of the central node, and then synchronizing the value network to the target network. It can sum up the data of all edge nodes, enable the edge nodes to cooperate with each other, and improve the accuracy of the value network.

[0012] In some embodiments of the present invention, the step of dividing the data into a plurality of data blocks based on the data size and marking each data block is to number the data blocks using one-hot encoding.

[0013] In some embodiments of the present invention, the step of constructing a first state vector based on the data block currently stored in the edge node and one of the called data blocks includes:

[0014] The number of one of the called data blocks and the number of the data block stored in the current edge node are jointly constructed into an initial vector;

[0015] The initial vector is input into a preset neural network model to obtain a first state vector.

[0016] In some embodiments of the present invention, the step of inputting the first state vector into the gate recurrent unit to obtain the first hidden state vector includes:

[0017] In the gate recurrence unit, a reset gate vector and an update gate vector are calculated based on the first hidden state vector, a candidate hidden state vector is calculated based on the reset gate vector, and a first hidden state vector is obtained based on the candidate hidden state vector, the reset gate vector, and the update gate vector.

[0018] In some embodiments of the present invention, the reset gate vector and the update gate vector are calculated based on the first hidden state vector in the gate recurrent unit according to the following formula:

[0019]

[0020]

[0021] R represents the reset gate vector, z represents the update gate vector, σ represents the sigmoid activation function, Both represent the weight matrix of the fully connected layer, b r 、b z Both represent bias vectors, h -1 represents the first hidden state vector of the previous round, and s represents the first state vector.

[0022] In some embodiments of the present invention, the candidate hidden state vector is calculated based on the reset gate vector according to the following formula:

[0023]

[0024] represents the candidate hidden state vector, and Both represent the weight matrix of the fully connected layer, b h represents the bias vector, h -1 Represents the first hidden state vector of the previous round, R represents the reset gate vector, * represents the Hadamard product, and s represents the first state vector.

[0025] In some embodiments of the present invention, the first hidden state vector is calculated based on the candidate hidden state vector, the reset gate vector, and the update gate vector according to the following formula:

[0026]

[0027] h represents the first hidden state vector, z represents the update gate vector, h -1 represents the first hidden state vector of the previous round, * represents the Hadamard product, represents the candidate hidden state vector.

[0028] In some embodiments of the present invention, the step of inputting the first hidden state vector into a target network pre-trained by an edge node and outputting a first action includes:

[0029] The target network outputs a value parameter corresponding to each preset action based on the input first hidden state vector, and the preset action corresponding to the maximum value parameter is the first action.

[0030] In some embodiments of the present invention, the second hidden state vector is input into the value network of the central node, and the estimated action is obtained in the same manner as the first action is obtained from the value network. An estimated value corresponding to the estimated action is output, and the estimated value is a value parameter corresponding to the estimated action. The target value is calculated based on the estimated value according to the following formula:

[0031]

[0032] r represents the reward function, represents the target value, represents the estimated value, and γ represents the discount coefficient.

[0033] In some embodiments of the present invention, the loss function value is calculated based on the target value according to the following formula:

[0034]

[0035] E represents the loss function value, Indicates the value parameter corresponding to the first action, Indicates the target value.

[0036] In some embodiments of the present invention, the step of updating the network parameters in the value network based on the loss function value is to update the network parameters in the value network using a gradient descent method based on the loss function value according to the following formula:

[0037]

[0038] w * represents the network parameters in the updated value network, w represents the network parameters in the value network before updating, η represents the learning rate, E represents the loss function value, Indicates the value parameter corresponding to the first action, Indicates the target value.

[0039] Another aspect of the present invention also provides an edge collaborative caching device based on deep reinforcement learning, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps implemented by the method described above.

[0040] Additional advantages, objects, and features of the present invention will be described in part in the following description and will become apparent to those skilled in the art after studying the following or may be learned by practice of the present invention. The objects and other advantages of the present invention may be particularly pointed out and attained in the description and drawings.

[0041] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0043] Figure 1 A schematic diagram of an embodiment of the edge collaborative caching method based on deep reinforcement learning of the present invention;

[0044] Figure 2 Schematic diagram of the distributed structure applied in the present invention;

[0045] Figure 3 It is a structural diagram of the preset neural network model;

[0046] Figure 4 A flow chart that outputs actions for a target network or value network. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0048] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0049] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0050] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0051] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0052] To solve the above problems, the present invention proposes an edge collaborative caching method based on deep reinforcement learning. The method is applied to a distributed system, and the distributed system includes a central node and multiple edge nodes. In the specific implementation process, we assume that the entire cache system is composed of several regions, and each region has several edge nodes for accepting user data access requests and actually storing cached data. At the same time, there is also a central node in each region, which is used to coordinate the work of the edge nodes in the region and does not actually store the cached data. The terminal device will make a data access request through the edge node closest to it. Since different regions will use the same caching method, we only need to consider the situation in one region, such as Figure 2 As shown;

[0053] like Figure 1 As shown, the steps of the method include:

[0054] Step S100: The edge node receives a data request from a terminal device, calls data for the terminal device based on the data request, divides the data into multiple data blocks based on the data size, and marks each data block;

[0055] In some embodiments of the present invention, the data can be called from the edge node itself, from other edge nodes, or from the cloud. As shown in the figure, the delay required for calling from the edge node itself is d edge , the total delay is d edge ; Called from other edge nodes, at this time, the data needs to be retrieved from other edge nodes to the central node and then returned to the current edge node. The total delay is d edge +2d center , where d center represents the delay between the central node and the edge node;,in case of cache miss, it is necessary to call from the cloud, retrieve the data from the cloud to the central node, and then return it to the edge node. The total delay is d edge +d center +d cloud , where d cloud represents the delay of the central node retrieving data from the cloud, d cloud >d center >d edge .

[0056] In some embodiments of the present invention, the total delay can be expressed as follows:

[0057]

[0058] We divide time into time slots of equal length, t, and each time slot is divided into two phases. In the first phase, the terminal device sends a data access request to the edge node. If the edge node has cached the data, it returns it directly to the terminal device. Otherwise, the edge node requests the data from the central node. The central node will then query whether other edge nodes in the area have cached the data. If so, it will obtain the data from the corresponding edge node and ultimately deliver it to the terminal device. If no edge node has cached the data requested by the terminal device, the central node will request the data from the cloud and return it to the terminal device. In the second phase, the edge node replaces the cached content. The data obtained by the edge node from the central node and other edge nodes in the first phase will be temporarily saved in the second phase for the decision to replace it in the second phase.

[0059] For edge node m, we use To represent the cache status of each data in time slot t. 1 indicates that data block n is cached, and 0 indicates that data block n is not cached; we use To represent the number of requests for each data block by the terminal device within time slot t.

[0060] In some embodiments of the present invention, when dividing data blocks, this solution can divide the data into sets with the same data size based on the data size.

[0061] In some embodiments of the present invention, all data that a terminal device may access is divided into blocks of equal size, each block size is q, and the total number of data blocks is N. The number of edge nodes in the region is M, and each edge node can store a maximum of C max data block.

[0062] In time slot t, the outbound traffic of all nodes is as follows:

[0063]

[0064] Therefore, the cache strategy with the minimum latency is equivalent to solving the following integer programming problem:

[0065]

[0066]

[0067] Similarly, the cache strategy that minimizes outbound traffic is equivalent to solving the following integer programming problem:

[0068]

[0069]

[0070] In some embodiments of the present invention, a method for marking data blocks may adopt a one-hot encoding method.

[0071] like Figure 4 As shown, step S200, constructing a first state vector based on the data block stored in the current edge node and one of the called data blocks, inputting the first state vector into the gate recurrent unit to obtain a first hidden state vector, inputting the first hidden state vector into the target network pre-trained in the edge node, and outputting a first action;

[0072] Figure 4 In , V and D represent the two parts of the duel network respectively.

[0073] In some embodiments of the present invention, this solution processes each data block one by one. When processing each data block, the data block and the data block stored in the current edge node can be used together to construct a first state vector.

[0074] In some embodiments of the present invention, the gated recurrent unit may be a GRU, which can capture dependencies in time series data and utilize the GRU to memorize past states, thereby improving the accuracy of the state vector.

[0075] With the above approach, the more information available for decision-making, the more likely it is to make a better decision. Ideally, we should record the state of every past step and use this comprehensive information to make the final replacement decision. However, as the system operates, the number of historical states will continue to grow, making it impossible to directly use all of them as neural network input. The introduction of the GRU allows us to condense the vast number of historical states into a fixed-length vector, effectively capturing the implicit information and connections between different states at a lower cost. Compared to using only the current state, leveraging the hidden historical state allows the rest of the model to make more optimal replacement decisions, reducing latency and data usage.

[0076] In its implementation, this solution uses the D3QN network structure, which combines the Dueling Network architecture with the Double Deep Q Network training method. Each edge node is an agent, and we apply the Double Deep Q Learning training method to multi-agent scenarios.

[0077] During the specific implementation process, both the target network and the value network adopt a duel network structure.

[0078] This solution sets up a network structure of reinforcement learning model at both edge nodes and central nodes, namely the target network and the value network. The target network and the value network have the same structure and both adopt a duel network structure.

[0079] During specific implementation, the first action may be an action of replacing any data block in the edge node with the current data block or an action of not replacing any data block in the edge node.

[0080] Step S300, based on the first action, changing the data block stored in the edge node, and constructing a second state vector based on the edge node of the changed data block and another called data block, and further obtaining a second hidden state vector based on the gate recurrent unit;

[0081] In some embodiments of the present invention, the second hidden state vector is constructed in the same manner as the first hidden state vector.

[0082] Step S400: Input the second hidden state vector into the value network of the central node to obtain an estimated action, output an estimated value corresponding to the estimated action, and calculate a target value based on the estimated value;

[0083] In some embodiments of the present invention, the estimated action is obtained from the second hidden state vector in the same manner as the first action is obtained from the first hidden state vector.

[0084] Step S500 , calculating a loss function value based on the target value, updating network parameters in the value network based on the loss function value, and synchronizing the network parameters in the value network to the target network of each edge node.

[0085] In some embodiments of the present invention, the method for synchronizing the network parameters in the value network to the target network of each edge node can be to preset training rounds. When the value network training rounds reach a preset number of times, the network parameters in the value network are synchronized to the target network of each edge node.

[0086] During the specific implementation process, the central node can simultaneously save the network parameters of the target network and the value network, synchronize the network parameters in the value network to the target network every preset training round, and then send the network parameters of the target network to each edge node.

[0087] Adopting the above scheme, this scheme is based on the distributed system that can receive data from each edge node, thereby obtaining the actions taken by the edge node and the status after taking the action, and transmitting the status of each edge node after taking the action to the central node, training the value network of the central node, and then synchronizing the value network to the target network. It can sum up the data of all edge nodes, enable the edge nodes to cooperate with each other, and improve the accuracy of the value network.

[0088] In some embodiments of the present invention, the step of dividing the data into a plurality of data blocks based on the data size and marking each data block is to number the data blocks using one-hot encoding.

[0089] In some embodiments of the present invention, the step of constructing a first state vector based on the data block currently stored in the edge node and one of the called data blocks includes:

[0090] The number of one of the called data blocks and the number of the data block stored in the current edge node are jointly constructed into an initial vector;

[0091] The initial vector is input into a preset neural network model to obtain a first state vector.

[0092] In some embodiments of the present invention, the initial vector can be specifically expressed as (cached, candidate), where cached represents the one-hot encoding number of the data block currently cached by the edge node, there may be multiple data blocks in the edge node, and candidate represents the one-hot encoding number of the data block to be processed.

[0093] like Figure 3 As shown, the preset neural network model includes an input layer, at least one hidden layer and an output layer. The total number of data blocks is N. The x of the input layer is i is the data block number represented by one-hot encoding, and the data block numbered k corresponds to x i The parameters of the hidden layer and output layer are W1 and W2, which are N×M matrices and M×N matrices respectively.

[0094] During the pre-training process of a neural network model, the neural network parameters are initialized with random values before training begins. Then, stochastic gradient descent or another optimization algorithm is used to update the neural network parameters. After several iterations, training is terminated when the number of iterations or loss reaches a preset requirement. Finally, the output is W1, and the kth row of W1 is used as the vector representation of the kth data block, also known as the feature vector. In actual training, negative sampling and hierarchical softmax techniques may also be used.

[0095] In some embodiments of the present invention, the preset neural network model may be a skip-gram model.

[0096] Using the above solution, each data block has a number as a unique identifier, but the number itself does not contain any information about the content characteristics of the data block, nor can it reflect the relationship between data blocks. At the same time, in the subsequent machine learning process, if one-hot encoding is used to represent the data blocks, there will be the problem of too many vector dimensions. This problem is more prominent for edge nodes with limited computing resources. Therefore, by pre-setting the neural network model and using the embedding method to represent the data blocks with low-dimensional vectors, it is possible to not only extract valuable features of the data blocks, but also explore the relationships between the data blocks, which is beneficial for subsequent further processing. We use historical access data collected by the central node and then use embedding technology to enable data blocks to be represented using low-dimensional vectors, which facilitates the recording of data access by all terminal devices.

[0097] By adopting the above scheme, this scheme can continuously improve the accuracy of the model by online learning based on the pre-completed training.

[0098] In some embodiments of the present invention, the step of inputting the first state vector into the gate recurrent unit to obtain the first hidden state vector includes:

[0099] In the gate recurrence unit, a reset gate vector and an update gate vector are calculated based on the first hidden state vector, a candidate hidden state vector is calculated based on the reset gate vector, and a first hidden state vector is obtained based on the candidate hidden state vector, the reset gate vector, and the update gate vector.

[0100] In some embodiments of the present invention, the gate recurrent unit includes a fully connected layer and an activation function layer, through the fully connected layer and the sigmoid activation function σ, using the current first hidden state vector and the first hidden state vector h of the previous round -1 , calculate the reset gate vector R; then, through the fully connected layer and tanh activation function, using the first hidden state vector, the first hidden state vector of the previous round h -1 And reset the gate vector R, calculate the candidate hidden state vector Finally, using the update gate z and the first hidden state vector h of the previous round -1 and the candidate hidden state vector Get the current first hidden state vector h and output it to the subsequent steps.

[0101] In some embodiments of the present invention, the reset gate vector and the update gate vector are calculated based on the first hidden state vector in the gate recurrent unit according to the following formula:

[0102]

[0103]

[0104] R represents the reset gate vector, z represents the update gate vector, σ represents the sigmoid activation function, Both represent the weight matrix of the fully connected layer, b r 、b z Both represent bias vectors, h -1 represents the first hidden state vector of the previous round, and s represents the first state vector.

[0105] In some embodiments of the present invention, the candidate hidden state vector is calculated based on the reset gate vector according to the following formula:

[0106]

[0107] represents the candidate hidden state vector, and Both represent the weight matrix of the fully connected layer, b h represents the bias vector, h -1 Represents the first hidden state vector of the previous round, R represents the reset gate vector, * represents the Hadamard product, and s represents the first state vector.

[0108] In some embodiments of the present invention, the first hidden state vector is calculated based on the candidate hidden state vector, the reset gate vector, and the update gate vector according to the following formula:

[0109]

[0110] h represents the first hidden state vector, z represents the update gate vector, h -1 represents the first hidden state vector of the previous round, * represents the Hadamard product, represents the candidate hidden state vector.

[0111] In some embodiments of the present invention, the step of inputting the first hidden state vector into a target network pre-trained by an edge node and outputting a first action includes:

[0112] The target network outputs a value parameter corresponding to each preset action based on the input first hidden state vector, and the preset action corresponding to the maximum value parameter is the first action.

[0113] In some embodiments of the present invention, the preset action may be replacing the first data block of the edge node, replacing the second data block of the edge node, replacing the third data block of the edge node, or not changing the data block of the edge node.

[0114] The action space can be defined as {1, 2, .., C max , C max +1}, for 1≤c≤Cmax ,,c means using the candidate data block to replace the cached data block at position c, C max +1 means not to keep the current candidate data block. In other words, the edge node can make two decisions: one is to keep the candidate data block in a certain location of the cache, and the other is to discard the candidate data block.

[0115] In some embodiments of the present invention, the first action may be output by a classification layer.

[0116] The reward function of this scheme can be set as follows:

[0117] r=-d t ;

[0118] In the specific implementation process, the last data block can be calculated after each data is received at the edge node, and the reward function of the scheme can be set as follows:

[0119]

[0120] Using the above scheme, we hope to minimize the total delay in the area, so we can define the above reward function. At the same time, we define the discount coefficient γ when calculating the cumulative reward. The reward value is the inverse of the total delay in the area within the time slot.

[0121] In some embodiments of the present invention, the second hidden state vector is input into the value network of the central node, and the estimated action is obtained in the same manner as the first action is obtained from the value network. An estimated value corresponding to the estimated action is output, and the estimated value is a value parameter corresponding to the estimated action. The target value is calculated based on the estimated value according to the following formula:

[0122]

[0123] r represents the reward function, represents the target value, represents the estimated value, and γ represents the discount coefficient.

[0124] In some embodiments of the present invention, the loss function value is calculated based on the target value according to the following formula:

[0125]

[0126] E represents the loss function value, Indicates the value parameter corresponding to the first action, Indicates the target value.

[0127] In some embodiments of the present invention, the step of updating the network parameters in the value network based on the loss function value is to update the network parameters in the value network using a gradient descent method based on the loss function value according to the following formula:

[0128]

[0129] w * represents the network parameters in the updated value network, w represents the network parameters in the value network before updating, η represents the learning rate, E represents the loss function value, Indicates the value parameter corresponding to the first action, Indicates the target value.

[0130] Using the above scheme, common cache replacement strategies such as LRU and LFU have a clear quantitative metric that directly determines the cache replacement target. This metric is determined by access patterns over the previous period. In addition to manually constructing quantitative metrics, recurrent neural networks can also be used to extract implicit features from time series. In the edge node's reinforcement learning model, we consider the data blocks requested by the end device and the cached data blocks as states, associate latency or outbound traffic with a reward function, and use the decision of whether to replace as the action space. We use a recurrent neural network to extract features from the time series and then use the D3QN model to learn a cache replacement policy. During this process, the edge node interacts with the end device, making cache replacement decisions using the ∈-greedy policy. The state, action, and reward are collected and uploaded to the central node. The central node uses experience replay techniques to train the D3QN model and periodically distributes the latest model to the edge nodes.

[0131] In this solution, the edge node adopts any caching strategy or does not cache data, and receives requests from terminal devices at the same time. In each time slot, for each accessed data block, the edge node reports the access frequency of the terminal device to the central node. The reporting format is (time, node, data, frequency), which represents the time slot number, edge node number, data block number and access count respectively. After the central node collects enough records, it summarizes the above records into the access record set A = {(time, node, data, frequency)}. i , node i , data i , frequency i )}, sorted by time slot number and edge node number.

[0132] For the records in A, according to the overall distribution of access times, they are divided into several categories according to the number of access times. For records of different categories, different numbers of duplicate records are used to replace them, and the specific access times are no longer retained, forming a new access record set B = {(time i , node i , data i )}, still maintaining the order of time slot numbers and node numbers. For example, if we collect 10,000 access records, where the number of accesses ranges from 1 to 1000, we can divide all records into 10 categories. The first category contains records with accesses from 1 to 100, and we replace the original record with a new record without an access count. The second category contains records with accesses from 101 to 200, and we replace the original record with two new records without an access count. And so on, replacing all records.

[0133] During the specific implementation process, we divide time into time slots of equal length, and each time slot is divided into two phases. In the first phase, the terminal device sends a data access request to the edge node. If the edge node has cached the data, it will be returned directly to the terminal device. Otherwise, the edge node will request the data from the central node. The central node will then query whether other edge nodes in the area have cached the data. If so, it will obtain the data from the corresponding edge node and finally hand it over to the terminal device. If no edge node has cached the data requested by the terminal device, the central node will request the data from the cloud and then return it to the terminal device. In the second phase, the edge node replaces the cached content. The data obtained by the edge node from the central node and other edge nodes in the first phase will be temporarily saved in the second phase for the decision to replace it in the second phase.

[0134] Using edge caching can reduce data access latency and the traffic required to request data from the cloud. This solution uses the interaction between the target network and the value network to find a second-stage caching strategy to minimize the total data access latency of all terminal devices in the entire area, or minimize the external traffic of the entire area.

[0135] During the specific implementation process, the target network and the value network need to be trained in advance. The training steps include:

[0136] (1) Initialize the parameters of the two neural networks with the same random values and set the target network Q(s, a; w t ) is sent to each edge node. At the same time, a fixed-size experience replay array is set at the central node.

[0137] (2) Each edge node uses the ∈-greedy strategy to interact with the environment. The edge node transmits the collected experience quadruple (s, a, r, s′) back to the central node, where s represents the initial state (which can be the first state vector), a is the action, r is the reward function value, and s′ is the second state vector. The central node stores the quadruple in the experience replay array. When saving, the edge quadruple generated by different edge nodes is saved separately, and the array is arranged in the order of generation. If the experience replay array is full, the oldest data is deleted. The ∈-greedy strategy can be expressed by the following formula.

[0138]

[0139] (3)w t Represents the parameters of the target network. If the number of quadruple pairs collected by the central node reaches a pre-set threshold, (4) to (5) are repeated to update the neural network parameters. During this process, the edge nodes continue to execute step (2) to collect more experience.

[0140] (4) Before each iteration, for the central node, we set the hidden state h of the GRU in the two neural networks -1 Set to zero. (Note: Because the central node uses the experience from different edge nodes for training, and the states between different edge nodes are incoherent, the memory needs to be cleared.) Then randomly select an edge node, and randomly select a quadruple from the experience quadruple generated by the edge node as the starting quadruple.

[0141] (5) Starting from the starting quadruple, select the remaining empirical quadruple of the same edge node in the order of generation, ensuring that the states are connected, until the number reaches the preset size or there are no more quadruple. For each selected quadruple (s, a, r, s′), perform the following steps (6) to (13) in order.

[0142] (6) Use the current state s to forward propagate the value network Q(s, a; w), where w represents the value network parameter, to obtain the estimated value of the current state-action pair (s, a)

[0143]

[0144] (7) Use the next state s′ to forward propagate the value network Q(s, a; w) and select the action a with the greatest value. * .

[0145]

[0146] (8) Use the next state s′ to target the network Q(s, a; w t) to do forward propagation and get (s′, a * )Estimated value

[0147]

[0148] (9) Calculate TD target

[0149]

[0150] (10) Define the loss function as

[0151]

[0152] (11) Gradient descent or other optimization methods are then used to update the parameters of the value network Q(s, a; w).

[0153]

[0154] (12) Parameters w of the value network and w of the target network t The parameters of the value network are not updated synchronously. The central node will periodically synchronize the parameters w of the value network to the target network w. t , and the updated target network parameters w t Sent to edge nodes.

[0155] (13) When the set number of iterations is reached or the loss value is less than the set value, the training is stopped and the parameters of the value network are synchronized to the target network and edge nodes for the last time.

[0156] Adopting the above approach, this paper first describes a system model for collaborative edge caching, conducts theoretical analysis based on this model, and then proposes a deep reinforcement learning approach to solve the problem. This paper also introduces embedding technology into the data preprocessing stage of reinforcement learning, improving the training efficiency of subsequent steps and fully utilizing historical information to improve model performance. Furthermore, a reinforcement learning model with a recurrent neural network is used for training, combined with the distributed structure of the collaborative caching system, to obtain the final cache replacement strategy. Compared with conventional cache replacement methods, the method proposed in this paper is more suitable for the actual scenario of collaborative edge caching, with lower latency and data consumption.

[0157] An embodiment of the present invention also provides an edge collaborative caching device based on deep reinforcement learning, which includes a computer device, the computer device including a processor and a memory, the memory storing computer instructions, and the processor being used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps implemented by the method described above.

[0158] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned edge collaborative caching method based on deep reinforcement learning. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0159] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0160] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0161] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0162] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. An edge collaborative caching method based on deep reinforcement learning, characterized in that: The method is applied to a distributed system, the distributed system including a central node and multiple edge nodes, and the steps of the method include: The edge node receives the data request from the terminal device, calls the data for the terminal device based on the data request, divides the data into multiple data blocks based on the data size, and marks each data block; Construct a first state vector based on the data block stored in the current edge node and one of the called data blocks, input the first state vector into the gated recurrent unit to obtain a first hidden state vector, input the first hidden state vector into the target network pre-trained by the edge node, and output a first action, where the gated recurrent unit is a GRU; Based on the first action, a data block stored in the edge node is changed, and a second state vector is constructed based on the edge node of the changed data block and another called data block, and a second hidden state vector is further obtained based on the gate recurrent unit; Inputting the second hidden state vector into the value network of the central node to obtain an estimated action, outputting an estimated value corresponding to the estimated action, and calculating a target value based on the estimated value, wherein both the target network and the value network adopt a duel network structure; The loss function value is calculated based on the target value, the network parameters in the value network are updated based on the loss function value, and the network parameters in the value network are synchronized to the target network of each edge node.

2. The edge collaborative caching method based on deep reinforcement learning according to claim 1 is characterized in that: The step of constructing a first state vector based on the data block stored in the current edge node and one of the called data blocks includes: The number of one of the called data blocks and the number of the data block stored in the current edge node are jointly constructed into an initial vector; The initial vector is input into a preset neural network model to obtain a first state vector.

3. The edge collaborative caching method based on deep reinforcement learning according to claim 1 is characterized in that: The step of inputting the first state vector into the gate recurrent unit to obtain the first hidden state vector includes: In the gate recurrence unit, a reset gate vector and an update gate vector are calculated based on the first hidden state vector, a candidate hidden state vector is calculated based on the reset gate vector, and a first hidden state vector is obtained based on the candidate hidden state vector, the reset gate vector, and the update gate vector.

4. The edge collaborative caching method based on deep reinforcement learning according to claim 3 is characterized in that: The reset gate vector and the update gate vector are calculated in the gate recurrent unit based on the first hidden state vector according to the following formula: R represents the reset gate vector, z represents the update gate vector, σ represents the sigmoid activation function, Both represent the weight matrix of the fully connected layer, b r 、b z Both represent bias vectors, h -1 represents the first hidden state vector of the previous round, and s represents the first state vector.

5. The edge collaborative caching method based on deep reinforcement learning according to claim 4 is characterized in that: The candidate hidden state vector is calculated based on the reset gate vector according to the following formula: represents the candidate hidden state vector, and Both represent the weight matrix of the fully connected layer, b h represents the bias vector, h -1 Represents the first hidden state vector of the previous round, R represents the reset gate vector, * represents the Hadamard product, and s represents the first state vector.

6. The edge collaborative caching method based on deep reinforcement learning according to claim 5, characterized in that: The first hidden state vector is calculated based on the candidate hidden state vector, the reset gate vector, and the update gate vector according to the following formula: h represents the first hidden state vector, z represents the update gate vector, h -1 represents the first hidden state vector of the previous round, * represents the Hadamard product, represents the candidate hidden state vector.

7. The edge collaborative caching method based on deep reinforcement learning according to any one of claims 1 to 6, characterized in that: The step of inputting the first hidden state vector into the target network pre-trained by the edge node and outputting the first action includes: The target network outputs a value parameter corresponding to each preset action based on the input first hidden state vector, and the preset action corresponding to the maximum value parameter is the first action.

8. The edge collaborative caching method based on deep reinforcement learning according to claim 7, characterized in that: The second hidden state vector is input into the value network of the central node. The estimated action is obtained in the same way as the first action is obtained from the value network. The estimated value corresponding to the estimated action is output. The estimated value is the value parameter corresponding to the estimated action. The target value is calculated based on the estimated value according to the following formula: r represents the reward function, represents the target value, represents the estimated value, and γ represents the discount coefficient.

9. The edge collaborative caching method based on deep reinforcement learning according to claim 8, characterized in that: The loss function value is calculated based on the target value according to the following formula: E represents the loss function value, Indicates the value parameter corresponding to the first action, Indicates the target value.

10. An edge collaborative caching device based on deep reinforcement learning, characterized in that: The apparatus includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus implements the steps implemented by the method according to any one of claims 1 to 9.

Images