Cloud edge-end collaborative content caching method

Abstract

The invention provides a cloud side-end collaborative content caching method, which comprises the following steps of: obtaining an initial caching value according to the number of network content objects in a network content list required by terminal equipment and the popularity degree of each network content object, so that the terminal equipment puts forward a content request; the edge server performs online updating on the cache value in each time slot by obtaining and responding to the content request to obtain the current cache value; constructing a single-side caching strategy according to the established constraint conditions of the edge server and the terminal equipment and the current caching value, and obtaining a single-side caching strategy of the edge server and a single-side caching strategy of the terminal equipment to construct a strategy space; combining the constructed state space, the reward function and the strategy space to obtain an action space, inputting the action space into the trained deep reinforcement learning network for action selection, and obtaining an optimal action as a multilateral cooperative caching strategy; the long-term delay is optimized to achieve efficient collaborative content caching without generating a dimensionality curse.

Description

Technical Field

[0001] The present invention relates to the field of edge caching technology in mobile communications, and in particular to a cloud-edge collaborative content caching method. Background Art

[0002] In recent years, the rapid development of mobile smart devices has brought great convenience to human life, but also caused a great burden on network infrastructure. During peak traffic hours or crowded areas, a large number of smart networked devices are concentrated in small hot spots, resulting in a sharp increase in regional Internet traffic, which puts a serious burden on servers, causes serious network congestion, and brings unacceptable delays to users. If this situation cannot be alleviated for a long time, a large number of users will not be able to obtain content within an acceptable time, and repeated requests for acquisition will increase further, thereby aggravating network congestion and causing more serious network problems. Therefore, in order to relieve the pressure, it is necessary to offload the huge traffic of traditional Internet to other networks. Mobile Edge Computing Network (MEC) has certain computing and communication capabilities and can share huge amounts of traffic. MEC network supports caching of commonly used content. Through cloud-edge-end collaborative optimization on MEC network, it can effectively alleviate the pressure of traditional Internet, reduce latency, and improve user experience.

[0003] MEC network is a common network in cloud-edge optimization, usually composed of three parts: cloud server, edge server and terminal device. Content caching in MEC network refers to the storage of frequently used or requested content for high-speed access by users. By caching commonly used content to edge servers closer to users, the data transmission path can be shortened, allowing users to access data more quickly. The limited storage capacity of edge servers is still inadequate in the face of massive amounts of popular content, and collaboration between terminal devices and edge servers is required to further improve caching capabilities.

[0004] In order to achieve efficient collaborative content caching, it is necessary to coordinate the cache content of edge servers and terminal devices. Some researchers predict the data that users may need based on the user's historical behavior, location and other characteristics, and cache the expected data in advance to edge devices close to the user. However, predictions are bound to be imperfect, and some mistakes often lead to serious consequences. Many researchers focus on balancing the distribution of cache content between edge servers and terminal devices. Both terminal devices and edge servers are selfish and tend to cache popular content, even if it has been cached by other devices or edge servers around them, wasting storage space. Although collaborative content caching can enhance the caching capabilities of edge servers and terminal devices, it is still a challenging problem.

[0005] At present, there are two main methods for collaborative content caching in cloud-edge optimization: traditional heuristic algorithms and neural network AI algorithms. Traditional heuristic optimization algorithms can solve the problem of cache content allocation in static scenarios. However, considering that terminal devices are mobile and have multiple dynamic constraints (such as energy) and collaborative content caching is a long-term process, the environment formed by edge servers and terminal devices is highly dynamic. Traditional heuristic optimization methods decompose dynamic environments into independent static environments for optimization. However, problems in dynamic environments are contextual over time, and making decisions based solely on static environments is one-sided. Thanks to the development of artificial intelligence technology and hardware, many researchers have adopted deep reinforcement learning (DRL) to make decisions on complex problems. However, the demand for diverse content on mobile devices has generated a huge cache combination, which has led to the curse of dimensionality in the DRL algorithm. In addition, the DRL algorithm tends to explore greedily, which may lead to different solutions in the same state, which is dangerous for the device. Summary of the invention

[0006] The present invention provides a cloud-edge collaborative content caching method, the purpose of which is to optimize long-term delay without generating the curse of dimensionality to achieve efficient collaborative content caching.

[0007] In order to achieve the above object, the present invention provides a cloud-edge-device collaborative content caching method, which is applied to a cloud-edge-device collaborative system including a cloud server, an edge server and a terminal device, and the method includes:

[0008] Step 1, obtaining a network content list required by the terminal device, the network content list including multiple network content objects and the popularity of each network content object, and obtaining an initial cache value by using the number of network content objects in the network content list and the popularity of each network content object, wherein the popularity refers to the possibility of the network content object being selected first;

[0009] Step 2: The terminal device within the communication range of the edge server makes a content request based on the initial cache value, and the edge server obtains and responds to the content request by searching its own cache space, the cache space of the terminal device, and the storage space of the cloud server;

[0010] Step 3, updating the cache value online in each time slot according to the content request responded by the edge server to obtain the current cache value;

[0011] Step 4: Establish constraints for the edge server and the terminal device, and construct a unilateral cache strategy based on the constraints and the current cache value to obtain the unilateral cache strategy of the edge server and the unilateral cache strategy of the terminal device;

[0012] Step 5: construct a policy space based on the unilateral cache policy of the edge server and the unilateral cache policy of the terminal device;

[0013] Step 6: construct a state space and reward function based on the current cache status of the edge server and the terminal device, combine the state space, reward function and strategy space to obtain the action space, and input the action space into the trained deep reinforcement learning network for action selection to obtain the optimal action as the multilateral collaborative caching strategy. The multilateral collaborative caching strategy is:

[0014] The terminal device and / or the edge server executes a unilateral cache update strategy to update the content cache;

[0015] or,

[0016] The end device and / or edge server maintains the current cache content.

[0017] Specifically, the initial cache value is calculated as:

[0018]

[0019] in, Indicates the initial cache value, p s represents the popularity of the network content object, s represents the sth network content object directory, S represents the number of network content object directories, Represents the set of network content object directories, ψ represents a fixed parameter with no physical meaning.

[0020] More specifically, step 2 includes:

[0021] In each time slot, the terminal devices within the communication range of the edge server make content requests based on the initial cache value;

[0022] The edge server determines whether there is a network content object related to the content request by searching its own cache space;

[0023] If the network content object related to the content request exists in its own cache space, the edge server directly responds to the content request;

[0024] If the network content object related to the content request does not exist in its own cache space, the edge server searches the cache space of the terminal device to determine whether the network content object related to the content request exists in the cache space of the terminal device;

[0025] If the network content object related to the content request exists in the cache space of the terminal device, the edge server obtains and responds to the content request from the cache space of the terminal device;

[0026] If the network content object related to the content request does not exist in the cache space of the terminal device, the edge server obtains and responds to the content request from the storage space of the cloud server.

[0027] Specifically, the update formula for the current cache value is:

[0028]

[0029] in, Indicates the current cache value, represents the cache value in the t-1 time slot, Indicates the number of times a network content object is requested in the current time slot.

[0030] Furthermore, the constraints for edge servers and terminal devices are established, including:

[0031] The storage space constraints of the edge server are:

[0032]

[0033] in, represents the caching decision of the edge server at time slot t, l s Indicates the size of the network content object, ts indicates the total size of the edge server's own cache space;

[0034] The storage space constraints of the terminal device are:

[0035]

[0036] in, represents the cache decision of terminal device i at time slot t, ts i Indicates the total size of the cache space of the terminal device. Indicates the total number of terminal devices within the communication range of the edge server;

[0037] The energy constraint when the terminal device sends a network content object to the edge server is:

[0038]

[0039] in, represents the communication energy consumption of terminal device i, e i represents the communication energy budget of terminal device i;

[0040] The energy cost of the terminal equipment is:

[0041]

[0042] in, Represents the potential communication energy consumption of network content objects.

[0043] Furthermore, a unilateral caching strategy is constructed based on the constraints and the current cache value to obtain the unilateral caching strategy of the edge server and the unilateral caching strategy of the terminal device, including:

[0044] The storage space constraints of the edge server, the storage space constraints of the terminal device, the energy constraints, the energy cost, and the current cache value are input into the improved ACS algorithm to update, generate a new unilateral cache strategy, and retain the original unilateral cache strategy;

[0045] New unilateral cache strategy A new (t) is:

[0046]

[0047] Original unilateral cache strategy A old (t) is:

[0048]

[0049] in, represents the cache decision of terminal device i at time slot t-1, represents the cache decision set of terminal device i, Represents the cache decision set of the edge server.

[0050] More specifically, the state space expression is:

[0051]

[0052] in, represents the state space, t represents the time slot t, represents the caching decision of the edge server, Indicates The cache decision of each terminal device, Indicates the communication relationship between the edge server and the terminal device in the current time slot. Indicates the online value of the content in the current time slot.

[0053] Specifically, the reward function is expressed as:

[0054]

[0055] in, represents the reward function, Indicates that in time slot t The total delay for each terminal device to obtain a specific network content object.

[0056] Further, the deep reinforcement learning network consists of a dual deep Q network including a training Q network and a target Q network, and a noisy network;

[0057] Train the Q network for action selection;

[0058] The target Q network is used to calculate the value of the selected action;

[0059] Noisy Network is used to add Gaussian noise to the training Q-Network and the target Q-Network.

[0060] Specifically, the loss function of the deep reinforcement learning network is:

[0061]

[0062] Among them, ω and ω - They represent the parameter sets of the training Q network and the target Q network, which are updated by back propagation. K represents the amount of data. Indicates in status Take action reward, γ refers to the discount factor, and its value range is [0,1), and They represent the value of taking a specified action under the input state calculated by the training Q network with additional noise and the target Q network, respectively.

[0063] The above scheme of the present invention has the following beneficial effects:

[0064] The present invention is applied to a cloud-edge-end collaboration system including a cloud server, an edge server and a terminal device. The system obtains a network content list required by the terminal device, the network content list includes multiple network content objects and the popularity corresponding to each network content object, and uses the number of network content objects in the network content list and the popularity of each network content object to obtain an initial cache value, wherein the popularity refers to the possibility of the network content object being preferentially selected; the terminal device within the communication range of the edge server makes a content request based on the initial cache value, and the edge server obtains and responds to the content request by retrieving its own cache space, the cache space of the terminal device and the storage space of the cloud server; the cache value is updated online in each time slot according to the content request responded by the edge server to obtain the current cache value; establishes a communication channel between the edge server and the terminal device. The constraints of the equipment are taken into consideration, and a unilateral caching strategy is constructed according to the constraints and the current cache value to obtain the unilateral caching strategy of the edge server and the unilateral caching strategy of the terminal device; a strategy space is constructed based on the unilateral caching strategy of the edge server and the unilateral caching strategy of the terminal device; a state space and a reward function are constructed based on the current caching status of the edge server and the terminal device, the state space, the reward function and the strategy space are combined to obtain an action space, and the action space is input into a trained deep reinforcement learning network for action selection to obtain the optimal action as the multilateral collaborative caching strategy; compared with the prior art, the present invention takes into account the long-term content caching benefits and the unstable topological connection and complex strategy space caused by the mobility of the terminal device, and optimizes the long-term delay without generating the curse of dimensionality to achieve efficient collaborative content caching.

[0065] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of an application scenario of an embodiment of the present invention;

[0068] Figure 3 Schematic diagram of the process of the improved ACS algorithm in the embodiment of the present invention;

[0069] Figure 4 is a model structure diagram of a deep reinforcement learning network in an embodiment of the present invention;

[0070] Figure 5 Schematic diagram of the process of deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0073] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0074] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0075] In view of the existing problems, the present invention provides a cloud-edge collaborative content caching method.

[0076] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a cloud-edge-device collaborative content caching method, which is applied to a cloud-edge-device collaborative system including a cloud server, an edge server and a terminal device, and the method includes:

[0077] Step 1, obtaining a network content list required by the terminal device, the network content list including multiple network content objects and the popularity of each network content object, and obtaining an initial cache value by using the number of network content objects in the network content list and the popularity of each network content object, wherein the popularity refers to the possibility of the network content object being selected first, and the greater the popularity, the greater the possibility of the network content object being selected first;

[0078] Step 2: The terminal device within the communication range of the edge server makes a content request based on the initial cache value. The edge server obtains and responds to the content request by searching its own cache space, the cache space of the terminal device, and the storage space of the cloud server.

[0079] Step 3, updating the cache value online in each time slot according to the content request responded by the edge server to obtain the current cache value;

[0080] Step 4: Establish constraints for the edge server and the terminal device, and construct a unilateral cache strategy based on the constraints and the current cache value to obtain the unilateral cache strategy of the edge server and the unilateral cache strategy of the terminal device;

[0081] Step 5: construct a policy space based on the unilateral cache policy of the edge server and the unilateral cache policy of the terminal device;

[0082] Step 6: construct a state space and reward function based on the current cache status of the edge server and the terminal device, combine the state space, reward function and strategy space to obtain the action space, and input the action space into the trained deep reinforcement learning network for action selection to obtain the optimal action as the multilateral collaborative caching strategy. The multilateral collaborative caching strategy is:

[0083] The terminal device and / or the edge server executes a unilateral cache update strategy to update the content cache;

[0084] or,

[0085] The end device and / or edge server maintains the current cache content.

[0086] In the embodiment of the present invention, it should be noted that the cloud server in the cloud-edge-device collaboration system is located in a remote data center and has powerful computing and storage capabilities for processing complex applications and algorithms;

[0087] The edge server is used as a bridge for communication between users and cloud servers;

[0088] A terminal device generally refers to a user's personal device, such as a smart computer, a smart car, a tablet computer, etc., which has limited storage capacity, computing resources and energy and can only undertake some tasks. In an embodiment of the present invention, the number of terminal devices is multiple.

[0089] Step 1: Obtain a network content list required by a terminal device, the network content list including multiple network content objects and the popularity of each network content object, and obtain an initial cache value using the number of network content objects in the network content list and the popularity of each network content object.

[0090] In order to finely process the collaborative caching of edge servers and terminal devices, considering that the number of network content objects in the network is variable, the embodiment of the present invention regards the collaborative caching process as multiple rounds of services. The network content list of one round of services is the same, and the difference between different rounds of services is that the network content list is different. The service process of one round is divided into several time slots. In the first time slot of one round of services, the network content list must first be obtained;

[0091] The network content object is the network content object required by the terminal device, and the collection of network content objects is represented as The network content object directory is represented by Web Content Objects The characteristic has size l s and the value of the network content object in the time slot The value set of time slots is expressed as

[0092] Content value is a key factor in caching. Value can be expressed as popularity, which reflects the probability that a certain network content object will be preferred over other objects, denoted as p s ,in

[0093]

[0094] In the embodiment of the present invention, the product of the number of network content objects and the popularity is used as the initial cache value, and the calculation expression is:

[0095]

[0096] in, Indicates the initial cache value, p s represents the popularity of the network content object, s represents the sth network content object directory, S represents the number of network content object directories, Represents the set of network content object directories, ψ represents a fixed parameter with no physical meaning.

[0097] Step 2: The terminal device within the communication range of the edge server makes a content request based on the initial cache value. The edge server obtains and responds to the content request by retrieving its own cache space, the cache space of the terminal device, and the storage space of the cloud server.

[0098] Specifically, step 2 includes:

[0099] In each time slot, the terminal devices within the communication range of the edge server make content requests based on the initial cache value;

[0100] The edge server determines whether there is a network content object related to the content request by searching its own cache space;

[0101] If the network content object related to the content request exists in its own cache space, the edge server directly responds to the content request;

[0102] If the network content object related to the content request does not exist in its own cache space, the edge server searches the cache space of the terminal device to determine whether the network content object related to the content request exists in the cache space of the terminal device;

[0103] If the retrieval is successful, it means that the network content object related to the content request exists in the cache space of the terminal device, and the edge server obtains and responds to the content request from the cache space of the terminal device;

[0104] If the retrieval fails, it means that there is no network content object related to the content request in the cache space of the terminal device. The edge server obtains and responds to the content request from the storage space of the cloud server.

[0105] Since the cache value gradually deviates from the content request over time, in order to better reflect the demand for network content objects, the embodiment of the present invention uses the Least Frequently Used (LFU) cache update algorithm to adjust the cache value online, and sets a counter for each network content object to record the number of requests, which is recorded as Therefore, the update formula that defines the current cache value is:

[0106]

[0107] in, Indicates the current cache value, represents the cache value in the t-1 time slot, Indicates the number of times a network content object is requested in the current time slot.

[0108] Before edge servers and terminal devices collaborate, they should have their own caching strategies. This caching strategy is made without knowing the cache information of other devices or edge servers and only considers their own situation. It is called a unilateral caching strategy. The existence of a unilateral caching strategy ensures that edge servers or terminal devices can still use the content caching function normally even without collaboration.

[0109] The embodiment of the present invention defines the attributes of the edge server as follows: Among them, ts represents the total size of the cache space, represents the device communicating with the edge server at time slot t, t∈T={1,…,T}, A(t) represents the cache decision of the edge server at time slot t, Each is defined as a 0-1 binary variable.

[0110] Specifically, if It means the network content object c s is not cached, while a s =1 indicates network content object c s The cached network content objects are limited by the edge server storage space capacity. Therefore, the edge server storage space constraints are:

[0111]

[0112] in, represents the caching decision of the edge server at time slot t, l s Indicates the size of the network content object, ts indicates the total size of the edge server's own cache space;

[0113] At time slot t, the devices within the communication range of the edge server are identified as Each terminal device Depend on Represents its characteristics, where e i represents the communication energy budget, ts i Indicates the size of the cache space. express Distance from the edge server, Indicates terminal equipment The cache decision, Similar to edge servers, It is a 0-1 binary variable. 0 indicates that the network content object is not cached by this device. On the contrary, 1 indicates that it is cached. The cached network content object is limited by the storage space capacity of the terminal device. Therefore, the storage space constraint of the terminal device is:

[0114]

[0115] in, represents the cache decision of terminal device i at time slot t, ts i Indicates the total size of the cache space of the terminal device. Indicates the total number of terminal devices within the communication range of the edge server;

[0116] When targeting When a request is submitted, the edge server searches its cache space. Edge servers can store network content objects Send to if However, vehicles within the communication range have The edge server can communicate with the jth terminal device Communicate to obtain network content objects

[0117] To simplify the calculation, the delay of obtaining network content objects from the cloud server is set to a fixed value, denoted as D C , this value is larger than other methods; in the communication between the edge server and the terminal device, the orthogonal frequency division multiple access technology is used, which enables multiple users to share channel resources, specifically:

[0118] set up The allocated bandwidth is B, and its transmission power is p i , the channel gain is h i , σ 2 represents the background noise, the communication rate can be expressed as:

[0119]

[0120] Similar to the terminal device, the communication power of the edge server is p u , the channel gain is h u , then the communication rate is expressed as:

[0121]

[0122] Therefore, c s The transmission delay calculation formula is:

[0123]

[0124] The calculation of communication energy consumption is expressed as:

[0125]

[0126] The energy of the terminal device is limited. When the edge server needs When the network content object It needs to communicate with it, and the communication needs to meet the energy constraint. Therefore, the energy constraint condition when the terminal device sends the network content object to the edge server is:

[0127]

[0128] in, represents the communication energy consumption of terminal device i, e i represents the communication energy budget of terminal device i;

[0129] In addition, in order to ensure that the collaboration can proceed normally and avoid the situation where the device cannot provide the network content object assistance that meets the energy constraint due to insufficient energy, the terminal device needs to consider the potential energy consumption of the network content object. The potential communication energy consumption of the network content object is expressed as The calculation method is:

[0130]

[0131] in, Represents a network content object c s The cache hit ratio, calculated as the number of hits divided by the total number of requests for the web content object.

[0132] Since the terminal device needs to make an estimate to avoid the potential communication cost exceeding the allocated communication energy consumption budget, the energy cost of the terminal device is expressed as:

[0133]

[0134] in, Represents the potential communication energy consumption of network content objects.

[0135] Specifically, a unilateral caching strategy is constructed based on the constraints and the current cache value to obtain the unilateral caching strategy of the edge server and the unilateral caching strategy of the terminal device, including:

[0136] The storage space constraints of the edge server, the storage space constraints of the terminal device, the energy constraints, the energy cost, and the current cache value are input into the improved ACS algorithm to update, generate a new unilateral cache strategy, and retain the original unilateral cache strategy;

[0137] New unilateral cache strategy A new (t) is:

[0138]

[0139] Original unilateral cache strategy A old (t) is:

[0140]

[0141] in, represents the cache decision of terminal device i at time slot t-1, represents the cache decision set of terminal device i, Represents the cache decision set of the edge server.

[0142] In the embodiment of the present invention, when the edge server or terminal device receives a new network content object, the improved ACS algorithm is used to perform a local cache value update to find the best network content object combination. The algorithm steps are as follows:

[0143] First, a directed graph is constructed based on the edge server or terminal device according to the original cache strategy. in and Represents a set of vertices and edges respectively. Each cached web content object is represented as a vertex, and an edge connects every two vertices. 0 is the initial vertex. The old optimal solution A old The total cache value of (t) is W old (t), calculated as follows:

[0144]

[0145] Initialize new solution A new (t) and the total value W new = 0. The edge record visited is CE new The embodiment of the present invention initializes the path pheromone based on the old optimal solution and the current cache value.

[0146] Specifically, the edge (s v ,s w ) is expressed as:

[0147]

[0148] The goal is to find a unilateral caching strategy A that can maximize the sum of content value under constraints at time slot t. new (t), expressed as follows:

[0149]

[0150] Denote the vertices and edges visited by ant j as cv j and j , j selects the next node as follows:

[0151]

[0152] Among them, θ represents a random number uniformly distributed in the interval [0,1], and θ 0 is a preset parameter used to adjust the trade-off between utilizing existing experience and exploring new paths, and α and β are heuristic factors.

[0153] Every time ant j finishes searching, it updates the pheromones of the edges it passed by:

[0154]

[0155] Among them, 0<ξ 1 <1 indicates the rate of pheromone volatilization. This local update method enhances the exploration ability and guides ants to explore unexplored edges;

[0156] When ant j is ready to move to the new vertex s wWhen it calculates the size and value of the visited vertices, according to the communication energy consumption expression, if the size of the network content object meets the storage space constraint and the energy meets the communication energy consumption and Then the vertex s w and edge(s v ,s w ) will be marked as visited, otherwise, ant j will end its search.

[0157] After all ants have completed their search, a global update of pheromones will be performed. Only the edge of the optimal solution will be updated with pheromones. The update formula is:

[0158]

[0159] Among them, ξ 2 Similar to ξ 1 , and satisfy 0<ξ 2 <1.

[0160] When the maximum number of iterations is reached or the solution is no longer updated, the optimal solution becomes A new (t).

[0161] It is worth noting that A new (t) do not necessarily contain new web content objects; they represent a unilateral caching strategy when new web content objects are received.

[0162] Relatively speaking, A old (t) is the cache strategy before receiving a new web content object, which means that if the cost value of the new web content object is low, A new (t) and A old (t) may be the same.

[0163] Step 5: construct a policy space based on the unilateral cache policy of the edge server and the unilateral cache policy of the terminal device.

[0164] Before building a collaborative strategy, you need to obtain a unilateral cache strategy A old (t) is sorted and simplified. The reason for this is that in cloud-edge optimization, due to the limited communication range, the number of terminal devices is capped, while the number of network content objects is uncertain and a large number. Even if the number of terminal devices is 1, the number of network content objects can easily exceed 100, so there will be 2 100 If the number of terminal devices increases further, there will be a larger strategy space, which greatly affects the efficiency of the algorithm, so a streamlined strategy space must be constructed.

[0165] The embodiment of the present invention uses a size of space to represent the collaborative cache update strategy.

[0166] Specifically, the policy space of the streamlined collaborative content cache can be expressed as in When m=0, it indicates the cooperation strategy of the edge server, and m≠0 indicates the cooperation strategy of the terminal device; if Then it means that the corresponding edge server and / or terminal device updates its content cache, that is, adopts A new (t), otherwise use A old (t), which is specifically expressed as follows:

[0167]

[0168] If we do not construct a streamlined strategy space, we must face a size of Thanks to the above processing, we now have an action space of size The action space of this number depends on the number of vehicles communicating with the edge server in the current time slot, which is not very large. Therefore, the strategy space is greatly simplified and provides an action space for the construction of collaborative caching strategy.

[0169] Step 6: construct a state space and reward function based on the current cache status of the edge server and the terminal device, combine the state space, reward function and strategy space to obtain the action space, and input the action space into the trained deep reinforcement learning network for action selection to obtain the optimal action as the multilateral collaborative caching strategy. The multilateral collaborative caching strategy is:

[0170] The terminal device and / or the edge server executes a unilateral cache update strategy to update the content cache;

[0171] or,

[0172] The end device and / or edge server maintains the current cache content.

[0173] In the embodiment of the present invention, the purpose of constructing a multi-sided caching strategy is to consider the global impact from a long-term perspective, coordinate the unilateral cache update strategy of the edge server and the terminal device, and avoid short-sighted updates. For example, the terminal device will want to cache a popular network content object, even if it has been cached by other devices around it, which will cause a waste of space. It is unreasonable to simply set that the network content object that has been cached by other devices cannot be cached repeatedly, because it will cause the network content object to be lost due to the departure of the device, which is also a trap that needs to be avoided; therefore, the construction of a multi-sided caching strategy must consider both the long-term impact and the global impact.

[0174] In order to construct a multilateral caching strategy, based on the above action space, a deep reinforcement learning network is used to implement it. The specific process is as follows:

[0175] First, the time required for an edge server to retrieve a web content object is expressed as:

[0176]

[0177] Then, the terminal device The total delay for obtaining network content objects is D i,s (t) is expressed as:

[0178]

[0179] At each time slot t, the terminal device generates a request, and the edge server searches for network content objects to respond to these requests. In the embodiment of the present invention, the process of the edge server and the terminal device updating the cache after receiving a new network content object is regarded as a Markov decision process (MDP), and the goal is to minimize the long-term delay, which can be expressed as:

[0180]

[0181] We represent the MDP as in represents the state space, represents the action space, represents the state transition probability, represents the reward function. The edge server is based on the state space From the action space Select actions according to the state transition probability Transfer to the next state and according to the reward function Receive your reward.

[0182] Specifically, the state space is the environment, which is the basis for making decisions and the object of influence. In the embodiment of the present invention, the cache space, connection status and content value of the edge server and the terminal device are used as the state space, and its expression is:

[0183]

[0184] in, represents the state space, t represents the time slot t, represents the caching decision of the edge server, Indicates The cache decision of each terminal device, Indicates the communication relationship between the edge server and the terminal device in the current time slot. Indicates the online value of the content in the current time slot.

[0185] Specifically, the action space includes all possible decision actions. In the embodiment of the present invention, the action space is the simplified strategy space, which is defined as:

[0186]

[0187] Specifically, the state transition probability represents the probability of taking an action From the status To status The probability of is defined as:

[0188]

[0189] Specifically, the edge server will receive different rewards after making different action decisions, and the neural network is optimized based on the rewards. Since the goal of the Markov decision process is to minimize long-term delay, in reinforcement learning, the edge server optimizes its strategy according to the reward function to maximize the reward. The expression of the reward function is defined as:

[0190]

[0191] in, represents the reward function, Indicates that in time slot t The total delay for each terminal device to obtain a specific network content object.

[0192] Since in MDP, the return It is a key factor affecting decision making. It is defined as the sum of all benefits from the start of the task until the end of the MDP, and the calculation formula is:

[0193]

[0194] Here, γ represents the attenuation factor.

[0195] The embodiment of the present invention defines the behavior strategy of the agent in reinforcement learning as π, which represents the tendency of selecting actions in reinforcement learning and is only related to the current state. It is worth noting that strategy π is a concept in reinforcement learning that guides the selection of actions. Indicates in status The probability of taking action α under the policy π is recorded as The calculation formula is:

[0196]

[0197] The update formula is expressed as:

[0198]

[0199] Among them, λ represents the update step size.

[0200] The purpose of reinforcement learning is to find the solution that can achieve the optimal In cloud-edge collaborative caching, the optimization of strategy π is to solve the corresponding relationship between cache state, cache strategy and cache value of edge servers and terminal devices to build an efficient cache strategy, thereby optimizing cloud-edge collaborative caching. However, considering that the state space and action space based on cloud-edge collaborative caching are relatively complex, it is difficult for traditional methods to record state-action-value. Therefore, it is necessary to combine deep learning and reinforcement learning and use neural networks to solve it.

[0201] Most preferably, the deep reinforcement learning network consists of a double deep Q-learning network (DDQN) including a training Q network and a target Q network, and a noisy network NoisyNet, wherein the training Q network is used to select actions, the target Q network is used to calculate the value of the selected action, and the noisy network is used to add Gaussian noise to the training Q network and the target Q network. Such a combination can effectively avoid the Q network from overestimating the value of the action.

[0202] Specifically, the embodiment of the present invention uses ω and ω - represents the parameter set of the dual deep Q network DDQN, where ω represents the parameters of the training network, ω - Representing the parameters of the target network, the goal of DDQN can be written as:

[0203]

[0204] In addition, greedy action selection is unstable. For example, in the greedy method, even based on the same state, the actions taken may be different. To avoid this problem, the embodiment of the present invention adds Gaussian noise to the network through Noisy Net, changing Q to By combining DDQN and NoisyNet, the optimization objective is reformulated as:

[0205]

[0206] Finally, the loss function of the deep reinforcement learning network is defined as:

[0207]

[0208] Among them, ω and ω -They represent the parameter sets of the training Q network and the target Q network, which are updated by back propagation. K represents the amount of data. Indicates in status Take action The reward is , γ is the discount factor, and its value range is [0,1). and They represent the value of taking a specified action under the input state calculated by the training Q network with additional noise and the target Q network, respectively.

[0209] In addition, since the parameters ω and ω of the training Q network and the target Q network - The update is not synchronous. Each training will update the parameters ω of the training Q network, and ω will be used to update ω only after a certain step size is met. - , this is the delayed update strategy.

[0210] According to the above conditions, the deep reinforcement learning network can be trained based on the acquired data.

[0211] Using a trained deep reinforcement learning network, you can select the optimal action based on the current environment, namely the multilateral caching strategy, and make cache updates based on this strategy, thereby optimizing long-term delays.

[0212] The unilateral caching strategy ensures that edge servers and terminal devices have available caching strategies under any collaborative relationship. However, the unilateral caching strategy only considers its own current interests. Although it improves the interests of some terminal devices or edge servers, it ignores the overall long-term connection and is a local optimal solution. Implementing only a unilateral caching strategy will limit the cloud-edge collaborative caching. For example, selfish terminal devices will cache high-value content, even if the content has been cached by multiple neighboring devices, resulting in redundancy of content objects in the same edge server coverage area. In addition, if there is only one copy of a specific network content object in a region, then when its owner leaves, the region will lose the cache of the corresponding content object, which is also a trap that needs to be avoided.

[0213] The multi-sided caching strategy is different from the unilateral caching strategy. It is a caching strategy that comprehensively considers the long-term interests of multiple parties. It is a comprehensive adjustment of the unilateral caching strategy of the edge server and the terminal device. Specifically, when new content is obtained, the terminal device and the edge server will have two caching strategies, the new and the old, through the method described above, that is, A new (t) and A old (t), if all new unilateral caching strategies are adopted, it will fall into the trap mentioned above, while the multi-sided caching strategy is to determine the combination of terminal devices and edge servers that need to be updated according to the new strategy and the combination that implements the old strategy while considering the overall interests. Figure 2As shown in the figure, the edge server caches content A, C, and D, device 1 caches content A and C, and device 2 caches content E. Assuming that both device 1 and device 2 request content B, if only a unilateral strategy is implemented, the edge server will use B to replace D, device 1 will use B to replace C, and device 2 will use B to replace E, causing the system to lose the cache of content D and E, and the benefits obtained are limited. Through the multilateral caching strategy, it is determined that the edge server adopts the old strategy, device 1 adopts the new strategy, and device 2 adopts the old strategy, while not losing D and E, ensuring low-latency acquisition of popular content B. Therefore, the multilateral caching strategy determines whether the edge server or terminal device updates the cache. According to the previous article, the optimal action output by the deep reinforcement learning network corresponds to the cache update decision of the terminal device and the edge server, so it can be regarded as a multilateral caching strategy. After obtaining the multilateral caching strategy, the action number can be parsed into the above-mentioned one through binary conversion. Set, that is, each bit of the binary number corresponds to one bit according to The value of sets whether the corresponding edge server or terminal device adopts the new cache strategy and updates the cache. When , the terminal device and / or edge server represented by m is considered to adopt a conservative strategy, that is, avoid updating the cache content and keep the current cache content. When , the terminal device and / or edge server represented by m should adopt a new cache strategy to update. The edge server and terminal device are updated according to the multi-edge cache strategy, thereby realizing cloud-edge-end collaborative content cache optimization.

[0214] The embodiment of the present invention obtains a list of network content required by a terminal device, and obtains an initial cache value according to the number of network content objects in the obtained network content list and the popularity of each network content object; the terminal device within the communication range of the edge server makes a content request based on the initial cache value, and the edge server obtains and responds to the content request by retrieving its own cache space, the cache space of the terminal device, and the cloud server; the cache value is updated online in each time slot according to the content request responded by the edge server to obtain the current cache value; the constraint conditions of the edge server and the terminal device are established, and a unilateral cache strategy is constructed according to the constraint conditions and the current cache value to obtain the unilateral cache of the edge server. The invention discloses a method for implementing a multi-layer collaborative caching strategy based on a single-sided caching strategy of an edge server and a terminal device; constructing a strategy space based on the single-sided caching strategy of an edge server and a terminal device; constructing a state space and a reward function based on the current caching status of the edge server and the terminal device, combining the state space, the reward function and the strategy space to obtain an action space, and inputting the action space into a trained deep reinforcement learning network for action selection to obtain the optimal action as a multi-layer collaborative caching strategy; compared with the prior art, the embodiment of the invention takes into account the long-term content caching benefits and the unstable topological connection and complex strategy space caused by the mobility of the terminal device, and optimizes the long-term delay to achieve efficient collaborative content caching without generating the curse of dimensionality.

[0215] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A cloud-edge-device collaborative content caching method, applied to a cloud-edge-device collaborative system including a cloud server, an edge server and a terminal device, characterized in that: Methods include: Step 1, obtaining a network content list required by the terminal device, the network content list including a plurality of network content objects and a popularity corresponding to each network content object, and obtaining an initial cache value by using the number of network content objects in the network content list and the popularity of each network content object, the popularity referring to the possibility of the network content object being preferentially selected; Step 2, a terminal device within the communication range of the edge server makes a content request based on the initial cache value, and the edge server obtains and responds to the content request by retrieving its own cache space, the cache space of the terminal device, and the storage space of the cloud server; Step 3, updating the cache value online in each time slot according to the content request responded by the edge server to obtain the current cache value; Step 4, establishing constraints of the edge server and the terminal device, and constructing a unilateral cache strategy according to the constraints and the current cache value, to obtain the unilateral cache strategy of the edge server and the unilateral cache strategy of the terminal device; Step 5: constructing a policy space based on the unilateral cache policy of the edge server and the unilateral cache policy of the terminal device; Step 6: construct a state space and a reward function based on the current cache status of the edge server and the terminal device, combine the state space, the reward function and the strategy space to obtain an action space, and input the action space into the trained deep reinforcement learning network for action selection to obtain the optimal action as the multilateral collaborative caching strategy, and the multilateral collaborative caching strategy is; The terminal device and / or the edge server executes the unilateral cache update strategy to update the content cache; or, The terminal device and / or the edge server maintains current cache content.

2. The cloud-edge-device collaborative content caching method according to claim 1, characterized in that: The calculation expression of the initial cache value is: in, Indicates the initial cache value, p s represents the popularity of the network content object, s represents the sth network content object directory, S represents the number of network content object directories, Represents the set of network content object directories, ψ represents a fixed parameter with no physical meaning.

3. The cloud-edge-device collaborative content caching method according to claim 2, characterized in that: The step 2 comprises: In each time slot, a terminal device within the communication range of the edge server makes a content request based on the initial cache value; The edge server determines whether there is a network content object related to the content request by searching its own cache space; If the network content object related to the content request exists in its own cache space, the edge server directly responds to the content request; If the network content object related to the content request does not exist in its own cache space, the edge server searches the cache space of the terminal device to determine whether the network content object related to the content request exists in the cache space of the terminal device; If the network content object related to the content request exists in the cache space of the terminal device, the edge server obtains and responds to the content request from the cache space of the terminal device; If the network content object related to the content request does not exist in the cache space of the terminal device, the edge server obtains and responds to the content request from the storage space of the cloud server.

4. The cloud-edge-device collaborative content caching method according to claim 3, characterized in that: The update formula of the current cache value is: in, Indicates the current cache value, represents the cache value in the t-1 time slot, Indicates the number of times a network content object is requested in the current time slot.

5. The cloud-edge-device collaborative content caching method according to claim 4, characterized in that: Establishing constraints between the edge server and the terminal device includes: The storage space constraints of the edge server are: in, represents the caching decision of the edge server at time slot t, l s Indicates the size of the network content object, ts indicates the total size of the edge server's own cache space; The storage space constraint condition of the terminal device is: in, represents the cache decision of terminal device i at time slot t, ts i Indicates the total size of the cache space of the terminal device. Indicates the total number of terminal devices within the communication range of the edge server; The energy constraint condition when the terminal device sends the network content object to the edge server is: in, represents the communication energy consumption of terminal device i, e i represents the communication energy budget of terminal device i; The energy cost of the terminal equipment is: in, Represents the potential communication energy consumption of network content objects.

6. The cloud-edge-device collaborative content caching method according to claim 5, characterized in that: Constructing a unilateral cache strategy according to the constraint condition and the current cache value to obtain the unilateral cache strategy of the edge server and the unilateral cache strategy of the terminal device includes: Input the storage space constraint of the edge server, the storage space constraint of the terminal device, the energy constraint, the energy cost, and the current cache value into the improved ACS algorithm for updating, generating a new unilateral cache strategy and retaining the original unilateral cache strategy; The new unilateral cache strategy A new (t) is: The original unilateral cache strategy A old (t) is: in, represents the cache decision of terminal device i at time slot t-1, represents the cache decision set of terminal device i, Represents the cache decision set of the edge server.

7. The cloud-edge-device collaborative content caching method according to claim 6, characterized in that: The expression of the state space is: in, represents the state space, t represents the time slot t, represents the caching decision of the edge server, Indicates The cache decision of each terminal device, Indicates the communication relationship between the edge server and the terminal device in the current time slot. Indicates the online value of the content in the current time slot.

8. The cloud-edge-device collaborative content caching method according to claim 7, characterized in that: The expression of the reward function is: in, represents the reward function, Indicates that in time slot t The total delay for each terminal device to obtain network content objects.

9. The cloud-edge-device collaborative content caching method according to claim 7, characterized in that: The deep reinforcement learning network consists of a dual deep Q network including a training Q network and a target Q network, and a noisy network; The trained Q network is used to select actions; The target Q network is used to calculate the value of the selected action; The noisy network is used to add Gaussian noise to the training Q network and the target Q network.

10. The cloud-edge-device collaborative content caching method according to claim 7, characterized in that: The loss function of the deep reinforcement learning network is: Among them, ω and ω - They represent the parameter sets of the training Q network and the target Q network, which are updated by back propagation. K represents the amount of data. Indicates in status Take action reward, γ refers to the discount factor, and its value range is [0,1), and They represent the value of taking a specified action under the input state calculated by the training Q network with additional noise and the target Q network, respectively.

Images