A cloud edge-end collaborative content caching method

Abstract

The application provides a cloud-edge-terminal collaborative content caching method, according to the number of network content objects in a network content list required by a terminal device and the popularity of each network content object, an initial caching value is obtained to enable the terminal device to make a content request, an edge server performs online updating on the caching value in each time slot by acquiring and responding to the content request to obtain a current caching value; a single-edge caching strategy is constructed according to the established constraint condition of the edge server and the terminal device and the current caching value, a strategy space of the single-edge caching strategy of the edge server and the single-edge caching strategy of the terminal device is obtained; the constructed state space, reward function and strategy space are combined to obtain an action space, a trained deep reinforcement learning network is inputted to perform action selection, and an optimal action is obtained as a multi-edge collaborative caching strategy; long-term delay is optimized without generating a dimension curse to realize efficient collaborative content caching.

Description

Technical Field

[0001] This invention relates to the field of edge caching technology in mobile communications, and particularly to a cloud-edge-device collaborative content caching method. Background Technology

[0002] In recent years, the rapid development of mobile smart devices has brought great convenience to people's lives, but it has also placed a huge burden on network infrastructure. During peak traffic hours or in areas with high population density, a large number of smart networked devices are concentrated in small hotspots, causing a sharp increase in local internet traffic, severely burdening servers, leading to serious network congestion and unacceptable latency for users. If this situation is not alleviated for a long time, many users will not be able to obtain content within an acceptable time, and repeated requests will further increase, exacerbating network congestion and causing more serious network problems. Therefore, to alleviate the pressure, it is necessary to offload the huge traffic of the traditional internet to other networks. Mobile edge computing networks (MEC) have certain computing and communication capabilities and can share the massive traffic. MEC networks support caching of frequently used content, and through cloud-edge-device collaborative optimization on MEC networks, the pressure on the traditional internet can be effectively alleviated, latency reduced, and user experience improved.

[0003] MEC (Multi-access Edge Computing) networks are a common type of network used in cloud-edge-device optimization, typically consisting of cloud servers, edge servers, and terminal devices. Content caching in MEC networks refers to storing frequently used or requested content for high-speed user access. Caching frequently used content on edge servers closer to users shortens data transmission paths, making data access faster. However, the limited storage capacity of edge servers is still insufficient to handle massive amounts of popular content, necessitating collaboration between terminal devices and edge servers to further enhance caching capabilities.

[0004] To achieve efficient collaborative content caching, it's necessary to coordinate the cached content between edge servers and terminal devices. Some researchers predict the data users might need based on their historical behavior, location, and other characteristics, and pre-cache this anticipated data on edge devices close to the user. However, predictions are inevitably imperfect, and some errors can lead to serious consequences. Many researchers focus on balancing the cached content allocation between edge servers and terminal devices. Both terminal devices and edge servers are self-serving, tending to cache popular content even if it's already cached by other nearby devices or edge servers, wasting storage space. While collaborative content caching can enhance the caching capabilities of edge servers and terminal devices, it remains a challenging problem.

[0005] Currently, collaborative content caching in cloud-edge-device optimization mainly employs two methods: traditional heuristic algorithms and neural network AI algorithms. Traditional heuristic optimization algorithms can effectively solve cached content allocation in static scenarios. However, considering the mobility of terminal devices and various dynamic constraints (such as energy), and the fact that collaborative content caching is a long-term process, the environment comprised of edge servers and terminal devices is highly dynamic. Traditional heuristic optimization methods decompose the dynamic environment into independent static environments for optimization. However, problems in the dynamic environment are interconnected over time, and making decisions based solely on the static environment is one-sided. Thanks to the development of artificial intelligence technology and hardware, many researchers have adopted Deep Reinforcement Learning (DRL) to make decisions on complex problems. However, the diverse content demands of mobile devices generate massive cache combinations, leading to the curse of dimensionality in DRL algorithms. Furthermore, DRL algorithms tend to greedily explore, which may result in different solutions for the same state, posing a danger to devices. Summary of the Invention

[0006] This invention provides a cloud-edge-device collaborative content caching method, the purpose of which is to optimize long-term latency without incurring the curse of dimensionality in order to achieve efficient collaborative content caching.

[0007] To achieve the above objectives, this invention provides a cloud-edge-device collaborative content caching method, applied to a cloud-edge-device collaborative system including cloud servers, edge servers, and terminal devices. The method includes:

[0008] Step 1: Obtain the list of network content required by the terminal device. The list of network content includes multiple network content objects and the popularity of each network content object. The initial cache value is obtained by using the number of network content objects in the list and the popularity of each network content object. Popularity refers to the probability that a network content object is selected first.

[0009] Step 2: Terminal devices within the communication range of the edge server submit content requests based on the initial cache value. The edge server retrieves and responds to the content requests by searching its own cache space, the cache space of the terminal devices, and the storage space of the cloud server.

[0010] Step 3: Update the cache value online in each time slot based on the content request responded by the edge server to obtain the current cache value;

[0011] Step 4: Establish constraints for edge servers and terminal devices, and construct unilateral caching strategies based on constraints and current cache value to obtain unilateral caching strategies for edge servers and terminal devices.

[0012] Step 5: Construct a policy space based on the unilateral caching policies of the edge server and the unilateral caching policies of the terminal device;

[0013] Step 6: Construct a state space and reward function based on the current caching status of the edge server and terminal device. Combine the state space, reward function and policy space to obtain the action space. Input the action space into the trained deep reinforcement learning network to select actions and obtain the optimal action as the multilateral collaborative caching strategy. The multilateral collaborative caching strategy is as follows:

[0014] Terminal devices and / or edge servers execute a one-sided cache update strategy to update the content cache;

[0015] or,

[0016] Terminal devices and / or edge servers maintain the current cached content.

[0017] Furthermore, the expression for calculating the initial cache value is:

[0018]

[0019] in, p represents the initial cache value. s This indicates the popularity of a web content object, where s represents the s-th web content object directory and S represents the number of web content object directories. This represents a set of network content object directories, where ψ represents a fixed parameter with no physical meaning.

[0020] Furthermore, step 2 includes:

[0021] In each time slot, terminal devices within the communication range of the edge server make content requests based on the initial cache value;

[0022] Edge servers determine whether a network content object related to the content request exists by searching their own cache space;

[0023] If the edge server has a network content object related to the content request in its own cache space, the edge server will directly respond to the content request.

[0024] If the edge server does not have a network content object related to the content request in its own cache space, it searches the cache space of the terminal device to determine whether the network content object related to the content request exists in the cache space of the terminal device.

[0025] If the cache space of the terminal device contains a network content object related to the content request, the edge server retrieves and responds to the content request from the cache space of the terminal device.

[0026] If the network content object related to the content request does not exist in the cache space of the terminal device, the edge server retrieves and responds to the content request from the storage space of the cloud server.

[0027] Furthermore, the formula for updating the current cache value is:

[0028]

[0029] in, Indicates the current cache value. This represents the cache value in time slot t-1. This indicates the number of times a network content object has been requested in the current time slot.

[0030] Furthermore, the constraints for establishing edge servers and terminal devices include:

[0031] The storage space constraints for the edge server are:

[0032]

[0033] in, This represents the caching decision of the edge server at time slot t, l s The value represents the size of the network content object, and ts represents the total size of the edge server's own cache space;

[0034] The storage space constraints of the terminal device are as follows:

[0035]

[0036] in, This represents the cache decision of terminal device i in time slot t, ts i This indicates the total size of the terminal device's cache space. This indicates the total number of terminal devices within the communication range of the edge server;

[0037] The energy constraints for terminal devices sending network content objects to edge servers are as follows:

[0038]

[0039] in, e represents the communication power consumption of terminal device i. i This represents the communication energy budget of terminal device i;

[0040] The energy cost of the terminal equipment is:

[0041]

[0042] in, This represents the potential communication energy consumption of network content objects.

[0043] Furthermore, based on the constraints and the current cache value, a one-sided caching strategy is constructed, resulting in the one-sided caching strategy for the edge server and the one-sided caching strategy for the terminal device, including:

[0044] The storage space constraints of the edge server, the storage space constraints of the terminal device, the energy constraints, the energy cost, and the current cache value are input into the improved ACS algorithm to update the new one-sided caching strategy while retaining the original one-sided caching strategy.

[0045] New one-sided caching strategy A new (t) is:

[0046]

[0047] Original one-sided caching strategy A old (t) is:

[0048]

[0049] in, This represents the cache decision of terminal device i in time slot t-1. This represents the cache decision set of terminal device i. This represents the caching decision set of the edge server.

[0050] Furthermore, the expression for the state space is:

[0051]

[0052] in, Let t represent the state space, and t represent time slot t. This indicates the caching decisions of the edge server. This represents the caching decision for the i-th terminal device. This indicates the current communication relationship between the edge server and the terminal device in the current time slot. This indicates the online value of the content in the current time slot.

[0053] Furthermore, the expression for the reward function is:

[0054]

[0055] in, Represents the reward function, This represents the total latency for the i-th terminal device to acquire a specific network content object in time slot t.

[0056] Furthermore, deep reinforcement learning networks consist of dual deep Q-networks, including a training Q-network and a target Q-network, as well as a noisy network;

[0057] Train the Q-network to select actions;

[0058] The target Q-network is used to calculate the value of the selected action;

[0059] Noisy networks are used to add Gaussian noise to both the training Q-network and the target Q-network.

[0060] Furthermore, the loss function of a deep reinforcement learning network is:

[0061]

[0062] Where ω and ω - Let represent the parameter sets for the training Q-network and the target Q-network, respectively, which are updated through backpropagation. K represents the amount of data. Indicates the state Take action below The reward, γ, refers to the discount factor, which ranges from [0,1). and These represent the values ​​of taking a specified action under the input state, calculated by the trained Q-network with added noise and the target Q-network, respectively.

[0063] The above-described solution of the present invention has the following beneficial effects:

[0064] This invention is applied to a cloud-edge-device collaboration system including cloud servers, edge servers, and terminal devices. It obtains a list of network content needed by the terminal devices, which includes multiple network content objects and the popularity of each object. An initial cache value is derived using the number of network content objects in the list and the popularity of each object, where popularity indicates the likelihood of a content object being prioritized. Terminal devices within the communication range of the edge server submit content requests based on this initial cache value. The edge server retrieves and responds to these requests by searching its own cache space, the terminal devices' cache space, and the cloud server's storage space. The cache value is updated online in each time slot based on the content requests responded to by the edge server, yielding the current cache value. A communication mechanism is established between the edge server and the terminal devices. The invention addresses the constraints of caching and constructs unilateral caching strategies based on these constraints and the current caching value, resulting in unilateral caching strategies for edge servers and terminal devices. A policy space is then constructed based on these strategies. A state space and reward function are built based on the current caching status of the edge servers and terminal devices. The state space, reward function, and policy space are combined to obtain an action space, which is then input into a trained deep reinforcement learning network for action selection. The optimal action is then chosen as the multilateral collaborative caching strategy. Compared to existing technologies, this invention considers long-term content caching benefits, as well as the unstable topological connections and complex policy space caused by terminal device mobility. It optimizes long-term latency without incurring the curse of dimensionality to achieve efficient collaborative content caching.

[0065] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram illustrating an application scenario of an embodiment of the present invention;

[0068] Figure 3 This is a flowchart illustrating the improved ACS algorithm in an embodiment of the present invention;

[0069] Figure 4 This is a model structure diagram of the deep reinforcement learning network in an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of the deep reinforcement learning process in an embodiment of the present invention. Detailed Implementation

[0071] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0072] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0073] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0074] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0075] This invention addresses existing problems by providing a cloud-edge-device collaborative content caching method.

[0076] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a cloud-edge-device collaborative content caching method, applied to a cloud-edge-device collaborative system including a cloud server, an edge server, and a terminal device. The method includes:

[0077] Step 1: Obtain the list of network content required by the terminal device. The list of network content includes multiple network content objects and the popularity of each network content object. The initial cache value is obtained by using the number of network content objects in the list and the popularity of each network content object. The popularity refers to the probability that a network content object is selected first. The higher the popularity, the greater the probability that the network content object is selected first.

[0078] Step 2: Terminal devices within the communication range of the edge server submit content requests based on the initial cache value. The edge server retrieves and responds to the content requests by searching its own cache space, the cache space of the terminal devices, and the storage space of the cloud server.

[0079] Step 3: Update the cache value online in each time slot based on the content request responded by the edge server to obtain the current cache value;

[0080] Step 4: Establish constraints for edge servers and terminal devices, and construct unilateral caching strategies based on constraints and current cache value to obtain unilateral caching strategies for edge servers and terminal devices.

[0081] Step 5: Construct a policy space based on the unilateral caching policies of the edge server and the unilateral caching policies of the terminal device;

[0082] Step 6: Construct a state space and reward function based on the current caching status of the edge server and terminal device. Combine the state space, reward function and policy space to obtain the action space. Input the action space into the trained deep reinforcement learning network to select actions and obtain the optimal action as the multilateral collaborative caching strategy. The multilateral collaborative caching strategy is as follows:

[0083] Terminal devices and / or edge servers execute a one-sided cache update strategy to update the content cache;

[0084] or,

[0085] Terminal devices and / or edge servers maintain the current cached content.

[0086] In this embodiment of the invention, it should be noted that the cloud server in the cloud-edge-device collaboration system is located in a remote data center and has powerful computing and storage capabilities to process complex applications and algorithms.

[0087] Edge servers serve as a bridge for communication between users and cloud servers;

[0088] Terminal devices typically refer to a user's personal devices, such as smart computers, smart cars, and tablets. They have limited storage capacity, computing resources, and energy, and can only undertake some tasks. In this embodiment of the invention, there are multiple terminal devices.

[0089] Step 1: Obtain the list of network content required by the terminal device. The list of network content includes multiple network content objects and the popularity of each network content object. The initial cache value is obtained by using the number of network content objects in the list and the popularity of each network content object.

[0090] In order to enable fine-grained processing of collaborative caching between edge servers and terminal devices, and considering that the number of network content objects in the network is variable, this embodiment of the invention treats the collaborative caching process as a multi-round service. The network content list is the same in one round of service, and the difference between different rounds of service is that the network content list is different. The one-round service process is divided into several time slots. In the first time slot of a round of service, the network content list must first be obtained.

[0091] Among them, network content objects are the network content objects required by the terminal device, and the collection of network content objects is represented as follows: The Network Content Object Directory is represented as... Network Content Object Features include size l s The value of network content objects in time slots The value set of time slots is represented as

[0092] Content value is a key factor in caching. Value can be represented by popularity, reflecting the likelihood that a particular piece of online content will be preferred over other content, denoted as p. s ,in

[0093]

[0094] In this embodiment of the invention, the product of the number of network content objects and their popularity is used as the initial cache value, and the calculation expression is as follows:

[0095]

[0096] in, p represents the initial cache value. s This indicates the popularity of a web content object, where s represents the s-th web content object directory and S represents the number of web content object directories. This represents a set of network content object directories, where ψ represents a fixed parameter with no physical meaning.

[0097] Step 2: Terminal devices within the communication range of the edge server submit content requests based on the initial cache value. The edge server retrieves and responds to the content requests by searching its own cache space, the cache space of the terminal devices, and the storage space of the cloud server.

[0098] Specifically, step 2 includes:

[0099] In each time slot, terminal devices within the communication range of the edge server make content requests based on the initial cache value;

[0100] Edge servers determine whether a network content object related to the content request exists by searching their own cache space;

[0101] If the edge server has a network content object related to the content request in its own cache space, the edge server will directly respond to the content request.

[0102] If the edge server does not have a network content object related to the content request in its own cache space, it searches the cache space of the terminal device to determine whether the network content object related to the content request exists in the cache space of the terminal device.

[0103] If the retrieval is successful, it means that there is a network content object related to the content request in the cache space of the terminal device. The edge server retrieves and responds to the content request from the cache space of the terminal device.

[0104] If the retrieval fails, it means that the network content object related to the content request does not exist in the cache space of the terminal device. The edge server then retrieves and responds to the content request from the storage space of the cloud server.

[0105] Because cached values ​​gradually deviate from content requests over time, in order to better reflect the needs of network content objects, this embodiment of the invention adopts the Least Frequently Used (LFU) cache update algorithm to adjust cached values ​​online. A counter is set for each network content object to record the number of requests, denoted as... Therefore, the formula for updating the current cache value is defined as follows:

[0106]

[0107] in, Indicates the current cache value. This represents the cache value in time slot t-1. This indicates the number of times a network content object has been requested in the current time slot.

[0108] Before edge servers and terminal devices can collaborate, they should have their own caching strategies. These caching strategies are formulated without knowing the cached information of other devices or edge servers and only considering their own situation. This is called a one-sided caching strategy. The existence of a one-sided caching strategy ensures that edge servers or terminal devices can still use the content caching function normally even without collaboration.

[0109] This invention defines the attributes of the edge server as follows: Where ts represents the total size of the cache space. Let A(t) represent the device communicating with the edge server in time slot t, where t∈T={1,…,T}, and let A(t) represent the caching decision of the edge server in time slot t. Each It is defined as a 0-1 binary variable.

[0110] Specifically, if This represents the network content object c. s Not cached, while a s =1 indicates that the network content object c s It is cached. The cached network content objects are limited by the storage space capacity of the edge server. Therefore, the storage space constraint of the edge server is:

[0111]

[0112] in, This represents the caching decision of the edge server at time slot t, l s The value represents the size of the network content object, and ts represents the total size of the edge server's own cache space;

[0113] In time slot t, devices within the communication range of the edge server are identified as... Each terminal device v i Depend on Indicate its characteristics, where e i ts represents the communication energy budget. i Indicates the size of the cache space. Indicates v i Distance to edge servers Indicates terminal device v i caching decisions, Similar to edge servers, This is a binary variable ranging from 0 to 1. A value of 0 indicates that the network content object is not cached by this device, while a value of 1 indicates that it is cached. The caching of network content objects is limited by the storage capacity of the terminal device. Therefore, the storage space constraint of the terminal device is:

[0114]

[0115] in, This represents the cache decision of terminal device i in time slot t, ts i This indicates the total size of the terminal device's cache space. This indicates the total number of terminal devices within the communication range of the edge server;

[0116] When targeting When a request is submitted, the edge server searches its cache space. If Edge servers can access network content objects. Send to v i .if However, vehicles within the communication range have The edge server can connect with the j-th terminal device vj To communicate and obtain network content objects

[0117] To simplify calculations, the latency for retrieving network content objects from the cloud server is set to a fixed value, denoted as D. C This value is relatively large compared to other methods; in the communication between the edge server and the terminal device, orthogonal frequency division multiple access (OFDMA) technology is used, which allows multiple users to share channel resources, specifically:

[0118] Let v i The allocated bandwidth B has a transmit power of p. i The channel gain is h i , σ 2 If we represent background noise, then the communication rate can be expressed as:

[0119]

[0120] Similar to terminal devices, the communication power of edge servers is p. u The channel gain is h u The communication rate is then expressed as:

[0121]

[0122] Therefore, c s The formula for calculating transmission delay is:

[0123]

[0124] The calculation of communication energy consumption is expressed as follows:

[0125]

[0126] The energy of terminal devices is limited, and when edge servers need v i When the network content object is v i Communication is required, and this communication must comply with energy constraints. Therefore, the energy constraint condition when the terminal device sends a network content object to the edge server is:

[0127]

[0128] in, e represents the communication power consumption of terminal device i. i This represents the communication energy budget of terminal device i;

[0129] Furthermore, to ensure proper collaboration and prevent devices from failing to provide network content objects that meet energy constraints due to insufficient power, terminal devices need to consider the potential energy consumption of network content objects. The potential communication energy consumption of network content objects is expressed as... The calculation method is as follows:

[0130]

[0131] in, Represents the network content object c s The cache hit rate is calculated by dividing the number of hits by the total number of requests for that network content object.

[0132] Since the terminal equipment needs to be estimated to avoid potential communication costs exceeding the allocated communication energy consumption budget, the energy cost of the terminal equipment is expressed as:

[0133]

[0134] in, This represents the potential communication energy consumption of network content objects.

[0135] Specifically, based on constraints and current cache value, a one-sided caching strategy is constructed to obtain the one-sided caching strategy for the edge server and the one-sided caching strategy for the terminal device, including:

[0136] The storage space constraints of the edge server, the storage space constraints of the terminal device, the energy constraints, the energy cost, and the current cache value are input into the improved ACS algorithm to update the new one-sided caching strategy while retaining the original one-sided caching strategy.

[0137] New one-sided caching strategy A new (t) is:

[0138]

[0139] Original one-sided caching strategy A old (t) is:

[0140]

[0141] in, This represents the cache decision of terminal device i in time slot t-1. This represents the cache decision set of terminal device i. This represents the caching decision set of the edge server.

[0142] In this embodiment of the invention, when an edge server or terminal device receives a new network content object, an improved ACS algorithm is used to update the local cache value to find the best combination of network content objects. The algorithm steps are as follows:

[0143] First, a directed graph is constructed based on the edge server or terminal device according to the original caching strategy. in and Let represent the sets of vertices and edges, respectively. Each cached network content object is represented as a vertex, and edges connect every two vertices. Vertex s0 is the initial vertex. The old optimal solution A old The total cache value of (t) is W. old (t), calculated as follows:

[0144]

[0145] Initialize new solution A new (t) and total value W new =0. The visited edge record is CE. new In this embodiment of the invention, the path pheromone is initialized based on the old optimal solution and the current cache value.

[0146] Specifically, edge(s) v ,s w The pheromone concentration on the surface is expressed as:

[0147]

[0148] The goal is to find a one-sided caching strategy A at time slot t that maximizes the total value of content under constraints. new (t), represented as follows:

[0149]

[0150] Let cv represent the vertices and edges visited by ant j. j and ce j j selects the next node as follows:

[0151]

[0152] Here, θ represents a random number uniformly distributed in the interval [0,1], while θ0 is a preset parameter used to adjust the trade-off between utilizing existing experience and exploring new paths, and α and β are heuristic factors.

[0153] Whenever ant j completes its search, it updates the pheromones of the edges it has traversed in the following way:

[0154]

[0155] Here, 0<ξ1<1 represents the rate of pheromone evaporation. This local update method enhances the exploratory ability and guides ants to explore unexplored edges.

[0156] When ant j is preparing to move to the new vertex s wAt that time, it calculates the sum of the size and value of the visited vertices, and according to the communication energy consumption expression, if the size of the network content object satisfies the storage space constraint and the energy satisfies the communication energy consumption and... Then vertex s w and edge (s v ,s w ) will be marked as visited; otherwise, ant j will end the search.

[0157] After all ants have completed their search, a global pheromone update will be performed. Only the edges of the optimal solution will have their pheromones updated. The update formula is as follows:

[0158]

[0159] Among them, ξ2 is similar to ξ1 and satisfies 0 < ξ2 < 1.

[0160] When the maximum number of iterations is reached or the solution is no longer updated, the optimal solution becomes A. new (t).

[0161] It is worth noting that A new (t) does not necessarily contain new network content objects; they represent a one-way caching strategy when new network content objects are received.

[0162] In comparison, A old (t) is the caching strategy before receiving a new network content object, which means that if the cost value of the new network content object is low, A new (t) and A old (t) may be the same.

[0163] Step 5: Construct a policy space based on the unilateral caching policies of the edge server and the unilateral caching policies of the terminal device.

[0164] Before constructing the collaboration strategy, it is necessary to obtain the one-sided caching strategy A. old (t) This process involves organization and simplification because, in cloud-edge-device optimization, the communication range is limited, and the number of terminal devices is capped. However, the number of network content objects is uncertain and a large number. Even if the number of terminal devices is 1, the number of network content objects can easily exceed 100, resulting in 2 100 With such combinations, if the terminal devices become more advanced, there will be an even larger policy space, which greatly affects the efficiency of the algorithm. Therefore, it is necessary to construct a streamlined policy space.

[0165] The embodiment of the present invention uses a size of The space is used to represent the collaborative cache update strategy.

[0166] Specifically, the strategy space for streamlined collaborative content caching can be represented as follows: in When m = 0, it represents the collaboration strategy of the edge server; when m ≠ 0, it represents the collaboration strategy of the terminal device. This indicates that the corresponding edge server and / or terminal device updates its content cache, i.e., using A. new (t), otherwise use A. old (t), specifically represented as follows:

[0167]

[0168] Without constructing a streamlined policy space, one must confront a space of size [missing information]. Thanks to the processing in the above formula, the action space now has a size of [size missing]. The action space, the size of which depends on the number of vehicles communicating with the edge server in the current time slot, is not very large. Therefore, it greatly simplifies the policy space and provides action space for constructing collaborative caching policies.

[0169] Step 6: Construct a state space and reward function based on the current caching status of the edge server and terminal device. Combine the state space, reward function and policy space to obtain the action space. Input the action space into the trained deep reinforcement learning network to select actions and obtain the optimal action as the multilateral collaborative caching strategy. The multilateral collaborative caching strategy is as follows:

[0170] Terminal devices and / or edge servers execute a one-sided cache update strategy to update the content cache;

[0171] or,

[0172] Terminal devices and / or edge servers maintain the current cached content.

[0173] In this embodiment of the invention, the purpose of constructing a multilateral caching strategy is to consider the global impact from a long-term perspective, coordinate the unilateral caching update strategies of edge servers and terminal devices, and avoid short-sighted updates. For example, a terminal device might want to cache popular online content objects, even if they have already been cached by other devices nearby. Doing so would waste space. Simply setting the cache to prevent duplicate caching of online content objects already cached by other devices is unreasonable, as it would lead to the loss of online content objects when the device leaves, which is a pitfall to avoid. Therefore, constructing a multilateral caching strategy must consider both long-term and global impacts.

[0174] To construct a multi-sided caching strategy, a deep reinforcement learning network is used based on the aforementioned action space. The specific process is as follows:

[0175] First, the time required for the edge server to retrieve network content objects is expressed as:

[0176]

[0177] Then, the terminal device v i The total latency for retrieving network content objects is D. i,s (t) is represented as:

[0178]

[0179] In each time slot t, the terminal device generates a request, and the edge server searches for network content objects to respond to these requests. This embodiment of the invention treats the process of the edge server and terminal device updating their caches upon receiving a new network content object as a Markov Decision Process (MDP), and defines the objective as minimizing long-term latency, which can be expressed as:

[0180]

[0181] We represent MDP as in Representing the state space, Represents the action space, Represents the state transition probability. Represents the reward function. The edge server, based on the state space... From the action space Choose an action based on the state transition probability. Transition to the next state and according to the reward function. Receive the reward.

[0182] Specifically, the state space is the environment, which is the basis for decision-making and the object of influence. In this embodiment of the invention, the cache space, connection state, and content value of the edge server and terminal device are used as the state space, and its expression is:

[0183]

[0184] in, Let t represent the state space, and t represent time slot t. This indicates the caching decisions of the edge server. Indicates the first caching decisions for terminal devices This indicates the current communication relationship between the edge server and the terminal device in the current time slot. This indicates the online value of the content in the current time slot.

[0185] Specifically, the action space includes all possible decision actions. In this embodiment of the invention, the action space is the simplified strategy space described above, defined as:

[0186]

[0187] Specifically, the state transition probability represents the probability of taking an action. From state to state The probability of is defined as:

[0188]

[0189] Specifically, edge servers receive different rewards for making different action decisions. The neural network is then optimized based on these rewards. Since the goal of a Markov decision process is to minimize long-term latency, in reinforcement learning, the edge server optimizes its policy according to the reward function to maximize the reward. The expression for the reward function is defined as follows:

[0190]

[0191] in, Represents the reward function, This represents the total latency for the i-th terminal device to acquire a specific network content object in time slot t.

[0192] Because in MDP, the benefits It is a key factor influencing decision-making. Benefits Defined as the sum of all rewards from the start of the task until the termination of the MDP, the calculation formula is:

[0193]

[0194] Where γ represents the attenuation factor.

[0195] In this embodiment of the invention, the behavioral policy of an agent in reinforcement learning is defined as π, representing the tendency to choose actions in reinforcement learning, which is only related to the current state. It is worth noting that policy π is a concept in reinforcement learning that guides the selection of actions. Indicates the state If the probability of taking action 'a' is given, then the expected reward obtained by following the policy π is denoted as π / 2. The calculation formula is as follows:

[0196]

[0197] The update formula is expressed as:

[0198]

[0199] Where λ represents the update step size.

[0200] The goal of reinforcement learning is to find the optimal solution. The strategy π is the value of the cache. In cloud-edge-device collaborative caching, optimizing strategy π involves solving the correspondence between cache state, cache strategy, and cache value for edge servers and terminal devices to construct an efficient cache strategy, thereby optimizing cloud-edge-device collaborative caching. However, considering the complexity of the state and action spaces in cloud-edge-device collaborative caching, traditional methods struggle to record state-action-value relationships. Therefore, it is necessary to combine deep learning and reinforcement learning, using neural networks to solve this problem.

[0201] The preferred approach is to use a deep reinforcement learning network consisting of a Double Deep Q-learning Network (DDQN) comprising a training Q-network and a target Q-network, along with a noisy network (NoisyNet). The training Q-network is used to select actions, the target Q-network is used to calculate the value of the selected actions, and the noisy network is used to add Gaussian noise to the training and target Q-networks. This combination can effectively prevent the Q-network from overestimating the value of actions.

[0202] Specifically, the embodiments of the present invention use ω and ω - Let represent the parameter set of the dual deep Q-network DDQN, where ω represents the parameters trained on the network. - The parameters representing the target network can be used to write the target of DDQN as:

[0203]

[0204] Furthermore, greedy action selection is unstable; for example, in a greedy method, even based on the same state, the actions taken may differ. To avoid this problem, embodiments of the present invention add Gaussian noise to the network using Noisy Net, changing Q to... By combining DDQN and NoisyNet, the optimization objective is reformulated as follows:

[0205]

[0206] Finally, the loss function for deep reinforcement learning networks is defined as follows:

[0207]

[0208] Where ω and ω - Let represent the parameter sets for the training Q-network and the target Q-network, respectively, which are updated through backpropagation. K represents the amount of data. Indicates the state Take action below The reward, γ, refers to the discount factor, which ranges from [0,1). and These represent the values ​​of taking a specified action under the input state, calculated by the trained Q-network with added noise and the target Q-network, respectively.

[0209] Furthermore, due to the parameters ω and ω0 of the training Q-network and the target Q-network... - The updates are not synchronous. The parameters ω of the Q-network are updated during each training session, and ω is only used to update ω after a certain step size is met. - This is the delayed update strategy.

[0210] Based on the above conditions, a deep reinforcement learning network can be trained using the acquired data.

[0211] A well-trained deep reinforcement learning network can be used to select the optimal action based on the current environment, which is a multi-sided caching strategy. The cache can then be updated according to this strategy, thereby optimizing long-term latency.

[0212] One-sided caching strategies ensure that both edge servers and terminal devices have available caching options in any collaborative relationship. However, one-sided caching only considers the immediate interests of the individual device or edge server, neglecting the overall long-term relationship and representing a locally optimal solution. Implementing only a one-sided caching strategy can limit cloud-edge-device collaborative caching. For example, selfish terminal devices may cache high-value content even if this content has already been cached by multiple neighboring devices, resulting in redundancy of content objects within the same edge server's coverage area. Furthermore, if a specific network content object has only one copy within a region, the region will lose the cached content object when its owner leaves, which is a pitfall to avoid.

[0213] Multi-sided caching, unlike single-sided caching, is a caching strategy that comprehensively considers the long-term interests of multiple parties. It represents a comprehensive adjustment of the single-sided caching strategies used by edge servers and terminal devices. Specifically, when new content is received, the terminal device and edge server will have two caching strategies—a new one and a old one—using the methods described above, i.e., A. new (t) and A old (t) If the new one-sided caching strategy is adopted for all, it will fall into the trap mentioned above. A multi-sided caching strategy, however, considers the overall benefit and determines the combination of terminal devices and edge servers that need to be updated according to the new strategy, as well as the combination that executes the old strategy. For example, ... Figure 2As shown, the edge server caches content A, C, and D; device 1 caches content A and C; and device 2 caches content E. Assuming both device 1 and device 2 request content B, if only a one-sided strategy is implemented, the edge server will replace D with B, device 1 will replace C with B, and device 2 will replace E with B, resulting in the loss of cached content D and E, and limited benefits. Through a multi-sided caching strategy, the edge server adopts the old strategy, device 1 adopts the new strategy, and device 2 adopts the old strategy, ensuring low-latency retrieval of the popular content B without losing D and E. Therefore, the multi-sided caching strategy determines whether the edge server or terminal device updates its cache. As mentioned earlier, the optimal action output by the deep reinforcement learning network corresponds to the cache update decisions of the terminal device and the edge server, and thus can be considered a multi-sided caching strategy. Once the multi-sided caching strategy is obtained, the action number can be parsed using binary conversion as mentioned above. A set, in other words, where each bit of a binary number corresponds to a single bit. according to The value of determines whether the corresponding edge server or terminal device adopts a new caching strategy and updates the cache. Specifically, when In this case, the terminal device and / or edge server represented by m is considered to adopt a conservative strategy, that is, avoid updating the cached content and maintain the current cached content. At that time, the terminal device and / or edge server represented by m should adopt a new caching strategy to update. The edge server and terminal device are updated according to the multilateral caching strategy, thereby achieving optimized caching of cloud-edge-device collaborative content.

[0214] This invention provides an embodiment of the method for obtaining a list of network content needed by a terminal device. Based on the number of network content objects in the list and the popularity of each object, an initial cache value is determined. Terminal devices within the communication range of an edge server then submit content requests based on this initial cache value. The edge server retrieves and responds to these requests by searching its own cache space, the terminal device's cache space, and the cloud server. The cache value is updated online in each time slot based on the content requests responded to by the edge server, resulting in the current cache value. Constraints are established between the edge server and the terminal device, and a one-sided caching strategy is constructed based on these constraints and the current cache value to obtain the edge server's one-sided cache. The invention proposes a multi-party collaborative caching strategy, which considers long-term content caching benefits as well as the unstable topological connections and complex strategy spaces caused by terminal device mobility. It optimizes long-term latency without inducing dimensionality curse to achieve efficient collaborative content caching. The strategy space is constructed based on the unilateral caching strategies of edge servers and terminal devices. A policy space is built using the unilateral caching strategies of edge servers and terminal devices, and a state space and reward function are constructed based on the current caching status of edge servers and terminal devices. The state space, reward function, and policy space are then combined to obtain an action space, which is input into a trained deep reinforcement learning network for action selection, yielding the optimal action as the multi-party collaborative caching strategy. Compared with existing technologies, this invention considers long-term content caching benefits as well as the unstable topological connections and complex policy spaces caused by terminal device mobility, optimizing long-term latency to achieve efficient collaborative content caching without inducing dimensionality curse.

[0215] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A cloud-edge-device collaborative content caching method, applied to a cloud-edge-device collaborative system including a cloud server, an edge server, and terminal devices, characterized in that, The methods include: Step 1: Obtain the list of network content required by the terminal device. The list of network content includes multiple network content objects and the popularity of each network content object. The initial cache value is obtained by using the number of network content objects in the list and the popularity of each network content object. The popularity refers to the probability that a network content object is selected first. Step 2: A terminal device within the communication range of the edge server makes a content request based on the initial cache value. The edge server retrieves and responds to the content request by searching its own cache space, the cache space of other terminal devices within the communication range of the edge server that are different from the terminal device that made the request, and the storage space of the cloud server. Step 3: Update the cache value online in each time slot according to the content request responded by the edge server to obtain the current cache value; Step 4: Establish constraints on the edge server and the terminal device, and construct a one-sided caching strategy based on the constraints and the current cache value to obtain the one-sided caching strategy of the edge server and the one-sided caching strategy of the terminal device. Step 5: Construct a policy space based on the one-sided caching policy of the edge server and the one-sided caching policy of the terminal device; Step 6: Construct a state space and a reward function based on the current caching status of the edge server and the terminal device. Combine the state space, the reward function, and the policy space to obtain an action space. Input the action space into the trained deep reinforcement learning network to select actions and obtain the optimal action as a multilateral collaborative caching strategy. The multilateral collaborative caching strategy is as follows: The terminal device and / or the edge server execute a one-sided cache update strategy to update the content cache; or, The terminal device and / or the edge server maintain the current cached content.

2. The cloud-edge-device collaborative content caching method according to claim 1, characterized in that, The expression for calculating the initial cache value is: in, p represents the initial cache value. s This indicates the popularity of a web content object, where s represents the s-th web content object directory and S represents the number of web content object directories. This represents a set of network content object directories, where ψ represents a fixed parameter with no physical meaning.

3. The cloud-edge-device collaborative content caching method according to claim 2, characterized in that, Step 2 includes: In each time slot, a terminal device within the communication range of the edge server makes a content request based on the initial cache value; The edge server determines whether there is a network content object related to the content request by searching its own cache space. If the edge server has a network content object related to the content request in its own cache space, the edge server directly responds to the content request. If the edge server does not have a network content object related to the content request in its own cache space, the edge server searches the cache space of other terminal devices different from the terminal device that made the request, and determines whether the cache space of other terminal devices different from the terminal device that made the request contains a network content object related to the content request. If a network content object related to the content request exists in the cache space of another terminal device that is different from the terminal device that made the request, the edge server retrieves and responds to the content request from the cache space of the other terminal device that made the request. If the cache space of other terminal devices, different from the terminal device that made the request, does not contain a network content object related to the content request, the edge server retrieves and responds to the content request from the storage space of the cloud server.

4. The cloud-edge-device collaborative content caching method according to claim 3, characterized in that, The formula for updating the current cache value is: in, Indicates the current cache value. This represents the cache value in slot t-1. This indicates the number of times a network content object has been requested in the current time slot.

5. The cloud-edge-device collaborative content caching method according to claim 4, characterized in that, Establish constraints for the edge server and the terminal device, including: The storage space constraints of the edge server are as follows: in, This represents the caching decision of the edge server at time slot t, l s The value represents the size of the network content object, and ts represents the total size of the edge server's own cache space; The storage space constraints of the terminal device are as follows: in, This represents the cache decision of terminal device i in time slot t, ts i This indicates the total size of the terminal device's cache space. This indicates the total number of terminal devices within the communication range of the edge server; The energy constraint condition for the terminal device to send network content objects to the edge server is: in, e represents the communication power consumption of terminal device i. i This represents the communication energy budget of terminal device i; The energy cost of the terminal device is: in, This represents the potential communication energy consumption of network content objects.

6. The cloud-edge-device collaborative content caching method according to claim 5, characterized in that, Based on the constraints and the current cache value, a one-sided caching strategy is constructed to obtain the one-sided caching strategy for the edge server and the one-sided caching strategy for the terminal device, including: The storage space constraints of the edge server, the storage space constraints of the terminal device, the energy constraints, the energy cost, and the current cache value are input into the improved ACS algorithm to update the algorithm, generate a new one-sided caching strategy, and retain the original one-sided caching strategy. The new one-sided caching strategy A new (t) is: The original one-sided caching strategy A old (t) is: in, This represents the buffering decision of terminal device t in time slot t-1. This represents the cache decision set of terminal device i. This represents the caching decision set of the edge server.

7. The cloud-edge-device collaborative content caching method according to claim 6, characterized in that, The expression for the state space is: in, Let t represent the state space, and t represent time slot t. This indicates the caching decisions of the edge server. Indicates the first caching decisions for terminal devices This indicates the current communication relationship between the edge server and the terminal device in the current time slot. This indicates the online value of the content in the current time slot.

8. The cloud-edge-device collaborative content caching method according to claim 7, characterized in that, The expression for the reward function is: in, Represents the reward function, This represents the total latency for the i-th terminal device to acquire network content objects in time slot t.

9. The cloud-edge-device collaborative content caching method according to claim 7, characterized in that, The deep reinforcement learning network consists of a dual deep Q network including a training Q network and a target Q network, as well as a noisy network; The trained Q-network is used to select actions; The target Q-network is used to calculate the value of the selected action; The noisy network is used to add Gaussian noise to the training Q-network and the target Q-network.

10. The cloud-edge-device collaborative content caching method according to claim 7, characterized in that, The loss function of the deep reinforcement learning network is: Where ω and ω - Let represent the parameter sets for the training Q-network and the target Q-network, respectively, which are updated through backpropagation. K represents the amount of data. Indicates the state Take action below The reward, γ, refers to the discount factor, which ranges from [0,1). and These represent the values ​​of taking a specified action under the input state, calculated by the trained Q-network with added noise and the target Q-network, respectively.

Images