Edge service cache optimization method and device, computer equipment and storage medium

Abstract

The invention relates to an edge service cache optimization method and device, computer equipment and a storage medium. The method comprises the following steps: modeling an edge service cache optimization problem into an MDP model, and collecting state information of a current environment through an intelligent agent at each time step; based on the state information of the current environment, the intelligent agent adopts a depth deterministic strategy gradient algorithm to optimize a cache strategy, and generates a cache strategy through a strategy network; executing the caching strategy, calculating a service access delay after executing the caching strategy through a value network, and calculating a reward function according to the service access delay; and evaluating the execution effect of the cache strategy according to the reward function, and optimizing the strategy network to generate an optimal target network. According to the embodiment of the invention, through intelligent agent learning and cache strategy optimization, the storage resources of the edge server can be utilized more effectively, so that the cache hit rate, the resource utilization efficiency and the service quality are improved.

Description

Technical Field

[0001] The present application belongs to the field of edge computing technology, and in particular, relates to an edge service cache optimization method, device, computer equipment, and storage medium. Background Art

[0002] In modern edge computing systems, edge service caching is a key technology to improve service quality and network performance. With the development of the Internet of Things (IoT) and edge computing, edge servers need to handle a large number of user requests and effectively use limited storage resources to provide services. Traditional cache optimization strategies are mainly based on content popularity and user preferences, using static caching strategies to store popular content to reduce network latency and bandwidth consumption. Common cache optimization algorithms include LFU (Least Frequently Used), LRU (Least Recently Used), and Most Recently Used (MRU). Among them, the LFU algorithm determines the cache content based on the frequency of content requests, and the entries with the lowest access frequency stored in the cache will be replaced. The LRU algorithm determines the cache content based on the time when the content was most recently accessed, and the content in the cache that has not been accessed for the longest time will be replaced. Some advanced solutions try to optimize cache strategies based on user behavior and request patterns, such as using user historical data to predict future requests.

[0003] In summary, existing cache optimization algorithms need to rely on static rules or fixed strategies to manage cache content, and still have the following shortcomings:

[0004] 1) Poor adaptability: Existing cache optimization algorithms are usually based on static rules and fixed algorithms, which cannot dynamically adapt to the rapid changes in content popularity and user request patterns. In a dynamic network environment, the cache strategy may not be adjusted in real time, resulting in low cache hit rate and degraded service quality.

[0005] 2) Prediction Difficulty: Traditional cache optimization methods rely on predictions of content popularity and user behavior, which are usually based on historical data. However, popularity and user demand are dynamic and difficult to predict, especially when facing bursty traffic or changing request patterns, traditional methods have limited predictive capabilities.

[0006] 3) Uneven resource utilization: Due to the static characteristics of traditional algorithms, some content may be over-cached, while other important content is not fully utilized, resulting in uneven utilization of edge server storage resources and affecting the overall service quality.

[0007] 4) Unable to handle complex network conditions: In industrial networks and edge computing environments, traffic loads and network conditions change frequently. Traditional caching solutions cannot fully consider these dynamic characteristics, thus affecting cache performance and network efficiency. Summary of the invention

[0008] The present application provides an edge service cache optimization method, apparatus, computer device and storage medium, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0009] In order to solve the above problems, this application provides the following technical solutions:

[0010] An edge service cache optimization method, comprising:

[0011] The edge service cache optimization problem is modeled as an MDP model, and at each time step, the state information of the current environment is collected through the agent;

[0012] Based on the state information of the current environment, the intelligent agent optimizes the cache strategy using a deep deterministic policy gradient algorithm and generates a cache strategy through a policy network;

[0013] Executing the cache strategy, and calculating the service access delay after executing the cache strategy through the value network, and calculating the reward function according to the service access delay;

[0014] The execution effect of the cache strategy is evaluated according to the reward function, and the strategy network is optimized to generate an optimal target network.

[0015] The technical solution adopted by the embodiment of the present application also includes: modeling the edge service cache optimization problem as an MDP model, specifically:

[0016] The MDP model includes a state space, an action space and a reward function. The state space includes the state information of the current environment, and the state information includes device requests, network conditions and cache status; the action space includes cache addition, cache update and cache deletion, and the reward function R t =-Service access delay, used to evaluate the agent's choice of strategy to reduce delay.

[0017] The technical solution adopted by the embodiment of the present application also includes: after modeling the edge service cache optimization problem as an MDP model, it also includes:

[0018] Initialize the cache state of the edge server, clear the cache and set the initial cache capacity; initialize the strategy network, value network and experience replay pool of the agent, and set the parameters of the target network as the initial parameters of the main network.

[0019] The technical solution adopted in the embodiment of the present application also includes: the intelligent agent includes a policy network, a value network and an experience replay pool, the policy network is composed of a certain number of layers of deep neural networks, the input is the state information of the current environment, the corresponding cache operation is generated according to the state information, and the cache strategy is output, and the training process of the policy network is stabilized by delayed updating to finally generate a target network; the value network includes a certain number of layers of deep neural networks, the input is the state information of the current environment and an action pair, the output is the Q value of the corresponding action, and the long-term return of the cache operation generated by the policy network is evaluated, the value of the cache strategy is calculated, and the delayed updated value network is used to stabilize the Q value training process; the experience replay pool is used to store the historical data of the interaction between the intelligent agent and the environment, and the model is trained by randomly extracting the historical data until the optimal target network is generated.

[0020] The technical solution adopted by the embodiment of the present application also includes: generating a corresponding cache operation according to the state information, specifically:

[0021] The cache operation includes cache addition, cache update or cache deletion. The cache addition is to assign priority to the content to be cached based on the demand and value of the content, select new content to add to the cache according to the selection criteria, and update the cache status; the cache update is to update the version of the content in the cache or supplement new information, and replace the low-priority or outdated content already in the cache; the cache deletion is to select the content to be deleted in the cache based on the frequency of use and obsolescence of the content, remove the selected content from the cache, and update the cache status.

[0022] The technical solution adopted by the embodiment of the present application also includes: the service access delay after executing the cache strategy through the value network calculation, and after calculating the reward function according to the service access delay, it also includes:

[0023] The state information of the current environment of the edge server is updated to the next state information, and the state information, action, reward function and next state information of the current environment of the edge server are stored in the experience replay pool as experience samples for training the model.

[0024] The technical solution adopted in the embodiment of the present application also includes: evaluating the execution effect of the cache strategy according to the reward function, and optimizing the strategy network to generate the optimal target network, specifically:

[0025] A batch of experience samples are randomly selected from the experience replay pool to iteratively train the policy network and the value network until the strategy converges or a predetermined number of training times is reached, thereby obtaining an optimal target network.

[0026] Another technical solution adopted by the embodiment of the present application is: an edge service cache optimization device, comprising:

[0027] Information collection module: used to model the edge service cache optimization problem as an MDP model and collect the state information of the current environment through the agent at each time step;

[0028] Strategy generation module: used for optimizing the cache strategy by the agent using a deep deterministic policy gradient algorithm based on the state information of the current environment, and generating a cache strategy through a policy network;

[0029] Strategy execution module: used to execute the cache strategy, calculate the service access delay after executing the cache strategy through the value network, and calculate the reward function according to the service access delay;

[0030] Strategy optimization module: used to evaluate the execution effect of the cache strategy according to the reward function, and optimize the strategy network to generate an optimal target network.

[0031] Another technical solution adopted by the embodiment of the present application is: a computer device, the computer device includes a processor and a memory coupled to the processor, wherein:

[0032] The memory stores program instructions for implementing the edge service cache optimization method;

[0033] The processor is used to execute the program instructions stored in the memory to control the edge service cache optimization method.

[0034] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the edge service cache optimization method.

[0035] Compared with the prior art, the beneficial effects of the embodiments of the present application are: the edge service cache optimization method, device, computer equipment and storage medium of the embodiments of the present application adopt an edge service cache optimization strategy based on deep reinforcement learning, modeling the edge service cache problem as a Markov decision process, the intelligent agent observes the state information of the current environment at each time step, takes corresponding actions to make cache decisions and generate cache strategies, and calculates the service access delay after executing the cache strategy, calculates the reward function according to the service access delay, evaluates the execution effect of the cache strategy through the reward function, and continuously optimizes the edge service cache strategy, thereby improving the cache hit rate, resource utilization efficiency and service quality. The embodiments of the present application adopt a deep reinforcement learning algorithm, which can adapt to the dynamic changes of network conditions and request patterns in real time, comprehensively considers multiple performance goals such as service access delay and cache hit rate, and realizes the overall optimization of performance goals by designing a reward function that adapts to performance goals, thereby improving the overall performance and user experience of the system. By dynamically updating cache content and strategies, the system is allowed to automatically learn and adjust strategies in complex environments, and can more accurately respond to dynamically changing environments, thereby improving the flexibility and accuracy of decision-making, significantly improving the system's adaptability to traffic load fluctuations, and optimizing service quality. The embodiments of the present application can more effectively utilize the storage resources of the edge server and realize the reasonable allocation and utilization of resources through intelligent agent learning and optimization of cache strategies, thereby improving the overall system performance. It can handle complex traffic loads and network conditions in industrial networks and edge computing environments, provide adaptive cache optimization solutions, and improve network efficiency and service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the edge service cache optimization method of an embodiment of the present application;

[0037] Figure 2 This is a schematic diagram of the structure of the edge service cache optimization device according to an embodiment of the present application;

[0038] Figure 3 A schematic diagram of the structure of a computer device according to an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] The terms "first", "second", "third" in this application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the embodiments of this application, all directional indications (such as up, down, left, right, front, back...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or computer device that includes a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or computer devices.

[0042] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0043] Specifically, see Figure 1 , is a flow chart of the edge service cache optimization method of the embodiment of the present application. The edge service cache optimization method of the embodiment of the present application comprises the following steps:

[0044] S10: Model the edge service cache optimization problem as a Markov decision process (MDP) and initialize the edge server;

[0045] In this step, MDP is a mathematical model and an important concept in reinforcement learning. It is used to describe the process of decision makers making decisions in an uncertain environment. The embodiment of the present application models the edge service cache optimization problem as an MDP model to describe the behavior and learning process of the agent in the environment. The MDP model includes a state space (S), an action space (A), and a reward function (R). The state space (S) includes state information such as device requests, network conditions, and cache status in the current environment. The action space (A) includes cache addition, cache update, and cache deletion. The reward function R t=-Service access delay, used to evaluate the agent's choice of strategy to reduce delay.

[0046] The initialization settings of the edge server include: initializing the cache state of the edge server, clearing the cache and setting the initial cache capacity; initializing the strategy network and value network of the intelligent agent, including network structure design and random parameter initialization; initializing the experience replay pool to empty, and setting the parameters of the target network as the initial parameters of the main network.

[0047] S20: At each time step t, the agent collects status information S such as device requests, network conditions, and cache status of the current environment. t ;

[0048] In this step, the collected device requests include request type, request frequency, and request time, among which request types include but are not limited to videos, pictures, documents, etc., which are used to analyze content demand; request frequency is the number of requests per unit time, which is used to evaluate the demand intensity of different content; request time is the timestamp of the request, which is used to help identify peak hours and timing patterns. Network conditions include the available bandwidth of the current network, the delay time of data transmission, the packet loss rate (the proportion of data packets lost during data transmission), and load conditions. The cache status includes cache content, cache hit rate, and cache usage frequency, among which cache content includes the type and size of the content, and the cache hit rate is the proportion of requested content found in the cache, which is used to evaluate the effectiveness of the cache strategy; the cache usage frequency is the access frequency of the cache content, which indicates the importance of the content and the frequency of access.

[0049] S30: Based on the state information of the current environment, the agent uses the Deep Deterministic Policy Gradient (DDPG) algorithm to optimize the cache strategy and generates the cache strategy through the policy network;

[0050] In the embodiment of the present application, the agent adopts a deep deterministic policy gradient algorithm. The agent mainly includes three parts: a policy network (Actor), a value network (Critic) and an experience replay pool. The policy network is composed of a certain number of layers of deep neural networks, and the input is the state information S of the current environment. t , according to the status information S t Generate corresponding cache operations such as adding, updating or deleting content, and output the cache strategy μ(S t ), stabilizes the training process of the policy network by delaying updates, reduces the problem of overestimation of the policy, and finally generates the target network. The value network includes a certain number of layers of deep neural networks, and the input is the state information of the current environment and the action pair (S t ,A t), the output is the Q value Q(S t ,A t ), and evaluate the long-term return of the cache operation generated by the policy network, calculate the value of the cache strategy, and delay the updated value network to stabilize the Q value training process to ensure the stability and convergence of the training. The experience replay pool is used to store historical data such as state information, actions, reward functions, and next state information that record the interaction between the agent and the environment. Training is performed by randomly extracting historical data until the optimal target network is generated to break the correlation between samples and improve model stability.

[0051] Specifically, cache operations include cache addition, cache update or cache deletion, wherein cache addition is to assign priority to the content to be cached based on the demand and value of the content, select new content to add to the cache according to selection criteria, and update the cache status. The selection criteria include but are not limited to the request frequency and importance of the content. Cache update is to update the version of the content in the cache or add new information, and replace the low-priority or outdated content already in the cache. Cache deletion is to select the content to be deleted in the cache based on the frequency of use and obsolescence of the content, remove the selected content from the cache, and update the cache status. The embodiment of the present application can also track and adjust the popularity of the cached content in real time, ensure the timely update of hot content and efficient cache management, thereby improving the effectiveness of the cache strategy and system performance.

[0052] S40: Execute cache strategy A generated by the strategy network t , and calculate the service access delay after executing the cache strategy through the value network, and calculate the reward function based on the service access delay;

[0053] In this step, the service access delay is the time delay from the request to the response. The reward function R t = - service access delay, and through the reward function R t =-Service access delay evaluates the execution effect of the cache strategy, continuously learns and optimizes the cache strategy through interaction with the environment, encourages the intelligent agent to choose a cache strategy that can reduce delay, and guides the optimization of the cache strategy.

[0054] S50: Update the state information of the current environment of the edge server to the next state information S t+1 , and store the state information, actions, reward functions and next state information of the edge server’s current environment into the experience replay pool as experience samples for training the model;

[0055] S60: Randomly extract a batch of experience samples from the experience replay pool to iteratively train the policy network and the value network until the strategy converges or reaches a predetermined number of training times, and the optimal target network is obtained;

[0056] In this step, the embodiment of the present application iteratively trains the policy network and the value network by randomly extracting experience samples from the experience replay pool, updates the parameters of the policy network and the value network to maximize the long-term return of the policy network generating the cache strategy, evaluates the value of the cache strategy, and regularly copies the parameters of the main network to the target network to stabilize the training process, improve the system's adaptability and cache strategy effect, ensure the stability of the training process and the effectiveness of the strategy. The delayed update strategy of the target network further enhances the stability and convergence speed of the training, so that the algorithm can achieve the optimization effect faster and more stably until the strategy converges or reaches a predetermined number of training times. The intelligent agent autonomously learns the edge service cache strategy by continuously interacting with the environment.

[0057] The edge service cache optimization method of the embodiment of the present application adopts an edge service cache optimization strategy based on deep reinforcement learning, and models the edge service cache problem as a Markov decision process. The intelligent agent observes the state information of the current environment at each time step, takes corresponding actions to make cache decisions and generate cache strategies, and calculates the service access delay after executing the cache strategy. The reward function is calculated according to the service access delay, and the execution effect of the cache strategy is evaluated through the reward function, and the edge service cache strategy is continuously optimized, thereby improving the cache hit rate, resource utilization efficiency and service quality. The embodiment of the present application adopts a deep reinforcement learning algorithm, which can adapt to the dynamic changes of network conditions and request patterns in real time, comprehensively considers multiple performance goals such as service access delay and cache hit rate, and realizes the overall optimization of performance goals by designing a reward function that adapts to performance goals, thereby improving the overall performance and user experience of the system. By dynamically updating the cache content and strategy, the system is allowed to automatically learn and adjust the strategy in a complex environment, and can more accurately respond to dynamically changing environments, thereby improving the flexibility and accuracy of decision-making, significantly improving the system's adaptability to traffic load fluctuations, and optimizing service quality. The embodiments of the present application can more effectively utilize the storage resources of the edge server and realize the reasonable allocation and utilization of resources through intelligent agent learning and optimization of cache strategies, thereby improving the overall system performance. It can handle complex traffic loads and network conditions in industrial networks and edge computing environments, provide adaptive cache optimization solutions, and improve network efficiency and service quality.

[0058] See also Figure 2 , is a schematic diagram of the structure of the edge service cache optimization device of the embodiment of the present application. The edge service cache optimization device 40 of the embodiment of the present application includes:

[0059] Information collection module 41: used to model the edge service cache optimization problem as an MDP model, and collect the state information of the current environment through the intelligent agent at each time step;

[0060] Strategy generation module 42: used for optimizing the cache strategy by the agent using a deep deterministic policy gradient algorithm based on the state information of the current environment, and generating a cache strategy through a policy network;

[0061] Strategy execution module 43: used for executing the cache strategy, calculating the service access delay after executing the cache strategy through the value network, and calculating the reward function according to the service access delay;

[0062] The strategy optimization module 44 is used to evaluate the execution effect of the cache strategy according to the reward function, and optimize the strategy network to generate an optimal target network.

[0063] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0064] The device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0065] See also Figure 3 , is a schematic diagram of the computer device structure of an embodiment of the present application. The computer device 50 includes:

[0066] A memory 51 storing executable program instructions;

[0067] A processor 52 connected to the memory 51;

[0068] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: modeling the edge service cache optimization problem as an MDP model, and collecting the state information of the current environment through the agent at each time step; based on the state information of the current environment, the agent uses a deep deterministic policy gradient algorithm to optimize the cache strategy, and generates a cache strategy through a policy network; executes the cache strategy, and calculates the service access delay after executing the cache strategy through a value network, and calculates a reward function based on the service access delay; evaluates the execution effect of the cache strategy based on the reward function, and optimizes the policy network to generate an optimal target network.

[0069] The processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0070] See also Figure 4 , is a schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores program instructions 61 that can implement the following steps: modeling the edge service cache optimization problem as an MDP model, and collecting the state information of the current environment through the agent at each time step; based on the state information of the current environment, the agent uses a deep deterministic policy gradient algorithm to optimize the cache strategy, and generates a cache strategy through a policy network; execute the cache strategy, and calculate the service access delay after executing the cache strategy through the value network, and calculate the reward function according to the service access delay; evaluate the execution effect of the cache strategy according to the reward function, and optimize the policy network to generate the optimal target network. Among them, the program instruction 61 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network computer device, etc.) or a processor (processor) to execute all or part of the steps of the various implementation methods of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program instructions, or terminal computer devices such as computers, servers, mobile phones, tablets, etc. Among them, the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0071] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0072] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the description and drawings of this application, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A method for optimizing edge service cache, It is characterized in that include: The edge service cache optimization problem is modeled as an MDP model, and at each time step, the state information of the current environment is collected through the agent; Based on the state information of the current environment, the intelligent agent optimizes the cache strategy using a deep deterministic policy gradient algorithm and generates a cache strategy through a policy network; Executing the cache strategy, and calculating the service access delay after executing the cache strategy through the value network, and calculating the reward function according to the service access delay; The execution effect of the cache strategy is evaluated according to the reward function, and the strategy network is optimized to generate an optimal target network.

2. The edge service cache optimization method according to claim 1, It is characterized in that The edge service cache optimization problem is modeled as an MDP model, specifically: The MDP model includes a state space, an action space and a reward function. The state space includes the state information of the current environment, and the state information includes device requests, network conditions and cache status; the action space includes cache addition, cache update and cache deletion, and the reward function R t =-Service access delay, used to evaluate the agent's choice of strategy to reduce delay.

3. The edge service cache optimization method according to claim 2, It is characterized in that After the edge service cache optimization problem is modeled as an MDP model, the following further includes: Initialize the cache state of the edge server, clear the cache and set the initial cache capacity; initialize the strategy network, value network and experience replay pool of the agent, and set the parameters of the target network as the initial parameters of the main network.

4. The edge service cache optimization method according to any one of claims 1 to 3, It is characterized in that The agent includes a policy network, a value network and an experience replay pool. The policy network is composed of a certain number of layers of deep neural networks, and the input is the state information of the current environment. The corresponding cache operation is generated according to the state information, and the cache strategy is output. The training process of the policy network is stabilized by delayed updating, and the target network is finally generated. The value network includes a certain number of layers of deep neural networks, and the input is the state information of the current environment and the action pair. The output is the Q value of the corresponding action, and the long-term return of the cache operation generated by the policy network is evaluated, and the value of the cache strategy is calculated. The delayed updated value network is used to stabilize the training process of the Q value. The experience replay pool is used to store historical data of the interaction between the agent and the environment, and model training is performed by randomly extracting the historical data until an optimal target network is generated.

5. The edge service cache optimization method according to claim 4, It is characterized in that The generating of the corresponding cache operation according to the state information is specifically: The cache operation includes cache addition, cache update or cache deletion. The cache addition is to assign priority to the content to be cached based on the demand and value of the content, select new content to add to the cache according to the selection criteria, and update the cache status; the cache update is to update the version of the content in the cache or supplement new information, and replace the low-priority or outdated content already in the cache; the cache deletion is to select the content to be deleted in the cache based on the frequency of use and obsolescence of the content, remove the selected content from the cache, and update the cache status.

6. The edge service cache optimization method according to claim 5, It is characterized in that After calculating the service access delay after executing the cache strategy through the value network and calculating the reward function according to the service access delay, the method further includes: The state information of the current environment of the edge server is updated to the next state information, and the state information, action, reward function and next state information of the current environment of the edge server are stored in the experience replay pool as experience samples for training the model.

7. The edge service cache optimization method according to claim 6, It is characterized in that The execution effect of the cache strategy is evaluated according to the reward function, and the strategy network is optimized to generate the optimal target network, specifically: A batch of experience samples are randomly selected from the experience replay pool to iteratively train the policy network and the value network until the strategy converges or a predetermined number of training times is reached, thereby obtaining an optimal target network.

8. An edge service cache optimization device, It is characterized in that include: Information collection module: used to model the edge service cache optimization problem as an MDP model and collect the state information of the current environment through the agent at each time step; Strategy generation module: used for optimizing the cache strategy by the agent using a deep deterministic policy gradient algorithm based on the state information of the current environment, and generating a cache strategy through a policy network; Strategy execution module: used to execute the cache strategy, calculate the service access delay after executing the cache strategy through the value network, and calculate the reward function according to the service access delay; Strategy optimization module: used to evaluate the execution effect of the cache strategy according to the reward function, and optimize the strategy network to generate an optimal target network.

9. A computer device, It is characterized in that The computer device includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the edge service cache optimization method according to any one of claims 1 to 7; The processor is used to execute the program instructions stored in the memory to control the edge service cache optimization method.

10. A storage medium, It is characterized in that Program instructions executable by a processor are stored, and the program instructions are used to execute the edge service cache optimization method according to any one of claims 1 to 7.

Images