Cache container layout methods, devices, media, and equipment for microservice architecture

Abstract

This invention provides a method, apparatus, medium, and device for deploying cache containers for microservice architectures. The method includes: determining the data dependencies of various containers in a container cluster corresponding to the microservice architecture, and dividing the containers into multiple candidate container pairs based on the data dependencies; wherein each candidate container pair includes an upstream container and a downstream container; selecting at least one target container pair from the multiple candidate container pairs according to preset cache container insertion conditions; periodically determining the cache capacity of the cache containers to be inserted in each target container pair using a deep reinforcement learning algorithm; and deploying and configuring the cache containers and cache capacity in the container cluster based on each target container pair and the cache capacity of the cache containers to be inserted in each target container pair. This invention can achieve maximum performance improvement with relatively low caching costs.

Description

Technical Field

[0001] This invention relates to the field of cache container technology, and in particular to a cache container layout method, apparatus, medium, and device for microservice architecture. Background Technology

[0002] In today's era of economic globalization and the information age, people's demands for information services have become increasingly diversified and personalized. To meet these growing user needs, the types of services and the scale of internet applications are constantly expanding, leading to a significant increase in the complexity of application internal design. The traditional monolithic application architecture, which packages all functional modules and components into a single deployment package, is becoming increasingly unsuitable as application functionality grows. A large codebase not only increases the difficulty of code management and updates but also significantly increases maintenance costs and severely impacts development efficiency. Furthermore, the tight coupling of monolithic architecture reduces system fault tolerance; a failure in one module can lead to the collapse of the entire system, affecting user experience and business continuity. Therefore, monolithic application architecture is no longer sufficient to meet the needs of modern enterprises for application systems that offer high reliability, flexibility, scalability, and ease of management and updates.

[0003] To address the aforementioned problems of monolithic application architectures, service-oriented architecture (SOA) has gradually emerged. This architecture aims to ensure the complete functionality of the entire application by breaking down an application into multiple independent services, each undertaking specific business functions and running independently. These services can be independently developed, deployed, managed, and scaled. Therefore, through standardized and modular service design, SOA significantly improves system flexibility and maintainability, while achieving higher service reusability and better system integration. Subsequently, the maturity of cloud computing and virtualization technologies further spurred the development of Docker container technology. This technology uses lightweight virtualization, sharing the host operating system kernel, enabling programs to run quickly and stably in independent environments with lower resource overhead. Based on SOA, developers created a Docker container for each independent service in large applications, allowing each service to run in its dependent environment without interfering with each other. This approach not only ensures loose coupling between services but also allows each service to be developed, tested, and deployed independently. By combining these Docker containers, developers can build application systems in a modular way, forming a flexible, scalable, and easily managed and maintained application architecture—microservice architecture.

[0004] While microservice architecture largely addresses the pain points of monolithic application architectures, it also introduces new challenges. According to relevant literature, modern application systems typically consist of thousands of microservices deployed and running on different servers, communicating remotely over the internet via RPC protocols. However, due to limited network bandwidth, frequent calls between microservices generate significant network overhead when users utilize a particular application function. Literature indicates that 70% of user request response time in microservice architectures is caused by inter-container network communication. This issue makes microservice architectures prone to performance bottlenecks during network congestion, leading to a decline in application service quality and impacting user experience. Furthermore, network congestion is often accompanied by a massive influx of user requests to the database. In such high-load, high-concurrency scenarios, the frequency of calls between microservices increases dramatically, especially at the last stage of the call chain—the microservice container that ultimately accesses the database—where performance bottlenecks become more pronounced. Therefore, the response performance of the database system becomes a critical factor affecting the performance of microservice architecture applications. If database access efficiency cannot be guaranteed, the overall application response time will increase significantly, ultimately negatively impacting user experience.

[0005] While database systems can improve performance through various techniques such as parallel processing, index optimization, and query optimization, these optimizations still have limitations. When faced with a large number of concurrent access requests, database performance can still be quickly exhausted. Even in high-performance database systems, these optimizations cannot completely eliminate performance bottlenecks. To address this issue, introducing a caching mechanism is crucial. By introducing a cache between the application layer and the database layer, the call chain can achieve high-speed access to frequently accessed data, thereby significantly reducing direct database access, effectively lowering database load and reducing network bandwidth pressure, ultimately improving the overall response performance of the application to user requests.

[0006] In modern microservice architectures, caching containers are typically deployed across different backend storage systems to accelerate data read and write operations. However, this also introduces several problems. First, due to uneven cache allocation, some nodes may be configured with excessive cache resources, while others may have insufficient cache. This uneven allocation leads to reduced cache utilization and further wastes system resources. Second, in a microservice architecture, completing a request requires accessing multiple containers, forming a container path with data dependencies. Adding caching to this data-dependent container path can significantly reduce request response time because once the requested data is hit in the cache, it doesn't need to access other containers on that path to obtain the required data. However, in large-scale microservice clusters, how to efficiently deploy caching containers to significantly reduce user request response time and caching costs is a pressing issue for improving microservice architecture performance. A caching container is a container that provides caching services. Summary of the Invention

[0007] To address at least one of the above technical problems, embodiments of the present invention provide a method, apparatus, medium, and device for laying out cache containers for microservice architectures.

[0008] According to the first aspect, the cache container layout method for microservice architecture provided by the embodiments of the present invention includes:

[0009] Determine the data dependencies of each container in the container cluster corresponding to the microservice architecture, and divide each container into multiple candidate container pairs according to the data dependencies; wherein each candidate container pair includes an upstream container and a downstream container;

[0010] Based on preset cache container insertion conditions, at least one target container pair that can be inserted into the cache container is selected from the plurality of candidate container pairs;

[0011] The cache capacity of the cache containers to be inserted into each target container pair is determined periodically by using a deep reinforcement learning algorithm; wherein, the deep reinforcement learning algorithm outputs a cache capacity that minimizes user request latency and total cache size by learning the relationship between cache capacity and user request latency;

[0012] Based on each target container pair and the cache capacity of the cache containers to be inserted in each target container pair, the cache containers are deployed and the cache capacity is configured in the container cluster.

[0013] In one embodiment, the cache container insertion condition includes at least one of the following:

[0014] Neither of the two containers in the target container pair is a cache container; and,

[0015] In a container dependency chain formed by the same type of request, select only one target container pair from the candidate container pairs on the sequential branch, and the target container pair is at the front of the sequential branch; or, the target container pair is on a container dependency chain that is associated with the database; or, the target container pair is at the front of the container dependency chain or at a position close to the front.

[0016] In one embodiment, the step of periodically determining the cache capacity of the cache containers to be inserted into each target container pair using a deep reinforcement learning algorithm includes:

[0017] S11. Initialize the Q value of each action in the action space to 0; wherein, the Q value represents the expected reward value of an action, and the action is used to characterize the cache capacity of the cache container inserted into each target container pair;

[0018] S12. Collect the state information of the cache containers inserted in each target container pair at preset time intervals, and form the state vector at the current moment based on the state information of the cache containers inserted in each target container pair.

[0019] S13. Generate a random number x in the range [0, 1], and compare the random number x with the probability parameter ε; if the random number x is less than or equal to the probability parameter ε, then select the action with the largest Q value from the Q values ​​corresponding to each action in the action space calculated in the previous round; if the random number x is greater than the probability parameter ε, then randomly select an action in the action space; and output the cache capacity of the cache container inserted into each target container pair according to the selected action.

[0020] S14. If the action with the largest Q value is selected, then the Q value of each action in the action space is calculated based on the state vector of the container pair in this round, and used in the next round S13.

[0021] In one embodiment, the status information of the cache container includes cache hit rate, cache access data volume, and cache capacity.

[0022] In one embodiment, the method further includes:

[0023] After completing the deployment of cache containers and configuration of cache capacity in the container cluster according to the action selected in S13 of the current round, the state vector of the next moment is determined.

[0024] Based on the state vector of the next moment, calculate the reward corresponding to the action selected in S13 of the current round, and store the state vector of the current moment, the action selected in S13 of the current round, the reward corresponding to the action, and the state vector of the next moment as a data entry in the experience replay buffer.

[0025] Determine whether the experience playback buffer is full;

[0026] If so, update the target depth Q-network when the update condition is met; calculate the loss value according to the preset loss function; determine the Q-value Q corresponding to the action selected by the target depth Q-network based on the state vector at the next time step. target And the Q-value Q corresponding to the action selected by the online deep Q-network based on the current state vector. online According to Q target and Q online Calculate the time difference error; update the online deep Q-network; wherein the target deep Q-network and the online deep Q-network are two networks in the deep reinforcement learning algorithm.

[0027] In one embodiment, calculating the reward corresponding to the action selected in S13 of the current round includes: calculating the reward using a first calculation formula, wherein the first calculation formula is:

[0028]

[0029] Where r is the reward and α is the proportionality coefficient. Cache provides the average response time for user requests. j Let j be the cache capacity of the j-th cache container, where j is the cache container number.

[0030] In one embodiment, calculating the time difference error includes calculating the time difference error using a second calculation formula, wherein the second calculation formula is:

[0031] TDError = r + γ·Q target –Q online

[0032] Where TDError is the time difference error, r is the reward obtained after the action is performed, γ is the discount factor, and Q... target Q is the Q-value corresponding to the action selected by the target depth Q-network based on the state vector at the next time step. online It is the Q-value corresponding to the action selected by the online deep Q-network based on the current state vector.

[0033] According to a second aspect, the cache container layout apparatus for microservice architecture provided in the embodiments of the present invention includes:

[0034] The first determining module is used to determine the data dependency relationship of each container in the container cluster corresponding to the microservice architecture, and divide each container into multiple candidate container pairs according to the data dependency relationship; wherein, each candidate container pair includes an upstream container and a downstream container;

[0035] The first filtering module is used to filter at least one target container pair that can be inserted into the cache container from the plurality of candidate container pairs according to the preset cache container insertion conditions.

[0036] The capacity determination module is used to periodically determine the cache capacity of the cache containers that need to be inserted into each target container pair through a deep reinforcement learning algorithm; wherein, the deep reinforcement learning algorithm outputs a cache capacity that minimizes the user request latency and the total cache volume by learning the relationship between the cache capacity and the user request latency;

[0037] The container layout module is used to deploy cache containers and configure cache capacity in the container cluster according to each target container pair and the cache capacity of the cache containers to be inserted in each target container pair.

[0038] According to a third aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first aspect.

[0039] According to a fourth aspect, the computing device provided in the embodiments of the present invention includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method provided in the first aspect.

[0040] This invention provides a method, apparatus, medium, and device for cache container layout in a microservice architecture. The method determines the data dependencies between containers in a container cluster corresponding to the microservice architecture, divides each container into multiple candidate container pairs based on these dependencies, and then selects at least one target container pair from these candidate pairs to insert a cache container. Next, a deep reinforcement learning algorithm periodically determines the cache capacity of the cache container to be inserted into each target container pair. Finally, based on each target container pair and the cache capacity of the cache container to be inserted into each target container pair, the cache containers are deployed and their cache capacity is configured in the container cluster, thereby achieving cache container layout. As can be seen, in this invention, firstly, by analyzing the dependencies between all containers in the microservice architecture, candidate container pairs with dependencies are identified; then, some target container pairs are selected from the candidate container pairs to insert cache containers; next, the deep reinforcement learning algorithm can mine the relationship between cache capacity and user request latency based on the cache container's state information, and then output the cache capacity allocated to the cache container, thereby performing cache container layout to minimize request latency with the lowest caching cost, i.e., to achieve the maximum performance improvement with a lower caching cost. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating a cache container layout method for a microservice architecture according to an embodiment of the present invention.

[0042] Figure 2 This is an example of a microservice architecture in one embodiment of the present invention;

[0043] Figure 3 This is a structural block diagram of a cache container layout device for a microservice architecture according to an embodiment of the present invention. Detailed Implementation

[0044] Firstly, embodiments of the present invention provide a cache container layout method for microservice architectures, see [link to relevant documentation]. Figure 1 The method includes the following steps S110 to S140:

[0045] S110. Determine the data dependencies of each container in the container cluster corresponding to the microservice architecture, and divide each container into multiple candidate container pairs according to the data dependencies; wherein, each candidate container pair includes an upstream container and a downstream container;

[0046] For example, in a microservice architecture, there are three dependency chains corresponding to three types of requests. Dependency chain 1 and dependency chain 3 need to read data from database A and database B, respectively, while dependency chain 2 needs to write data to database A. Candidate container pairs with dependencies are: container A and B, container B and C, container B and F, container B and H, container C and D, container D and E, and container F and G. These candidate container pairs have a dependency relationship, meaning they include an upstream container and a downstream container. The upstream container's request processing depends on the data returned by the downstream container; that is, the upstream container initiates a data request to the downstream container, and the downstream container returns the data. All of these container pairs are candidate container pairs.

[0047] S120. Based on the preset cache container insertion conditions, at least one target container pair that can be inserted into the cache container is selected from the plurality of candidate container pairs.

[0048] The cache container insertion conditions may include at least one of the following:

[0049] Neither of the two containers in the target container pair is a cache container; and,

[0050] In a container dependency chain formed by the same type of request, select only one target container pair from the candidate container pairs on the sequential branch, and the target container pair is at the front of the sequential branch; or, the target container pair is on a container dependency chain that is associated with the database; or, the target container pair is at the front of the container dependency chain or at a position close to the front.

[0051] It is understood that, in this embodiment of the invention, containers in a microservice architecture are divided into the following three categories:

[0052] (1) Front-end container: The front-end container includes containers that directly interact with users, receive and distribute user requests from clients, and present the user interface. Some common front-end containers include web application containers such as nginx, API containers, and content delivery network containers, etc. The upstream container of a container pair in the front end must be a front-end container, and the downstream container is either a front-end container or a container located at the junction of the "front-end-middle layer".

[0053] (2) Intermediate layer containers: Intermediate layer containers include containers that handle core business logic, request distribution, and inter-service coordination. The upstream and downstream containers of a container pair in the intermediate layer are all intermediate layer containers.

[0054] (3) Backend containers. Backend containers include cache or database type containers that are responsible for storing, retrieving, caching and managing data. Common backend containers include cache containers and database containers.

[0055] As can be seen, the above conditions for inserting a cache container can be summarized as follows: neither the upstream nor the downstream container is a cache container and one of the following conditions is met: it is in the front end; it is in the back end; it is in the middle layer and is at the beginning of the sequential branch in the container dependency chain formed by the same request.

[0056] As can be seen, the insertion range of cache containers can be narrowed based on the above-mentioned cache container insertion conditions.

[0057] S130. Periodically determine the cache capacity of the cache containers to be inserted into each target container pair using a deep reinforcement learning algorithm; wherein, the deep reinforcement learning algorithm outputs a cache capacity that minimizes user request latency and total cache size by learning the relationship between cache capacity and user request latency.

[0058] Understandably, if the cache capacity of the cache container to be inserted into a target container pair determined by a deep reinforcement learning algorithm is 0, it means that no cache container will be inserted into this target container pair. If it is not 0, then a cache container needs to be inserted, and the cache size of the cache container needs to be configured according to the determined cache capacity.

[0059] Deep reinforcement learning is a machine learning method that learns optimal strategies through interaction with the environment. It learns from feedback in the environment through trial and error, gradually adjusting its strategy to maximize long-term returns. In the cache layout problem, deep reinforcement learning algorithms can learn from historical data on cache resource usage to continuously and adaptively adjust and optimize cache capacity allocation schemes to cope with dynamic loads, optimize system performance, and reduce cache overhead.

[0060] The step S130, which involves periodically determining the cache capacity of the cache containers to be inserted into each target container pair using a deep reinforcement learning algorithm, may include steps S11 to S14:

[0061] S11. Initialize the Q value of each action in the action space to 0; wherein, the Q value represents the expected reward value of an action, and the action is used to characterize the cache capacity of the cache container inserted into each target container pair;

[0062] The action space includes multiple actions, each representing the optional container capacity of the cache container inserted into each target container pair.

[0063] S12. Collect the state information of the cache containers inserted in each target container pair at preset time intervals, and form the state vector at the current moment based on the state information of the cache containers inserted in each target container pair.

[0064] The status information of the cache container may include cache hit rate, cache access data volume, and cache capacity. That is, the cache hit rate, cache access data volume, and cache capacity of the cache containers inserted into each target container are collected at regular intervals.

[0065] The state vector formed at the current moment can be denoted as s. t .

[0066] S13. Generate a random number x in the range [0, 1], and compare the random number x with the probability parameter ε; if the random number x is less than or equal to the probability parameter ε, then select the action with the largest Q value from the Q values ​​corresponding to each action in the action space calculated in the previous round; if the random number x is greater than the probability parameter ε, then randomly select an action in the action space; and output the cache capacity of the cache container inserted into each target container pair according to the selected action.

[0067] As can be seen, a random number x is generated in the range [0,1]. x is compared with the probability parameter ε, 0≤ε≤1. If x is in the range [0,ε], the action with the largest Q value returned in the previous round can be adopted; otherwise, an action is randomly selected from the action space.

[0068] The deep reinforcement learning algorithm performs a judgment for each cache container to determine how much available space to allocate to it. To limit caching costs, the total cache size of the cache containers will not exceed a maximum value, `cachemax`. The cache allocation space for each cache container is between [0, `cachemax`], where `cachemax` is the maximum total cache capacity. The action vector is defined as an n*1 vector, where n is the total number of cache containers. The action vector satisfies the following conditions: the sum of all scalar values ​​does not exceed `cachemax`, i.e., the maximum available cache capacity; for the intermediate layer, only one pair of target containers is selected and inserted into the cache container on the sequential branch of the container dependency chain formed by each request.

[0069] As can be seen, the embodiments of the present invention adopt an ε-greedy exploration strategy, that is, whenever the deep reinforcement learning algorithm needs to select an action, there is an ε probability of randomly taking an action in the action space as the action decision; otherwise, the optimal action calculated by the deep reinforcement learning algorithm is selected as the action decision.

[0070] S14. If the action with the largest Q value is selected, then the Q value of each action in the action space is calculated based on the state vector of the container pair in this round, and used in the next round S13.

[0071] As can be seen, when selecting the action with the largest Q-value, the online deep Q-network will select the action based on the current input state vector s. t Calculate the Q-value for each action in the action space.

[0072] Taking a certain action 'a' as an example, the online deep Q-network first observes and collects the expected state vector 's' at the next moment after the action is performed. t+1 The reward for action a is calculated based on the estimated fitting function of the neural network, i.e., the reward r corresponding to action a. a The expected state vector s at the next time step. t+1 The input is fed into the target depth Q-network, which then adjusts the input based on the target network parameters θ. - Calculate the expected state vector s at the next time step. t+1 'Calculate the Q-value of each action in the corresponding action space, and select the maximum value from them, denoted as max.' a’ Q(s t ',a',θ - ), where a' is the action corresponding to the maximum Q value. At this point, based on the learning rate w and the discount factor γ, the calculated value for the current state s is updated. t The Q-value corresponding to the next action 'a', i.e., the Q-function: Q(s) t ,a)=w[r a +γmax a’ Q(s t',a',θ - )).

[0073] Specifically, the Q-function is used to estimate the expected reward for each action in the current action space, and the decision action is output in conjunction with the ε-greedy exploration strategy. The Q-function is a reward expectation calculation function for an action, which includes the current reward and all possible future rewards.

[0074] In one embodiment, the method provided by this invention may further include A1 to A4:

[0075] A1. After completing the deployment of cache containers and configuration of cache capacity in the container cluster according to the action selected in S13 of the current round, determine the state vector of the next moment of the current moment.

[0076] Understandably, after the cache container layout is completed according to the action selected in S13 of a round, the cache container will perform caching operations when the dependency chain it belongs to processes user requests. Then, in the next collection cycle, the state vector of each cache container is collected, that is, the state vector of the next moment.

[0077] A2. Based on the state vector of the next moment, calculate the reward corresponding to the action selected in S13 of the current round, and store the state vector of the current moment, the action selected in S13 of the current round, the reward corresponding to the action, and the state vector of the next moment as a data entry in the experience replay buffer.

[0078] In one embodiment, calculating the reward corresponding to the action selected in S13 of the current round may include: calculating the reward using a first formula, wherein the first formula is:

[0079]

[0080] Where r is the reward and α is the proportionality coefficient. Cache provides the average response time for user requests. j Let j be the cache capacity of the j-th cache container, where j is the cache container number.

[0081] Next, after calculating the reward, the current state vector s is... t The action selected in S13 of this round is based on the action selected in the round at the current time t, the reward r corresponding to that action, and the state vector s of the next time step. t+1 This data is stored as a single record in the experience replay buffer.

[0082] A3. Determine whether the experience playback buffer is full;

[0083] A4. If so, update the target depth Q-network when the update condition is met; calculate the loss value according to the preset loss function; determine the Q-value Q corresponding to the action selected by the target depth Q-network based on the state vector at the next moment. target And the Q-value Q corresponding to the action selected by the online deep Q-network based on the current state vector. online According to Q target and Q online Calculate the time difference error; update the online deep Q-network; wherein the target deep Q-network and the online deep Q-network are two networks in the deep reinforcement learning algorithm.

[0084] The experience replay buffer has a storage limit; when the number of data entries reaches this limit, the buffer is full. If full, the deep reinforcement learning algorithm executes the following instructions: First, it checks if the update condition is met. If so, it updates the "target deep Q-network," specifically copying the network parameter values ​​from the "main neural network" to the network parameter ω in the "target neural network." Then, it calculates the loss value (loss) according to the set loss function and calculates the Q-value (Q) of the online deep Q-network. online And the Q-value of the target depth Q-network. target Finally, the time difference error TDError is calculated, and the network parameters of the online deep Q network are updated.

[0085] The loss function used is the binary cross-entropy loss function BCELoss, whose formula is shown below:

[0086]

[0087] Where N is the number of samples; y i It is the true label of the i-th sample, and its value is 0 or 1; is the predicted probability of the i-th sample by the deep reinforcement learning algorithm, representing the probability of belonging to category 1, and its value ranges from 0 to 1.

[0088] The calculation of the time difference error may include: calculating the time difference error using a second calculation formula, wherein the second calculation formula is:

[0089] TDError = r + γ·Q target –Q online

[0090] Where TDError is the time difference error, r is the reward obtained after the action is performed, γ is the discount factor, and Q... target Q is the Q-value corresponding to the action selected by the target depth Q-network based on the state vector at the next time step. onlineIt is the Q-value corresponding to the action selected by the online deep Q-network based on the current state vector.

[0091] As can be seen from A1 to A4, after each execution of the cache container layout based on the action selected in S13 of a round, the loss function and time difference error are calculated, and the target deep Q-network and online deep Q-network in the deep reinforcement learning algorithm are updated. The calculation of the loss function and time difference error can be used to further optimize the deep reinforcement learning algorithm, and the updating of the target deep Q-network and online deep Q-network can further optimize the two main networks of the deep reinforcement learning algorithm. In this way, when S120 is executed next, a better action vector can be given based on the deep reinforcement learning algorithm.

[0092] As can be seen, the deep reinforcement learning algorithm incorporates an experience replay mechanism. The experience replay buffer stores the current state vector, the selected action, the reward corresponding to the selected action, and the state vector for the next time step during each round of cache allocation. This data is stored as experience data. Random samples can then be taken from the stored data to train the network in the deep reinforcement learning algorithm, thereby improving the algorithm's stability and performance. In this way, the selection effect gradually reaches the learning objective as the number of rounds increases. The learning objective is to minimize the average response time of user I / O requests and the total cache cost.

[0093] That is, the deep reinforcement learning algorithm uses a reward mechanism to provide feedback on the cache space allocation effect of each cache container, thereby optimizing the algorithm and allowing for the selection of better actions in the future.

[0094] S140. Based on each target container pair and the cache capacity of the cache containers to be inserted in each target container pair, deploy the cache containers and configure the cache capacity in the container cluster.

[0095] As can be seen, the cache capacity of the cache containers to be inserted into each target container pair can be determined based on the actions output by the deep reinforcement learning algorithm. The output actions are actually action vectors. Taking a container cluster containing n target container pairs as an example, the action vector output by the deep reinforcement learning algorithm is a one-dimensional vector composed of n scalars, i.e., [cache1, cache2, ..., cache...]. n And satisfy the following two conditions: (1) 0 ≤ cache i ≤cache max (1≤i≤n); (2)∑ i cache i ≤cache max (1≤i≤n), where cache max This is the maximum cache capacity threshold.

[0096] For each target container pair, the capacity setting of the insertion cache container is one of the following sets: Set = {0, cache} max / maxnum*1,cache max / maxnum*2,…,cache max / maxnum*(maxnum-1),cache max}. Here, `maxnum` represents the maximum number of cache containers that can be inserted simultaneously in a microservice architecture, i.e., the maximum number of non-zero scalars that can exist in the action vector. This value is limited by the aforementioned cache container insertion conditions. The above set will be in the interval [0, cache...]. max Cache by step size max The points are divided into equal parts (e.g., / maxnum), and the values ​​corresponding to the resulting set of points constitute the selectable range of capacity settings.

[0097] For example, see Figure 2 The service architecture shown contains three pairs of target containers: container pair A and B, container pair B and C, and container pair C and D. Assume the maximum available cache capacity is... max If the value is 6MB, then the interval [0,6] is divided according to a step size of 6 / 3 = 2, thus obtaining the set of possible values ​​for the capacity of each target container to the insertable container {0,2,4,6}. Therefore, the action space is defined as the set consisting of the following action vectors: {[0,0,0],[2,0,0],[0,2,0],[0,0,2],[4,0,0],[0,4,0],[0,0,4],[6,0,0],[0,6,0],[0,0,6],[2,2,0],[2,2,0],[0,2,2],[2,2,2]}.

[0098] The deep reinforcement learning algorithm can employ a typical Double DQN architecture, which consists of two deep Q-networks. One Q-network acts as the online network, and the other acts as the target network. Each deep Q-network can contain several layers of basic network structure, in the following order: input layer, one-dimensional convolutional neural network layer, several fully connected layers, and output layer.

[0099] In this embodiment of the invention, the state vectors of the cache containers are collected periodically and input into a deep reinforcement learning algorithm to obtain action vectors, thereby determining the cache container capacity allocation strategy for this round. Next, the cache container capacity allocation strategy is executed, i.e., inserting cache containers into the target container pair and setting their cache capacity to achieve the cache container layout. Finally, after the current round of actions is completed, the reward is calculated, and the current state vector, the action selected in S13 of this round, the reward corresponding to that action, and the state vector for the next time step are stored as a quadruple in the experience replay buffer, enabling related calculations and update operations.

[0100] In this embodiment of the invention, firstly, by analyzing the dependencies between all containers in the microservice architecture, candidate container pairs with dependencies are identified; then, some target container pairs are selected from the candidate container pairs to be inserted into the cache container; next, the deep reinforcement learning algorithm mines the relationship between cache capacity and user request latency based on the state information of the cache container, and then outputs the cache capacity allocated to the cache container, and then performs the layout of the cache container to minimize request latency with the lowest caching cost, that is, to obtain the maximum performance improvement with a lower caching cost.

[0101] Secondly, embodiments of the present invention provide a cache container layout apparatus for microservice architectures, see [link to relevant documentation]. Figure 3 The device 100 includes:

[0102] The first determining module 110 is used to determine the data dependency relationship of each container in the container cluster corresponding to the microservice architecture, and divide each container into multiple candidate container pairs according to the data dependency relationship; wherein, each candidate container pair includes an upstream container and a downstream container.

[0103] The first filtering module 120 is used to filter at least one target container pair that can be inserted into the cache container from the plurality of candidate container pairs according to the preset cache container insertion conditions.

[0104] The capacity determination module 130 is used to periodically determine the cache capacity of the cache containers that need to be inserted into each target container pair through a deep reinforcement learning algorithm; wherein, the deep reinforcement learning algorithm outputs a cache capacity that minimizes the user request latency and the total cache volume by learning the relationship between the cache capacity and the user request latency.

[0105] The container layout module 140 is used to deploy cache containers and configure cache capacity in the container cluster according to each target container pair and the cache capacity of the cache containers to be inserted in each target container pair.

[0106] In one embodiment, the cache container insertion condition includes at least one of the following:

[0107] Neither of the two containers in the target container pair is a cache container; and,

[0108] In a container dependency chain formed by the same type of request, select only one target container pair from the candidate container pairs on the sequential branch, and the target container pair is at the front of the sequential branch; or, the target container pair is on a container dependency chain that is associated with the database; or, the target container pair is at the front of the container dependency chain or at a position close to the front.

[0109] In one embodiment, the capacity determination module 130 is specifically used to perform the following steps:

[0110] S11. Initialize the Q value of each action in the action space to 0; wherein, the Q value represents the expected reward value of an action, and the action is used to characterize the cache capacity of the cache container inserted into each target container pair;

[0111] S12. Collect the state information of the cache containers inserted in each target container pair at preset time intervals, and form the state vector at the current moment based on the state information of the cache containers inserted in each target container pair.

[0112] S13. Generate a random number x in the range [0, 1], and compare the random number x with the probability parameter ε; if the random number x is less than or equal to the probability parameter ε, then select the action with the largest Q value from the Q values ​​corresponding to each action in the action space calculated in the previous round; if the random number x is greater than the probability parameter ε, then randomly select an action in the action space; and output the cache capacity of the cache container inserted into each target container pair according to the selected action.

[0113] S14. If the action with the largest Q value is selected, then the Q value of each action in the action space is calculated based on the state vector of the container pair in this round, and used in the next round S13.

[0114] In one embodiment, the status information of the cache container includes cache hit rate, cache access data volume, and cache capacity.

[0115] In one embodiment, the apparatus further includes:

[0116] The optimization and update module is used to determine the state vector of the next moment after the deployment of cache containers and configuration of cache capacity in the container cluster are completed according to the action selected in S13 of the current round; calculate the reward corresponding to the action selected in S13 of the current round according to the state vector of the next moment; store the state vector of the current moment, the action selected in S13 of the current round, the reward corresponding to the action, and the state vector of the next moment as a data entry in the experience replay buffer; determine whether the experience replay buffer is full; if so, update the target depth Q network when the update condition is met; calculate the loss value according to the preset loss function; and determine the Q value Q corresponding to the action selected by the target depth Q network according to the state vector of the next moment. target And the Q-value Q corresponding to the action selected by the online deep Q-network based on the current state vector. online According to Q target and Q online Calculate the time difference error; update the online deep Q-network; wherein the target deep Q-network and the online deep Q-network are two networks in the deep reinforcement learning algorithm.

[0117] In one embodiment, the optimization and update module is specifically used to: calculate the reward using a first calculation formula, wherein the first calculation formula is:

[0118]

[0119] Where r is the reward and α is the proportionality coefficient. Cache provides the average response time for user requests. j Let j be the cache capacity of the j-th cache container, where j is the cache container number.

[0120] In one embodiment, the optimization and update module is specifically used to: calculate the time difference error using a second calculation formula, wherein the second calculation formula is:

[0121] TDError = r + γ·Q target –Q online

[0122] Where TDError is the time difference error, r is the reward obtained after the action is performed, γ is the discount factor, and Q... target Q is the Q-value corresponding to the action selected by the target depth Q-network based on the state vector at the next time step. online It is the Q-value corresponding to the action selected by the online deep Q-network based on the current state vector.

[0123] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the contents of the apparatus provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0124] Thirdly, embodiments of the present invention provide a computer-readable medium storing computer instructions, which, when executed by a processor, cause the processor to perform the method provided in the first aspect.

[0125] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0126] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0127] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0128] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0129] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0130] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the contents in the computer-readable medium provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0131] Fourthly, one embodiment of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method of any embodiment of the specification.

[0132] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the computing device provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0133] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0134] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, widgets, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0135] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cache container layout method for microservice architecture, characterized in that, include: Determine the data dependencies of each container in the container cluster corresponding to the microservice architecture, and divide each container into multiple candidate container pairs according to the data dependencies; wherein each candidate container pair includes an upstream container and a downstream container; Based on preset cache container insertion conditions, at least one target container pair that can be inserted into the cache container is selected from the plurality of candidate container pairs; The cache capacity of the cache containers to be inserted into each target container pair is determined periodically by using a deep reinforcement learning algorithm; wherein, the deep reinforcement learning algorithm outputs a cache capacity that minimizes user request latency and total cache size by learning the relationship between cache capacity and user request latency; Based on each target container pair and the cache capacity of the cache containers to be inserted in each target container pair, the cache containers are deployed and the cache capacity is configured in the container cluster.

2. The method according to claim 1, characterized in that, The cache container insertion conditions include at least one of the following: Neither of the two containers in the target container pair is a cache container; and, In a container dependency chain formed by the same type of request, select only one target container pair from the candidate container pairs on the sequential branch, and the target container pair is at the front of the sequential branch; or; the target container pair is on a container dependency chain that is associated with the database; or; the target container pair is at the front of the container dependency chain or at a position close to the front.

3. The method according to claim 1, characterized in that, The process of periodically determining the cache capacity of the cache containers to be inserted into each target container pair using a deep reinforcement learning algorithm includes: S11. Initialize the Q value of each action in the action space to 0; wherein, the Q value represents the expected reward value of an action, and the action is used to characterize the cache capacity of the cache container inserted into each target container pair; S12. Collect the state information of the cache containers inserted in each target container pair at preset time intervals, and form the state vector at the current moment based on the state information of the cache containers inserted in each target container pair. S13. Generate a random number x in the range [0, 1], and compare the random number x with the probability parameter ε; if the random number x is less than or equal to the probability parameter ε, then select the action with the largest Q value from the Q values ​​corresponding to each action in the action space calculated in the previous round; if the random number x is greater than the probability parameter ε, then randomly select an action in the action space; and output the cache capacity of the cache container inserted into each target container pair according to the selected action. S14. If the action with the largest Q value is selected, then the Q value of each action in the action space is calculated based on the state vector of the container pair in this round, and used in the next round S13.

4. The method according to claim 3, characterized in that, The status information of the cache container includes cache hit rate, cache access data volume, and cache capacity.

5. The method according to claim 3, characterized in that, Also includes: After completing the deployment of cache containers and configuration of cache capacity in the container cluster according to the action selected in S13 of the current round, the state vector of the next moment is determined. Based on the state vector of the next moment, calculate the reward corresponding to the action selected in S13 of the current round, and store the state vector of the current moment, the action selected in S13 of the current round, the reward corresponding to the action, and the state vector of the next moment as a data entry in the experience replay buffer. Determine whether the experience playback buffer is full; If so, update the target depth Q-network when the update condition is met; calculate the loss value according to the preset loss function; determine the Q-value Q corresponding to the action selected by the target depth Q-network based on the state vector at the next time step. target And the Q-value Q corresponding to the action selected by the online deep Q-network based on the current state vector. online According to Q target and Q online Calculate the time difference error; update the online deep Q-network; wherein the target deep Q-network and the online deep Q-network are two networks in the deep reinforcement learning algorithm.

6. The method according to claim 5, characterized in that, The calculation of the reward corresponding to the action selected in S13 of the current round includes: calculating the reward using a first calculation formula, wherein the first calculation formula is: Where r is the reward and α is the proportionality coefficient. Cache provides the average response time for user requests. j Let j be the cache capacity of the j-th cache container, where j is the cache container number.

7. The method according to claim 5, characterized in that, The calculation of the time difference error includes calculating the time difference error using a second calculation formula, wherein the second calculation formula is: TDError=r+γ·Q target –Q online Where TDError is the time difference error, r is the reward obtained after the action is performed, γ is the discount factor, and Q... target Q is the Q-value corresponding to the action selected by the target depth Q-network based on the state vector at the next time step. online It is the Q-value corresponding to the action selected by the online deep Q-network based on the current state vector.

8. A cache container layout device for microservice architecture, characterized in that, include: The first determining module is used to determine the data dependency relationship of each container in the container cluster corresponding to the microservice architecture, and divide each container into multiple candidate container pairs according to the data dependency relationship; wherein, each candidate container pair includes an upstream container and a downstream container; The first filtering module is used to filter at least one target container pair that can be inserted into the cache container from the plurality of candidate container pairs according to the preset cache container insertion conditions. The capacity determination module is used to periodically determine the cache capacity of the cache containers that need to be inserted into each target container pair through a deep reinforcement learning algorithm; wherein, the deep reinforcement learning algorithm outputs a cache capacity that minimizes the user request latency and the total cache volume by learning the relationship between the cache capacity and the user request latency; The container layout module is used to deploy cache containers and configure cache capacity in the container cluster according to each target container pair and the cache capacity of the cache containers to be inserted in each target container pair.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1 to 7.

10. A computing device, characterized in that, The method includes a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in any one of claims 1 to 7.

Images