Micro-service architecture-oriented cache container layout method and device, medium and equipment

Abstract

The invention provides a micro-service architecture-oriented cache container layout method and device, a medium and equipment. The method comprises the following steps: determining a data dependency relationship of each container in a container cluster corresponding to the micro-service architecture, and dividing each container into a plurality of candidate container pairs according to the data dependency relationship; wherein each candidate container pair comprises an upstream container and a downstream container; screening out at least one target container pair capable of being inserted into a cache container from the plurality of candidate container pairs according to a preset cache container insertion condition; periodically determining the cache capacity of a cache container needing to be inserted in each target container pair through a deep reinforcement learning algorithm; and according to each target container pair and the cache capacity of the cache container needing to be inserted in each target container pair, deploying the cache container and configuring the cache capacity in the container cluster. According to the method and the device, the maximum performance improvement can be obtained with relatively low cache cost.

Description

Technical Field

[0001] The present invention relates to the field of cache container technology, and in particular to a cache container layout method, device, medium, and equipment for a microservice architecture. Background Art

[0002] In today's globalized economy and information age, people's demand for information services has become increasingly diversified and personalized. In order to meet the growing needs of users, the types of services and software scale of Internet applications have also continued to expand. This change has led to a significant increase in the complexity of internal application design. The traditional monolithic application architecture packages all functional modules and components in a single deployment package. This method is becoming less and less suitable as application functions gradually increase. The huge code base not only increases the difficulty of code management and updating, but also significantly increases maintenance costs and has a serious impact on development efficiency. In addition, the tightly coupled characteristics of the monolithic architecture reduce the fault tolerance of the system. Once a module fails, it may cause the entire system to crash, affecting user experience and business continuity. It can be seen that the monolithic application architecture can no longer meet the needs of modern enterprises for high reliability, flexibility, scalability, and easy-to-manage and update application systems.

[0003] In order to deal with the above-mentioned problems of the monolithic application architecture, the service-oriented architecture has gradually been proposed. This architecture aims to split the application into multiple independent services, allowing each service to assume specific business functions and run independently to ensure the complete functionality of the entire application, so that these services can be independently developed, deployed, managed and expanded. Therefore, through the standardization and modular design of services, the service-oriented architecture significantly improves the flexibility and maintainability of the system, while achieving a higher service reuse rate and better system integration. After that, the maturity of cloud computing and virtualization technology further gave rise to Docker container technology, which uses lightweight virtualization to enable programs to run quickly and stably in an independent environment with low resource overhead by sharing the kernel of the host operating system. Therefore, based on the service-oriented architecture, developers created a Docker container for each independent service in a large application, so that each service can run in its dependent environment without interfering with each other. This approach not only ensures the loose coupling characteristics between services, but also enables each service to be independently developed, tested and deployed. By combining these Docker containers together, developers can build application systems in a modular way, thus forming a flexible, scalable and easy-to-manage and maintain application architecture, namely the microservice architecture.

[0004] Although the microservice architecture has largely solved the pain points of the monolithic application architecture, it has also brought some new challenges. According to relevant literature, modern application systems are usually composed of thousands of microservices, which are usually deployed and run on different servers and communicate remotely through the RPC protocol on the Internet. However, due to limited network bandwidth resources, when users use a certain function of the application, frequent calls between microservices will generate large network overhead. According to relevant literature, in microservice architecture applications, 70% of user request response time is caused by inter-container network communication. This problem makes the microservice architecture prone to performance bottlenecks when the network is congested, resulting in a decline in application service quality, thereby affecting user experience. In addition, the emergence of network congestion is often accompanied by a large number of user requests to access the database. In this high-load, high-concurrency scenario, the call frequency between microservices will increase dramatically, especially at the last level of the call chain, that is, the microservice container that finally accesses the database, the performance bottleneck is more obvious. It can be seen that the response performance of the database system will become a key factor affecting the performance of the microservice architecture application system. If the access efficiency of the database cannot be guaranteed, the overall response time of the application will increase significantly, which will eventually have a negative impact on the user experience.

[0005] Although database systems can improve performance through parallel processing, index optimization, query optimization and other means, these optimization measures still have their limitations. When faced with a large number of concurrent access requests, database performance will still be exhausted quickly. Even in high-performance database systems, these optimization measures cannot completely eliminate performance bottlenecks. In order to solve this problem, it is particularly important to introduce a cache mechanism. By introducing a cache between the application layer and the database layer, the call chain can achieve high-speed access to hot data, thereby greatly reducing direct access to the database, effectively reducing the database load and reducing the pressure on network bandwidth, and ultimately improving the overall response performance of the application to user requests.

[0006] In modern microservice architectures, cache containers are usually deployed on different backend storage systems to speed up data reading and writing. However, this also brings some problems. First, due to uneven cache allocation, some nodes may be configured with too many cache resources, while other nodes may not have enough cache. This unbalanced allocation scheme leads to reduced cache utilization, further causing a waste of system resources. Secondly, since in a microservice architecture, multiple containers need to be accessed to complete a request, a container path with data dependencies is formed. Adding cache to a container path with data dependencies can greatly reduce the request response time, because after the request data hits the cache, it does not need to access other containers on the path to obtain the required data. However, in a large-scale microservice cluster, how to efficiently layout cache containers to significantly reduce user request response time and reduce cache costs is an urgent problem to be solved in the current microservice architecture performance improvement. Among them, a cache container is a container that can provide cache services. Summary of the invention

[0007] In response to at least one of the above technical problems, an embodiment of the present invention provides a cache container layout method, device, medium, and equipment for a microservice architecture.

[0008] According to the first aspect, the cache container layout method for microservice architecture provided by an embodiment of the present invention includes:

[0009] Determine the data dependency relationship of each container in the container cluster corresponding to the microservice architecture, and divide the containers into a plurality of candidate container pairs according to the data dependency relationship; wherein each candidate container pair includes an upstream container and a downstream container;

[0010] According to a preset cache container insertion condition, at least one target container pair capable of being inserted into the cache container is selected from the plurality of candidate container pairs;

[0011] The cache capacity of the cache container to be inserted into each target container pair is regularly determined by a deep reinforcement learning algorithm; wherein the deep reinforcement learning algorithm outputs the cache capacity that minimizes the user request delay and the total cache amount through the learned relationship between the cache capacity and the user request delay;

[0012] According to each target container pair and the cache capacity of the cache container to be inserted into each target container pair, the cache container is deployed and the cache capacity is configured in the container cluster.

[0013] In one embodiment, the cache container insertion condition includes at least one of the following:

[0014] Both containers in the target container pair are not cache containers; and,

[0015] Only one target container pair is selected from the candidate container pairs on the sequential branch in the container dependency chain formed by the same request, and the target container pair is at the front end of the sequential branch; or, the target container pair is on the container dependency chain associated with the database; or, the target container pair is at the front end of the container dependency chain or near the front end.

[0016] In one embodiment, the step of periodically determining the cache capacity of the cache container to be inserted into each target container pair by using a deep reinforcement learning algorithm includes:

[0017] S11, initializing the Q value of each action in the action space to 0; wherein the Q value represents the expected value of the reward of an action, and the action is used to characterize the cache capacity of the cache container inserted in each target container pair;

[0018] S12, collecting state information of the cache containers inserted in each target container pair at a preset time interval, and forming a state vector at the current moment according to the state information of the cache containers inserted in each target container pair;

[0019] S13, generate a random number x in the range of [0, 1], and compare the random number x with the probability parameter ε; if the random number x is less than or equal to the probability parameter ε, select the action with the largest Q value from the Q values ​​corresponding to the actions in the action space calculated in the previous round; if the random number x is greater than the probability parameter ε, randomly select an action in the action space; and output the cache capacity of the cache container inserted in each target container pair according to the selected action;

[0020] S14. If the action with the largest Q value is selected, the Q value of each action in the action space is calculated according to the state vector of the container pair in this round, so as to be used in S13 of the next round.

[0021] In one embodiment, the status information of the cache container includes a cache hit rate, a cache access data volume, and a cache capacity.

[0022] In one embodiment, the method further comprises:

[0023] After executing the deployment of cache containers and cache capacity configuration in the container cluster according to the action selected in S13 of the current round, determining a state vector at a next moment of the current moment;

[0024] According to the state vector at the next moment, calculate the reward corresponding to the action selected in S13 of the current round, and store the state vector at the current moment, the action selected in S13 of the current round, the reward corresponding to the action, and the state vector at the next moment as one piece of data in the experience replay buffer;

[0025] Determining whether the experience replay buffer is full;

[0026] If so, the target deep Q network is updated when the update conditions are met; the loss value is calculated according to the preset loss function; the Q value Q corresponding to the action selected by the target deep Q network according to the state vector at the next moment is determined target And the Q value Q corresponding to the action selected by the online deep Q network according to the state vector at the current moment online , according to Q target and Q online Calculating the time difference error; updating the online deep Q network; wherein the target deep Q network and the online deep Q network are two networks in the deep reinforcement learning algorithm.

[0027] In one embodiment, the calculating of the reward corresponding to the action selected in S13 of the current round includes: calculating the reward using a first calculation formula, wherein the first calculation formula is:

[0028]

[0029] Among them, r is the reward, α is the proportional coefficient, Average response delay for user requests, Cache j is the cache capacity of the jth cache container, and j is the number of the cache container.

[0030] In one embodiment, the calculating the time difference error includes using a second calculation formula to calculate the time difference error, and the second calculation formula is:

[0031] TDError=r+γ·Q target –Q online

[0032] Among them, TDError is the time difference error, r is the reward obtained after the action is performed, γ is the discount factor, Q target It is the Q value corresponding to the action selected by the target deep Q network according to the state vector at the next moment, Q online It is the Q value corresponding to the action selected by the online deep Q network according to the state vector at the current moment.

[0033] According to the second aspect, the cache container layout device for microservice architecture provided by an embodiment of the present invention includes:

[0034] A first determination module is used to determine the data dependency relationship of each container in the container cluster corresponding to the microservice architecture, and divide the containers into a plurality of candidate container pairs according to the data dependency relationship; wherein each candidate container pair includes an upstream container and a downstream container;

[0035] A first screening module, configured to screen out at least one target container pair that can be inserted into the cache container from the plurality of candidate container pairs according to a preset cache container insertion condition;

[0036] A capacity determination module, used to periodically determine the cache capacity of the cache container to be inserted into each target container pair through a deep reinforcement learning algorithm; wherein the deep reinforcement learning algorithm outputs the cache capacity that minimizes the user request delay and the total cache amount through the learned relationship between the cache capacity and the user request delay;

[0037] The container layout module is used to deploy cache containers and configure cache capacity in the container cluster according to each target container pair and the cache capacity of the cache container to be inserted into each target container pair.

[0038] According to a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method provided in the first aspect.

[0039] According to a fourth aspect, an embodiment of the present invention provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided in the first aspect is implemented.

[0040] The cache container layout method, device, medium, and equipment for microservice architecture provided in the embodiment of the present invention determine the data dependency of each container in the container cluster corresponding to the microservice architecture, and divide the each container into multiple candidate container pairs according to the data dependency, and then select at least one target container pair that can be inserted into the cache container from the multiple candidate container pairs. Then, the cache capacity of the cache container to be inserted into each target container pair is regularly determined by the deep reinforcement learning algorithm, and finally, according to the cache capacity of each target container pair and the cache container to be inserted into each target container pair, the cache container is deployed and the cache capacity is configured in the container cluster, thereby realizing the layout of the cache container. It can be seen that in the embodiment of the present invention, first, by analyzing the dependency between all containers in the microservice architecture, the candidate container pairs with dependencies are identified; then, some target container pairs are selected from the candidate container pairs to be inserted into the cache container; then, the deep reinforcement learning algorithm can mine the relationship between the cache capacity and the user request delay based on the state information of the cache container, and then output the cache capacity allocated to the cache container, and then layout the cache container, minimize the request delay with the lowest cache cost, that is, obtain the maximum performance improvement with a lower cache cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of a cache container layout method for a microservice architecture in one embodiment of the present invention;

[0042] Figure 2 This is an example of a microservice architecture in an embodiment of the present invention;

[0043] Figure 3 It is a structural block diagram of a cache container layout device for microservice architecture in one embodiment of the present invention. DETAILED DESCRIPTION

[0044] In a first aspect, an embodiment of the present invention provides a cache container layout method for a microservice architecture, see Figure 1 The method comprises the following steps S110 to S140:

[0045] S110, determining data dependencies of containers in a container cluster corresponding to a microservice architecture, and dividing the containers into a plurality of candidate container pairs according to the data dependencies; wherein each candidate container pair includes an upstream container and a downstream container;

[0046] For example, in a microservice architecture, there are three dependency chains corresponding to three types of requests. Dependency chain 1 and dependency chain 3 need to read data from type A database and type B database respectively, and dependency chain 2 needs to write data to type A database. The candidate container pairs with dependencies are: containers A and B, containers B and C, containers B and F, containers B and H, containers C and D, containers D and E, containers F and G. The above candidate container pairs have a dependency relationship, that is, they include an upstream container and a downstream container. The request processing of the upstream container depends on the data returned by the downstream container, that is, the upstream container initiates a data request to the downstream container, and the downstream container returns the data. The above container pairs are all candidate container pairs.

[0047] S120, selecting at least one target container pair capable of inserting the cache container from the plurality of candidate container pairs according to a preset cache container insertion condition;

[0048] The cache container insertion condition may include at least one of the following:

[0049] Both containers in the target container pair are not cache containers; and,

[0050] Only one target container pair is selected from the candidate container pairs on the sequential branch in the container dependency chain formed by the same request, and the target container pair is at the front end of the sequential branch; or, the target container pair is on the container dependency chain associated with the database; or, the target container pair is at the front end of the container dependency chain or near the front end.

[0051] It is understandable that in the embodiment of the present invention, the containers in the microservice architecture are divided into the following three categories:

[0052] (1) Front-end container: Front-end containers include containers that interact directly with users, receive and distribute user requests from clients, and present user interfaces. Some common front-end containers include Web application containers such as nginx, API containers, and content distribution network containers. The upstream container of a front-end container pair must be a front-end container, and the downstream container is a front-end container or a container at the junction of the "front-end-middle layer".

[0053] (2) Middle-layer containers: Middle-layer containers include containers that handle core business, request distribution, and coordination between services. The upstream and downstream containers of a container pair in the middle layer are both middle-layer containers.

[0054] (3) Backend containers. Backend containers include cache or database type containers that are responsible for storing, retrieving, caching, and managing data. Common backend containers include cache containers and database containers.

[0055] It can be seen that the above cache container insertion conditions can be summarized as: neither the upstream container nor the downstream container is a cache container and meets one of the following conditions: at the front end; at the back end; at the middle layer, and at the front of the sequential branch in the container dependency chain formed by the same request.

[0056] It can be seen that the insertion range of the cache container can be narrowed according to the above cache container insertion conditions.

[0057] S130, regularly determining the cache capacity of the cache container to be inserted into each target container pair through a deep reinforcement learning algorithm; wherein the deep reinforcement learning algorithm outputs the cache capacity that minimizes the user request delay and the total cache amount through the learned relationship between the cache capacity and the user request delay;

[0058] It is understandable that when the cache capacity of a cache container to be inserted into a target container pair determined based on the deep reinforcement learning algorithm is 0, it means that a cache container will not be inserted into this target container pair. If it is not 0, a cache container needs to be inserted, and the cache size of the cache container needs to be configured according to the determined cache capacity.

[0059] Among them, the deep reinforcement learning algorithm is a machine learning method that learns the best strategy through interaction with the environment. It obtains feedback from the environment through continuous trial and error, and gradually adjusts the strategy to maximize long-term returns. In the cache layout problem, the deep reinforcement learning algorithm can continuously and adaptively adjust and optimize the cache capacity allocation plan by learning the historical status data of cache resource usage to cope with dynamic loads, optimize system performance, and reduce cache overhead.

[0060] The step of periodically determining the cache capacity of the cache container to be inserted into each target container pair by using the deep reinforcement learning algorithm in S130 may include S11 to S14:

[0061] S11, initializing the Q value of each action in the action space to 0; wherein the Q value represents the expected value of the reward of an action, and the action is used to characterize the cache capacity of the cache container inserted in each target container pair;

[0062] The action space includes multiple actions, each of which represents an optional container capacity of a cache container inserted into each target container pair.

[0063] S12, collecting state information of the cache containers inserted in each target container pair at a preset time interval, and forming a state vector at the current moment according to the state information of the cache containers inserted in each target container pair;

[0064] The state information of the cache container may include cache hit rate, cache access data volume and cache capacity, that is, the cache hit rate, cache access data volume and cache capacity of the cache container inserted in each target container pair are collected at regular intervals.

[0065] Among them, the state vector formed at the current moment can be recorded as s t .

[0066] S13, generate a random number x in the range of [0, 1], and compare the random number x with the probability parameter ε; if the random number x is less than or equal to the probability parameter ε, select the action with the largest Q value from the Q values ​​corresponding to the actions in the action space calculated in the previous round; if the random number x is greater than the probability parameter ε, randomly select an action in the action space; and output the cache capacity of the cache container inserted in each target container pair according to the selected action;

[0067] It can be seen that a random number x is generated in the interval [0,1], and x is compared with the probability parameter ε, 0≤ε≤1. If x is in the range [0,ε], the action with the largest Q value returned in the previous round can be adopted; otherwise, an action is randomly selected in the action space.

[0068] Among them, the deep reinforcement learning algorithm performs a judgment for each cache container to determine how much available space to allocate to the cache container. In order to limit the cache cost, the cache size of the total cache container will not exceed a maximum value cachemax. The value of the cache allocation space of each cache container is between [0, cachemax], where cachemax is the maximum total cache capacity. The action vector is defined as an n*1 vector, where n is the total number of cache containers. The action vector satisfies the following conditions: the sum of the values ​​of all scalars does not exceed cachemax, that is, the maximum available cache capacity; for the middle layer, only one pair of target containers is selected on the sequential branch in the container dependency chain formed by each request to insert into the cache container.

[0069] It can be seen that the embodiment of the present invention adopts the ε-greedy exploration strategy, that is, whenever the deep reinforcement learning algorithm needs to select an action, there is a probability of ε to randomly take an action in the action space as an action decision, otherwise, the optimal action calculated by the deep reinforcement learning algorithm is selected as the action decision.

[0070] S14. If the action with the largest Q value is selected, the Q value of each action in the action space is calculated according to the state vector of the container pair in this round, so as to be used in S13 of the next round.

[0071] It can be seen that when selecting the action with the largest Q value, the online deep Q network will select the state vector s according to the current input state vector s. t Calculate the Q value of each action in the action space.

[0072] Among them, taking an action a as an example, the online deep Q network first observes and collects the expected state vector s at the next moment after executing the action t+1 ', calculate the reward of the action according to the estimated fitting function of the neural network, that is, the reward r corresponding to action a a . The expected state vector s at the next moment t+1 'Input into the target deep Q network, the target deep Q network is based on the target network parameter θ - Calculate the expected state vector s at the next moment t+1 'The Q value of each action in the corresponding action space, and select the maximum value from it, recorded as max a’ Q(s t ',a',θ - ), where a' is the action corresponding to the maximum Q value. At this time, according to the learning rate w and the discount factor γ, the update calculation is obtained in the current state s t The Q value corresponding to the next action a, that is, the Q function: Q(s t ,a)=w[r a +γmax a’ Q(s t',a',θ - )].

[0073] The Q function is used to estimate the expected reward for each action in the current action space, and the ε-greedy exploration strategy is combined to output the decision action. The Q function is a reward expectation calculation function for the action, and the reward expectation includes the current reward and all possible rewards in the future.

[0074] In one embodiment, the method provided by the embodiment of the present invention may further include A1 to A4:

[0075] A1. After executing the deployment of cache containers and cache capacity configuration in the container cluster according to the action selected in S13 of the current round, determining a state vector at a next moment of the current moment;

[0076] It is understandable that after the layout of the cache container is executed according to the action selected in a round of S13, the cache container will perform a cache operation when the dependency chain in which it is located processes the user request. Then, the state vector of each cache container is collected in the next collection cycle, that is, the state vector at the next moment.

[0077] A2. Calculate the reward corresponding to the action selected in S13 of the current round according to the state vector at the next moment, and store the state vector at the current moment, the action selected in S13 of the current round, the reward corresponding to the action, and the state vector at the next moment as one piece of data in the experience playback buffer;

[0078] In one embodiment, the calculating of the reward corresponding to the action selected in S13 of the current round may include: calculating the reward using a first calculation formula, where the first calculation formula is:

[0079]

[0080] Among them, r is the reward, α is the proportional coefficient, Average response delay for user requests, Cache j is the cache capacity of the jth cache container, and j is the number of the cache container.

[0081] Next, after calculating the reward, the current state vector s t The action selected in S13 of the current round is based on the action selected in the round at the current time t, the reward r corresponding to the action, and the state vector s at the next time t+1 Stored as a piece of data in the experience replay buffer.

[0082] A3, determining whether the experience playback buffer is full;

[0083] A4. If yes, then update the target deep Q network when the update condition is met; calculate the loss value according to the preset loss function; determine the Q value Q corresponding to the action selected by the target deep Q network according to the state vector at the next moment target And the Q value Q corresponding to the action selected by the online deep Q network according to the state vector at the current moment online , according to Q target and Q online Calculating the time difference error; updating the online deep Q network; wherein the target deep Q network and the online deep Q network are two networks in the deep reinforcement learning algorithm.

[0084] The experience replay buffer has an upper limit on storage capacity. When the number of data reaches the upper limit, it means that the experience replay buffer is full. If it is full, the deep reinforcement learning algorithm will execute the following instructions: First, determine whether the update conditions are met. If so, update the "target deep Q network". Specifically, copy the values ​​of the network parameters in the "main neural network" to the network parameters ω in the "target neural network". Then, calculate the loss value loss according to the set loss function, and calculate the Q value Q of the online deep Q network respectively. online And the Q value Q of the target deep Q network target ,Finally, the time difference error TDError is calculated and the network parameters of the online deep Q network are updated.

[0085] Among them, the loss function adopts the binary cross entropy loss function BCELoss, and its formula is as follows:

[0086]

[0087] Where N is the number of samples; y i is the true label of the i-th sample, which takes a value of 0 or 1; It is the predicted probability of the deep reinforcement learning algorithm for the i-th sample, indicating the probability of belonging to category 1, and its value range is between 0 and 1.

[0088] The calculating of the time difference error may include: calculating the time difference error using a second calculation formula, wherein the second calculation formula is:

[0089] TDError=r+γ·Q target –Q online

[0090] Among them, TDError is the time difference error, r is the reward obtained after the action is performed, γ is the discount factor, Q target It is the Q value corresponding to the action selected by the target deep Q network according to the state vector at the next moment, Q onlineIt is the Q value corresponding to the action selected by the online deep Q network according to the state vector at the current moment.

[0091] It can be seen that based on A1 to A4, after each execution of the layout of the cache container according to the action selected in a round of S13, the loss function and the time difference error will be calculated, and the target deep Q network and the online deep Q network in the deep reinforcement learning algorithm will be updated. The calculated loss function and the time difference error can be used to further optimize the deep reinforcement learning algorithm. Updating the target deep Q network and the online deep Q network can further optimize the two main networks of the deep reinforcement learning algorithm, so that the next time S120 is executed, a better action vector can be given according to the deep reinforcement learning algorithm.

[0092] It can be seen that the deep reinforcement learning algorithm has designed an experience replay mechanism. The experience replay buffer will store the current state vector, the action selected at the current moment, the reward corresponding to the selected action, and the state vector at the next moment when each round of cache allocation is performed. These data are stored as experience data. Then, random sampling can be performed from the stored data to train the network in the deep reinforcement learning algorithm, thereby improving the stability and performance of the algorithm. In this way, the screening effect gradually reaches the learning goal as the number of rounds increases. The learning goal is to reduce the average response time of user I / O requests and the total cache cost as much as possible.

[0093] That is, the deep reinforcement learning algorithm provides feedback on the cache space allocation effect of each cache container based on a reward mechanism, and then optimizes the algorithm so that better actions can be selected later.

[0094] S140: Deploy cache containers and configure cache capacity in the container cluster according to each target container pair and the cache capacity of the cache container to be inserted into each target container pair.

[0095] It can be seen that the cache capacity of the cache container that needs to be inserted into each target container pair can be determined based on the action output by the deep reinforcement learning algorithm. The output action is actually an action vector. Taking a container cluster containing n target container pairs as an example, the action vector output by the deep reinforcement learning algorithm is a one-dimensional vector composed of n scalars, that is, [cache 1 ,cache 2 ,…cache n ] and satisfy the following two conditions: (1) 0≤cache i ≤cache max (1≤i≤n); (2)∑ i cache i ≤cache max (1≤i≤n), where cachemax is the maximum cache capacity threshold.

[0096] For each target container pair, the capacity setting value of the inserted cache container is one of the following values. Set = {0, cache max / maxnum*1,cache max / maxnum*2,…,cache max / maxnum*(maxnum-1),cache max}. Where maxnum is the maximum number of cache containers that can be inserted simultaneously in the microservice architecture, that is, the maximum number of non-zero scalars that can exist in the action vector. This value is subject to the aforementioned cache container insertion conditions. max ]According to the step cache max / maxnum is divided into equal parts, and the values ​​corresponding to the divided point sets constitute the optional range of the capacity setting value.

[0097] For example, see Figure 2 The service architecture shown in the figure has three pairs of target container pairs, including container pair A and B, container pair B and C, and container pair C and D. Assume that the maximum available cache capacity is max If the target container is 6MB, the interval [0,6] is divided according to the step size of 6 / 3 = 2, so that the optional values ​​of the insertable container capacity for each target container are the set {0,2,4,6}. Therefore, the action space is defined as the set consisting of the following action vectors {[0,0,0],[2,0,0],[0,2,0],[0,0,2],[4,0,0],[0,4,0],[0,0,4],[6,0,0],[0,6,0],[0,0,6],[2,2,0],[2,2,0],[0,2,2],[2,2,2]}.

[0098] The network used by the deep reinforcement learning algorithm can be a typical Double DQN architecture, which consists of two deep Q networks, one of which acts as an online network and the other acts as a target network. Both deep Q networks can contain several layers of basic network structures, which are: input layer, one-dimensional convolutional neural network layer, several fully connected layers, and output layer.

[0099] In an embodiment of the present invention, the state vector of the cache container is collected regularly, and the state vector is input into the deep reinforcement learning algorithm to obtain the action vector, thereby knowing the cache container capacity allocation strategy in this round. Next, the cache container capacity allocation strategy is executed, that is, the cache container is inserted into the target container pair and the cache capacity is set to realize the layout of the cache container. Finally, after the execution of this round of actions, the reward is calculated, and the state vector at the current moment, the action selected in S13 of this round, the reward corresponding to the action, and the state vector at the next moment are stored as a four-tuple data in the experience playback buffer, and then related calculations and update operations can be performed.

[0100] In an embodiment of the present invention, first, by analyzing the dependencies between all containers in the microservice architecture, candidate container pairs with dependencies are identified; then, some target container pairs are selected from the candidate container pairs to be inserted into the cache container; then, the deep reinforcement learning algorithm mines the relationship between the cache capacity and the user request latency based on the status information of the cache container, and then outputs the cache capacity allocated to the cache container, and then layouts the cache container to minimize the request latency with the lowest cache cost, that is, to obtain the maximum performance improvement with a lower cache cost.

[0101] In a second aspect, an embodiment of the present invention provides a cache container layout device for a microservice architecture, see Figure 3 , the device 100 comprises:

[0102] The first determination module 110 is used to determine the data dependency relationship of each container in the container cluster corresponding to the microservice architecture, and divide the containers into a plurality of candidate container pairs according to the data dependency relationship; wherein each candidate container pair includes an upstream container and a downstream container;

[0103] A first screening module 120, configured to screen out at least one target container pair capable of being inserted into the cache container from the plurality of candidate container pairs according to a preset cache container insertion condition;

[0104] A capacity determination module 130 is used to periodically determine the cache capacity of the cache container to be inserted into each target container pair through a deep reinforcement learning algorithm; wherein the deep reinforcement learning algorithm outputs the cache capacity that minimizes the user request delay and the total cache amount through the learned relationship between the cache capacity and the user request delay;

[0105] The container layout module 140 is used to deploy cache containers and configure cache capacity in the container cluster according to each target container pair and the cache capacity of the cache container to be inserted into each target container pair.

[0106] In one embodiment, the cache container insertion condition includes at least one of the following:

[0107] Both containers in the target container pair are not cache containers; and,

[0108] Only one target container pair is selected from the candidate container pairs on the sequential branch in the container dependency chain formed by the same request, and the target container pair is at the front end of the sequential branch; or, the target container pair is on the container dependency chain associated with the database; or, the target container pair is at the front end of the container dependency chain or near the front end.

[0109] In one embodiment, the capacity determination module 130 is specifically configured to perform the following steps:

[0110] S11, initializing the Q value of each action in the action space to 0; wherein the Q value represents the expected value of the reward of an action, and the action is used to characterize the cache capacity of the cache container inserted in each target container pair;

[0111] S12, collecting state information of the cache containers inserted in each target container pair at a preset time interval, and forming a state vector at the current moment according to the state information of the cache containers inserted in each target container pair;

[0112] S13, generate a random number x in the range of [0, 1], and compare the random number x with the probability parameter ε; if the random number x is less than or equal to the probability parameter ε, select the action with the largest Q value from the Q values ​​corresponding to the actions in the action space calculated in the previous round; if the random number x is greater than the probability parameter ε, randomly select an action in the action space; and output the cache capacity of the cache container inserted in each target container pair according to the selected action;

[0113] S14. If the action with the largest Q value is selected, the Q value of each action in the action space is calculated according to the state vector of the container pair in this round, so as to be used in S13 of the next round.

[0114] In one embodiment, the status information of the cache container includes a cache hit rate, a cache access data volume, and a cache capacity.

[0115] In one embodiment, the apparatus further comprises:

[0116] The optimization update module is used to determine the state vector of the next moment of the current moment after the deployment of the cache container and the cache capacity configuration are completed in the container cluster according to the action selected in S13 of the current round; calculate the reward corresponding to the action selected in S13 of the current round according to the state vector of the next moment, and store the state vector of the current moment, the action selected in S13 of the current round, the reward corresponding to the action, and the state vector of the next moment as a piece of data in the experience replay buffer; determine whether the experience replay buffer is full; if so, update the target deep Q network when the update condition is met; calculate the loss value according to the preset loss function; determine the Q value Q corresponding to the action selected by the target deep Q network according to the state vector of the next moment target And the Q value Q corresponding to the action selected by the online deep Q network according to the state vector at the current moment online , according to Q target and Q online Calculating the time difference error; updating the online deep Q network; wherein the target deep Q network and the online deep Q network are two networks in the deep reinforcement learning algorithm.

[0117] In one embodiment, the optimization update module is specifically used to calculate the reward using a first calculation formula, where the first calculation formula is:

[0118]

[0119] Among them, r is the reward, α is the proportional coefficient, Average response delay for user requests, Cache j is the cache capacity of the jth cache container, and j is the number of the cache container.

[0120] In one embodiment, the optimization and updating module is specifically used to: calculate the time difference error using a second calculation formula, and the second calculation formula is:

[0121] TDError=r+γ·Q target –Q online

[0122] Among them, TDError is the time difference error, r is the reward obtained after the action is performed, γ is the discount factor, Q target It is the Q value corresponding to the action selected by the target deep Q network according to the state vector at the next moment, Q online It is the Q value corresponding to the action selected by the online deep Q network according to the state vector at the current moment.

[0123] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0124] In a third aspect, an embodiment of the present invention provides a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method provided in the first aspect.

[0125] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can read and execute the program code stored in the storage medium.

[0126] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0127] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0128] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0129] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0130] It is understandable that the explanation, specific implementation methods, beneficial effects, examples, etc. of the relevant contents in the computer-readable medium provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0131] In a fourth aspect, an embodiment of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method in any embodiment of the specification.

[0132] It is understandable that the explanation, specific implementation, beneficial effects, examples, etc. of the relevant contents in the computing device provided in the embodiment of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0133] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0134] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, widgets, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0135] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cache container layout method for microservice architecture, characterized in that: include: Determine the data dependency relationship of each container in the container cluster corresponding to the microservice architecture, and divide the containers into a plurality of candidate container pairs according to the data dependency relationship; wherein each candidate container pair includes an upstream container and a downstream container; According to a preset cache container insertion condition, at least one target container pair capable of being inserted into the cache container is selected from the plurality of candidate container pairs; The cache capacity of the cache container to be inserted into each target container pair is regularly determined by a deep reinforcement learning algorithm; wherein the deep reinforcement learning algorithm outputs the cache capacity that minimizes the user request delay and the total cache amount through the learned relationship between the cache capacity and the user request delay; According to each target container pair and the cache capacity of the cache container to be inserted into each target container pair, the cache container is deployed and the cache capacity is configured in the container cluster.

2. The method according to claim 1, characterized in that The cache container insertion condition includes at least one of the following: Both containers in the target container pair are not cache containers; and, Only one target container pair is selected from the candidate container pairs on the sequential branch in the container dependency chain formed by the same request, and the target container pair is at the front end of the sequential branch; or; the target container pair is on the container dependency chain associated with the database; or; the target container pair is at the front end of the container dependency chain or near the front end.

3. The method according to claim 1, characterized in that The step of periodically determining the cache capacity of the cache container to be inserted into each target container pair by using a deep reinforcement learning algorithm includes: S11, initializing the Q value of each action in the action space to 0; wherein the Q value represents the expected value of the reward of an action, and the action is used to characterize the cache capacity of the cache container inserted in each target container pair; S12, collecting state information of the cache containers inserted in each target container pair at a preset time interval, and forming a state vector at the current moment according to the state information of the cache containers inserted in each target container pair; S13, generate a random number x in the range of [0, 1], and compare the random number x with the probability parameter ε; if the random number x is less than or equal to the probability parameter ε, select the action with the largest Q value from the Q values ​​corresponding to the actions in the action space calculated in the previous round; if the random number x is greater than the probability parameter ε, randomly select an action in the action space; and output the cache capacity of the cache container inserted in each target container pair according to the selected action; S14. If the action with the largest Q value is selected, the Q value of each action in the action space is calculated according to the state vector of the container pair in this round, so as to be used in S13 of the next round.

4. The method according to claim 3, characterized in that The status information of the cache container includes a cache hit rate, a cache access data volume, and a cache capacity.

5. The method according to claim 3, characterized in that: Also includes: After executing the deployment of cache containers and cache capacity configuration in the container cluster according to the action selected in S13 of the current round, determining a state vector at a next moment of the current moment; According to the state vector at the next moment, calculate the reward corresponding to the action selected in S13 of the current round, and store the state vector at the current moment, the action selected in S13 of the current round, the reward corresponding to the action, and the state vector at the next moment as one piece of data in the experience replay buffer; Determining whether the experience replay buffer is full; If so, the target deep Q network is updated when the update conditions are met; the loss value is calculated according to the preset loss function; the Q value Q corresponding to the action selected by the target deep Q network according to the state vector at the next moment is determined target And the Q value Q corresponding to the action selected by the online deep Q network according to the state vector at the current moment online , according to Q target and Q online Calculating the time difference error; updating the online deep Q network; wherein the target deep Q network and the online deep Q network are two networks in the deep reinforcement learning algorithm.

6. The method according to claim 5, characterized in that The calculating the reward corresponding to the action selected in S13 of the current round includes: calculating the reward using a first calculation formula, where the first calculation formula is: Among them, r is the reward, α is the proportional coefficient, Average response delay for user requests, Cache j is the cache capacity of the jth cache container, and j is the number of the cache container.

7. The method according to claim 5, characterized in that The calculating of the time difference error includes using a second calculation formula to calculate the time difference error, wherein the second calculation formula is: TDError=r+γ·Q target –Q online Among them, TDError is the time difference error, r is the reward obtained after the action is performed, γ is the discount factor, Q target It is the Q value corresponding to the action selected by the target deep Q network according to the state vector at the next moment, Q online It is the Q value corresponding to the action selected by the online deep Q network according to the state vector at the current moment.

8. A cache container layout device for microservice architecture, characterized in that: include: A first determination module is used to determine the data dependency relationship of each container in the container cluster corresponding to the microservice architecture, and divide the containers into a plurality of candidate container pairs according to the data dependency relationship; wherein each candidate container pair includes an upstream container and a downstream container; A first screening module, configured to screen out at least one target container pair that can be inserted into the cache container from the plurality of candidate container pairs according to a preset cache container insertion condition; A capacity determination module, used to periodically determine the cache capacity of the cache container to be inserted into each target container pair through a deep reinforcement learning algorithm; wherein the deep reinforcement learning algorithm outputs the cache capacity that minimizes the user request delay and the total cache amount through the learned relationship between the cache capacity and the user request delay; The container layout module is used to deploy cache containers and configure cache capacity in the container cluster according to each target container pair and the cache capacity of the cache container to be inserted into each target container pair.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.

Images