Joint control method and device for adjusting the number of container instances and user request diversion

Abstract

A joint control method and device for adjusting the number of container instances and diverting user requests is implemented using a trained deep reinforcement learning model. The model takes the number of container instances that can be started, the number of container instances running in the current time slot, and the user request status information of the current time slot as input, and outputs the increase and decrease information of the number of container instances required for the current time slot and the user request diversion information of the current time slot. Based on the output results of the model, the number of container instances in the cross-domain edge network is increased or decreased, and the process of diverting user requests to the corresponding edge cloud is controlled. The training of the model is achieved by solving a preset problem model, which is obtained by modeling the problem using a Markov decision method with the goal of minimizing the average user request cost per time slot of the cross-domain edge network. Therefore, the present application uses user requests as the scheduling granularity to achieve dynamic management of computing resources in a cross-domain edge network.

Description

Technical Field

[0001] The present application relates to the field of edge computing in computer networks, and in particular to a joint control method and device for adjusting the number of container instances and diverting user requests. Background Art

[0002] In the edge computing of the cross-domain edge network, the dynamic supply of computing resources is used to ensure the service quality of large-scale distributed artificial intelligence applications in the entire network. Specifically, the cross-domain edge network includes multiple distributed edge clouds loaded with artificial intelligence applications. Container instances are dynamically set in each edge cloud, and each container instance can process user requests from multiple users. When the cross-domain edge network manages computing resources for user requests, it adopts a containerized approach, that is, after calculating the diversion decision of the user request based on the computing resources required for the current user request, the number of container instances in the involved edge cloud is adjusted according to the obtained diversion decision, and the corresponding user request is diverted to the container instance in the corresponding edge cloud for processing.

[0003] In the above-mentioned computing resource management process, when the computing resources required for user requests in the cross-domain edge network increase, the corresponding edge cloud should increase the number of container instances used to process user requests, ensure sufficient computing resource supply in the cross-domain edge network, and avoid user requests queuing on the server side of the network, resulting in excessive delays; when the computing resources required for user requests in the cross-domain edge network decrease, the corresponding edge cloud needs to reduce the number of running capacity instances, thereby saving computing resources in the cross-domain edge network and improving computing resource utilization.

[0004] However, simply dynamically increasing the supply of computing resources in the cross-domain edge network has limitations in ensuring the service quality of large-scale distributed artificial intelligence applications in the cross-domain edge network. This is because, on the one hand, starting or terminating container instances in the edge cloud in the cross-domain edge network will bring time costs. For container instances, this process may last from several seconds to tens of seconds, which makes the adjustment of computing resources in the cross-domain edge network have lags and cannot achieve instant response; on the other hand, since each container instance can handle multiple user requests, such cross-domain edge network scheduling granularity cannot achieve user request-level management. Therefore, currently adjusting the supply of computing resources in the entire cross-domain edge network by changing the number of container instances is only suitable for coarse-grained and low-frequency management.

[0005] In order to solve the above problems, it is possible to consider transplanting the traditional user-level computing resource scheduling method based on computer network architecture into the cross-domain edge network. However, since the different edge clouds in the cross-domain edge network are distributed and isolated across domains, it is impossible to directly use the traditional method to schedule user requests. In addition, when the cross-domain edge network is based on a mobile communication network, due to the user plane bearer mechanism, the user's Internet Protocol (IP) data packet is encapsulated into the user-level General Radio Packet Service (GPRS) Tunneling Protocol (GTP-U) tunnel, and the IP address in the data packet cannot affect the routing of the tunnel, resulting in the domain name system (DNS) mechanism of the traditional method cannot be directly applied to schedule user requests.

[0006] Therefore, how cross-domain edge networks can manage computing resources based on user requests as the scheduling granularity has become an urgent problem to be solved. Summary of the invention

[0007] In view of this, an embodiment of the present application provides a joint control method for adjusting the number of container instances and user request diversion, which can use user requests as the scheduling granularity to achieve dynamic management of computing resources in a cross-domain edge network.

[0008] The embodiment of the present application also provides a joint control device for adjusting the number of container instances and diverting user requests, which can use user requests as the scheduling granularity to achieve dynamic management of computing resources in a cross-domain edge network.

[0009] The embodiment of the present application is implemented as follows:

[0010] In one embodiment of the present application, a joint control method for adjusting the number of container instances and user request diversion is provided, the method comprising:

[0011] With the goal of minimizing the average user request cost per time slot of the cross-domain edge network, the Markov decision method is used to model the problem. The state in the problem model is set based on the number of container instances that the system can start, the number of container instances running in the current time slot, and the user request state information of the current time slot; the action in the problem model is set based on the container instance control information of each edge cloud in the current time slot and the diversion information of the user request in the current time slot; the instant reward in the problem model is set based on the strategy of minimizing the cost of each user request in the current time slot;

[0012] Using a deep reinforcement learning model to solve the problem model to train the deep reinforcement learning model;

[0013] The number of container instances that can currently be started in the network, the number of container instances running in the current time slot, and the user request status information of the current time slot are input into the trained deep reinforcement learning model, and the increase or decrease information of the number of container instances required for the current time slot and the user request diversion information of the current time slot are output.

[0014] In the above embodiment, the problem of minimizing the average user request cost per time slot of the network includes:

[0015] In the network, an upper limit value of the number of container instances in each edge cloud is set, and the number of container instances in the edge cloud is dynamically adjusted and user request diversion information is calculated in each time slot, so that the average cost for each user request in each time slot is minimized.

[0016] In the above embodiment, the state in the problem model is set based on the number of container instances that can be started by the system, the number of container instances running in the current time slot, and the user request state information of the current time slot, including:

[0017] The upper limit of the number of container instances that the network can start (M i );

[0018] The number of container instances running in the current time slot

[0019] The number of user requests that are in service and have not yet been completed in the current time slot;

[0020] and the number of user requests received by each edge cloud in the current time slot.

[0021] In the above embodiment, the actions in the problem model are based on the container instance control information of each edge cloud in the current time slot and the diversion information setting of the user request in the current time slot, including:

[0022] Each action (a i ) contains the container instance control information of each edge cloud in the current time slot and the diversion information of the user request in the current time slot;

[0023] The container instance control information of each edge cloud includes information about adding a container instance, information about reducing a container instance, and information about the container instance remaining unchanged;

[0024] For a first number n e edge clouds and correspondingly have a set second number n q The network requested by a user, action a i Expressed as the first number n e The container instance control information of the edge cloud and the second number nq A vector of diversion information requested by a user;

[0025] The action (a i ) is implemented as follows: multiple consecutive actions (a′ i ) is discretized, and the discretized action (a i ), where the discretized action (a i ) in the first number (n e ) vectors represent the capacity instance control information corresponding to each edge cloud, and the second number (n q ) vectors represent whether the corresponding user request is diverted to the edge cloud.

[0026] In the above embodiment, the instant reward is based on the strategy setting with the lowest cost per user request in the current time slot, including:

[0027] The immediate reward is defined as: w t -c t ,in, represents the reward value brought by the user request whose delay satisfaction rate reaches the set delay satisfaction rate threshold in the current time slot (t), and w0 is the reward value when a single user request meets the set delay satisfaction threshold; c t is the total cost of the system in the current time slot, including the operating cost of the occupied computing resources from the previous time slot to the current time slot, the container instance switch cost, and the user request default cost.

[0028] In the above embodiment, the deep reinforcement learning model is implemented using the dominant action comment algorithm A2C.

[0029] In the above embodiment, the method further includes:

[0030] In the next time slot, the core network of the network increases or decreases the number of corresponding container instances in each edge cloud in the network according to the increase or decrease information of the number of container instances required in the current time slot;

[0031] In the next time slot, the core network of the network establishes a protocol data unit session to the corresponding user, and diverts the user request to the corresponding edge cloud according to the user request diversion information of the current time slot.

[0032] In another embodiment of the present application, a joint control device for adjusting the number of container instances and user request diversion is provided, comprising: a problem modeling unit, a model training unit and a model application unit, wherein:

[0033] A problem modeling unit is used to minimize the average user request cost per time slot of the cross-domain edge network and adopt a Markov decision method to perform problem modeling. The state in the problem model is set based on the number of container instances that can be started by the system, the number of container instances running in the current time slot, and the user request state information of the current time slot; the action in the problem model is set based on the container instance control information of each edge cloud in the current time slot and the diversion information of the user request in the current time slot; the instant reward in the problem model is set based on the strategy of minimizing the cost of each user request in the current time slot;

[0034] The model training unit is used to solve the problem model using a deep reinforcement learning model to train the deep reinforcement learning model until the immediate reward in the problem model reaches a set evaluation value;

[0035] The model application unit is used to input the number of container instances that can currently be started in the network, the number of container instances running in the current time slot, and the user request status information of the current time slot into the trained deep reinforcement learning model, and output the increase or decrease information of the number of container instances required for the current time slot and the user request diversion information of the current time slot.

[0036] In the above device, it also includes:

[0037] A control unit is used to increase or decrease the number of corresponding container instances in each edge cloud in the network in the next time slot according to the increase or decrease information of the number of container instances required in the current time slot by the core network of the network; in the next time slot, the core network of the network establishes a protocol data unit session to the corresponding user, and according to the user request diversion information of the current time slot, diverts the user request to the corresponding edge cloud.

[0038] In another embodiment of the present application, an electronic device is provided, including:

[0039] processor;

[0040] The memory stores a program, wherein the program is configured to implement the above-mentioned joint control method for adjusting the number of container instances and diverting user requests when being executed by the processor.

[0041] As can be seen above, the cross-domain edge network of the embodiment of the present application uses a trained deep reinforcement learning model to manage computing resources. The deep reinforcement learning model takes the number of container instances that can be started, the number of container instances running in the current time slot, and the user request status information of the current time slot as input, and outputs the increase and decrease information of the number of container instances required for the current time slot and the user request diversion information of the current time slot. Based on the output results of the deep reinforcement learning model, the number of container instances in the cross-domain edge network is increased and decreased, and the process of diverting user requests to the corresponding edge cloud is controlled. The training of the deep reinforcement learning model is achieved by solving a preset problem model. The problem model is based on the goal of minimizing the average user request cost per unit time slot of the cross-domain edge network, and is obtained by using the Markov decision method to model the problem. Therefore, the embodiment of the present application uses user requests as the scheduling granularity to achieve dynamic management of computing resources in a cross-domain edge network. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flow chart of a joint control method for adjusting the number of container instances and diverting user requests provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a cross-domain edge network provided in an embodiment of the present application that uses user requests as the scheduling granularity to implement management of computing resources;

[0044] Figure 3 A schematic diagram of the structure of a joint control device for adjusting the number of container instances and diverting user requests provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] When managing computing resources for user requests in a cross-domain edge network, it is necessary to finely schedule each user's request and dynamically deploy computing resource supply on the edge side. If only the number of container instances on the edge side is adjusted, there will be resource waste caused by coarse-grained management and slow response caused by delays in the container switch process. And because the session management based on the GTP-U tunnel in the communication network is different from the routing method directly based on the IP address of the user packet in the traditional computer network, it is impossible to directly transplant the traditional user-level resource scheduling method based on the computer network architecture to the cross-domain edge network for implementation.

[0049] In order to overcome the above-mentioned problems, the cross-domain edge network of the embodiment of the present application uses a trained deep reinforcement learning model to manage computing resources. The deep reinforcement learning model takes the number of container instances that can be started, the number of container instances running in the current time slot, and the user request status information of the current time slot as input, and outputs the increase and decrease information of the number of container instances required for the current time slot and the user request diversion information of the current time slot. Based on the output results of the deep reinforcement learning model, the number of container instances in the cross-domain edge network is increased or decreased, and the process of diverting user requests to the corresponding edge cloud is controlled. The training of the deep reinforcement learning model is achieved by solving a preset problem model, which is based on the goal of minimizing the average user request cost per time slot of the cross-domain edge network, and is obtained by using the Markov decision method to model the problem.

[0050] In this way, the embodiments of the present application can adjust the number of container instances started in each edge cloud, thereby dynamically adjusting the supply of computing resources to adapt to changes in user request requirements. At the same time, the cross-domain edge network dynamically offloads user requests to the corresponding edge cloud, thereby avoiding frequent adjustment of the number of container instances and ensuring the service quality on the user side.

[0051] Therefore, the embodiment of the present application uses user request as the scheduling granularity to achieve dynamic management of computing resources in a cross-domain edge network.

[0052] In the face of complex computing resources in a cross-domain edge network, the embodiment of the present application needs to ensure the operation of the cross-domain edge network at the lowest cost when jointly scheduling the diversion of user requests in the cross-domain edge network and the adjustment of the number of container instances on each edge cloud. The embodiment of the present application uses a trained deep reinforcement learning model to manage computing resources in the cross-domain edge network, which can minimize the average cost of user requests per time slot in the cross-domain edge network.

[0053] Figure 1 A flow chart of a joint control method for adjusting the number of container instances and user request diversion provided in an embodiment of the present application, wherein the specific steps include:

[0054] Step 101, with the goal of minimizing the average user request cost per time slot of the cross-domain edge network, a Markov decision method is used to model the problem, the state in the problem model is set based on the number of container instances that can be started by the system, the number of container instances running in the current time slot, and the user request state information of the current time slot; the action in the problem model is set based on the container instance control information of each edge cloud in the current time slot and the diversion information of the user request in the current time slot; the instant reward in the problem model is set based on the strategy of minimizing the cost of each user request in the current time slot;

[0055] Step 102: using a deep reinforcement learning model to solve the problem model to train the deep reinforcement learning model;

[0056] Step 103: Input the number of container instances that can currently be started in the network, the number of container instances running in the current time slot, and the user request status information of the current time slot into the trained deep reinforcement learning model, and output the increase or decrease information of the number of container instances required for the current time slot and the user request diversion information of the current time slot.

[0057] In the above method, the edge cloud includes at least one edge node, each of which may include multiple container instances, and under the control of the core network of the system, the number of container instances included is increased, decreased, or unchanged. Each container instance can process at least one user request.

[0058] In the above method, the problem is to minimize the average user request cost per time slot of the network, including:

[0059] In the network, an upper limit value of the number of container instances in each edge cloud is set, and the number of container instances in the edge cloud is dynamically adjusted in each time slot and the user request diversion information is calculated, so that the average cost of each user request in each time slot is the lowest. Specifically, the optimal problem of the average cost per time slot of user requests can be quantitatively described as: given an edge cloud e i The upper limit of the number of container instances M i , dynamically adjust the number of container instances opened in the edge cloud in each time slot And calculate the split relationship Make the average cost of each user request in each time slot the lowest. Indicates the computing split relationship. If the user request x is split to the edge cloud e j , then otherwise

[0060] In an embodiment of the present application, the problem is modeled with the goal of minimizing the average user request cost per time slot in the cross-domain edge network, and the problem modeling is performed using a Markov decision method. Specifically, when modeling the problem, the diversion of user requests and the dynamic deployment of computing resources are mapped to a Markov decision process model. A Markov decision process model M = (S, A, P, R) contains a finite set of states S, a finite set of actions A, a system's state transition probability P, and an immediate reward R after taking an action. In the above problem, the effect of the action on the network is deterministic, so the focus of the Markov decision process is to determine the state, action, and immediate reward.

[0061] Specifically, the state in the problem model is set based on the number of container instances that the system can start, the number of container instances running in the current time slot, and the user request state information of the current time slot, including: the upper limit value (M) of the number of container instances that the network can start i ); The value of the number of container instances running in the current time slot The number of user requests that are in service and have not yet ended in the current time slot; and the number of user requests received by each edge cloud in the current time slot.

[0062] The action in the problem model is based on the container instance control information of each edge cloud in the current time slot and the diversion information setting of the user request in the current time slot, including: each action (a i ) includes the container instance control information of each edge cloud in the current time slot and the diversion information of the user request in the current time slot; the container instance control information of each edge cloud includes the information of adding a container instance, reducing a container instance, and the information of keeping the container instance unchanged;

[0063] For a first number n e edge clouds and correspondingly have a set second number n q The network requested by a user, action a i Expressed as the first number n e The container instance control information of the edge cloud and the second number n q A vector of diversion information requested by a user;

[0064] The action (a i ) is implemented as follows: multiple consecutive actions (a′ i ) is discretized, and the discretized action (a i ), where the discretized action (a i ) in the first number (n e ) vectors represent the capacity instance control information corresponding to each edge cloud, and the second number (n q ) vectors represent whether the corresponding user request is diverted to the edge cloud.

[0065] In the problem model, the immediate reward is based on the strategy setting with the lowest cost for each user request in the current time slot, including:

[0066] The immediate reward is defined as: w t -c t ,in, represents the reward value brought by the user request whose delay satisfaction rate reaches the set delay satisfaction rate threshold in the current time slot (t), and w0 is the reward value when a single user request meets the set delay satisfaction threshold; c t is the total cost of the system in the current time slot, including the operating cost of the occupied computing resources from the previous time slot to the current time slot, the container instance switch cost, and the user request default cost.

[0067] In the above method, the problem model is solved by using a deep reinforcement learning model. In the process of training the deep reinforcement learning model, an immediate reward is calculated based on the action results obtained in each round of training, and the parameters in the deep reinforcement learning model are trained by backpropagation based on the immediate reward. After the total number of training rounds reaches a preset number of training rounds, the entire training ends.

[0068] In the above method, the deep reinforcement learning model is specifically implemented using the dominant action-criticism algorithm (A2C).

[0069] In the above method, when the trained deep reinforcement learning model is applied, the increase and decrease information of the number of container instances required for the current time slot and the user request diversion information of the current time slot are obtained, and the above diversion strategy can be applied to the diversion control, specifically including: in the next time slot, the core network of the network increases or decreases the number of corresponding container instances in each edge cloud in the network according to the increase and decrease information of the number of container instances required for the current time slot; in the next time slot, the core network of the network establishes a protocol data unit (PDU) session to the corresponding user, and diverts the user request to the corresponding edge cloud according to the user request diversion information of the current time slot.

[0070] That is to say, the control of the increase and decrease of container instances in each edge cloud in the network is achieved by utilizing the network's policy control function (PCF)-network open function (NEF)-application function (AF) mechanism, and the control of the diversion of user requests is achieved through PDU. The increase and decrease of container instances in each edge cloud realizes the allocation of computing resources, and the diversion of user requests realizes the allocation of user needs. The combination of the two realizes the complete scheduling of the computing resources of the network. The network can adopt the fifth-generation network (5G) system. The PCF-NEF-AF interaction mechanism of the 5G system can better solve the cross-domain information communication problem between the communication network status and the internal status of the edge cloud, and the scheduling of user requests can rely on the session establishment mechanism of the 5G system to realize the diversion of user requests.

[0071] Figure 2 The cross-domain edge network provided in the embodiment of the present application uses user requests as the scheduling granularity to implement the management structure diagram of computing resources. Figure 2 As shown, the core network in the cross-domain edge network controls the user request to be diverted to edge cloud e1, edge cloud e2 or edge cloud e3 by establishing different PDU sessions. At the same time, the PCF-NEF-AF mechanism is used to influence the number of container instances started in the three edge clouds. Figure 2 In the figure, the network micro-cellular base station (MacroBase Station) for accessing users, the micro base station (Micro Base Station) for accessing the Macro Base Station, the edge server (Edge Server) for controlling the edge cloud carrying the container instance, and the core network (CoreNetwork) of the network are marked.

[0072] Figure 3 A schematic diagram of the structure of a joint control device for adjusting the number of container instances and user request diversion provided in an embodiment of the present application includes: a problem modeling unit, a model training unit and a model application unit, wherein:

[0073] A problem modeling unit is used to minimize the average user request cost per time slot of the cross-domain edge network and adopt a Markov decision method to perform problem modeling. The state in the problem model is set based on the number of container instances that can be started by the system, the number of container instances running in the current time slot, and the user request state information of the current time slot; the action in the problem model is set based on the container instance control information of each edge cloud in the current time slot and the diversion information of the user request in the current time slot; the instant reward in the problem model is set based on the strategy of minimizing the cost of each user request in the current time slot;

[0074] The model training unit is used to solve the problem model using a deep reinforcement learning model to train the deep reinforcement learning model until the immediate reward in the problem model reaches a set evaluation value;

[0075] The model application unit is used to input the number of container instances that can currently be started in the network, the number of container instances running in the current time slot, and the user request status information of the current time slot into the trained deep reinforcement learning model, and output the increase or decrease information of the number of container instances required for the current time slot and the user request diversion information of the current time slot.

[0076] The above device can be set on a control entity of the core network in the cross-domain edge network, which is not limited here.

[0077] The above-mentioned device also includes: a control unit, which is used to increase or decrease the number of corresponding container instances in each edge cloud in the network in the next time slot according to the increase or decrease information of the number of container instances required in the current time slot by the core network of the network; in the next time slot, the core network of the network establishes a protocol data unit session to the corresponding user, and diverts the user request to the corresponding edge cloud according to the user request diversion information of the current time slot.

[0078] The following is a specific example to illustrate.

[0079] This example proposes a method for determining diversion information of user requests and the number of container instances in each edge cloud in a core network running in a cross-domain edge network, which minimizes the user request cost per time slot in the network.

[0080] First, it is necessary to quantitatively describe the network cost, which is described in detail below.

[0081] For the first time slot t1 and the second time slot t2, where t1, t2∈T, the network cost comes from three parts: container instance operation cost, container switch cost, and user request default cost. Container operation cost refers to the cost incurred in each time slot because a certain number of containers are run, thereby occupying the computing resources in the edge cloud. In traditional edge cloud computing, there are many ways to pay for container instances, including annual payment, monthly payment, and pay-as-you-go. This example is for a flexible pay-as-you-go model, that is, payment is made according to the number of time slots occupied by container instance resources. The specific definition is as follows:

[0082]

[0083] Among them, C r (t1, t2) represents the total operating cost of the network, p run The price for running a time slot for a single container instance, For edge cloud i The number of container instances running at time slot t.

[0084] The container switch cost mainly comes from the edge cloud adjusting the number of running container instances to adapt to load changes. However, the switch of container instances brings additional network overhead, and the computing resources of the network adjusted by switching the number of container instances have a certain lag. In order to avoid frequent container instance switch operations, this example introduces the container switch cost. The specific definition is as follows:

[0085]

[0086] Among them, C s (t1, t2) represents the total container switching cost in the network, p switch The cost of starting or stopping a single container instance once.

[0087] The default cost of user requests mainly comes from the total delay exceeding the standard of user requests. The reasons for the total delay exceeding the standard include the additional transmission delay of the system caused by improper diversion of user requests and the additional computing delay caused by insufficient computing resources. The total delay exceeding the standard of user requests will result in the quality of user requests failing to meet the demand. According to the service level agreement (SLA) between the service provider and the user of the network, the service provider needs to compensate the user, which leads to the default cost of user requests.

[0088]

[0089] Among them, C d (t1, t2) represents the default cost of the total user requests in the network, p delay The default value for a single user request default, is the delay satisfaction rate of user request x in time slot t, that is, the proportion of user requests that meet the delay requirements after sampling the delay multiple times in the time slot. is the agreed minimum delay satisfaction rate.

[0090] In summary, the total cost of the system within a certain period of time is:

[0091] C total (t1, t2) = C r (t1,t2)+C s (t1,t2)+C d (t1,t2)

[0092] In order to ensure the service quality of processing user requests, the average cost of each user request in each time slot is:

[0093]

[0094] in is the number of user requests in time slot t.

[0095] Secondly, the cross-domain edge network implements a method for determining the diversion information of user requests and the number of container instances in each edge cloud, that is, the core network of the cross-domain edge network performs a cost-optimal computational diversion decision algorithm.

[0096] The optimal problem of the average cost per time slot of user requests can be quantitatively described as follows: given an edge cloud e i The upper limit of the number of container instances M i , dynamically adjust the number of container instances opened in the edge cloud in each time slot And calculate the split relationship Make the average cost of each user request in each time slot the lowest. Indicates the computing split relationship. If the user request x is split to the edge cloud e j , then otherwise

[0097] In response to the above problem, the Markov decision process was used to model the problem, and then the classic A2C algorithm was used to solve it. When the Markov decision process is used to model the problem, the diversion of user requests in the network and the dynamic deployment of computing resources are mapped to a Markov decision process model. A Markov decision process model M = (S, A, P, R) contains a finite set of states S, a finite set of actions A, the state transition probability P of the network, and the immediate reward R after taking the action. In the above problem, the effect of the action on the network is deterministic, so the focus of the Markov decision process is to determine the state, action, and immediate reward.

[0098] The key points of this Markov decision process are the specific definitions of state, action and immediate reward as follows.

[0099] In each time slot, the state of the network includes: 1) the upper limit of the number of container instances that the network can start (M i ); 2) The value of the number of container instances running in the current time slot 3) The number of user requests that are in service and have not yet ended in the current time slot; 4) The number of user requests received by each edge cloud in the current time slot.

[0100] Each action a in a finite set of actions A t ∈A contains two parts: the control information of each edge cloud container instance in this time slot and the diversion information of each user request in this time slot. In this example, it is assumed that in a time slot, there are only three types of container instance operations for each edge cloud: adding a container instance, reducing a container instance, and keeping the number of container instances unchanged. Assume that for a first number (n e ) edge clouds, each time slot has at most a third number (n q ) user requests, then a t For an n e +n q Vector of bits, first n e The bits correspond to the control information of the container instances of each edge cloud, followed by n q The corresponding user request diversion information. In this example, it should be noted that due to the actual number of user requests in each time slot So the first to the second The bits are valid bits, and the rest are filled with 0 by default. In this example, the system outputs continuous action a′ t , and then discretize the output continuous action into a t The discretized action vector a t The first n eThe bit values ​​1, 0, and -1 respectively indicate that the corresponding edge cloud adds a container instance, the number of container instances remains unchanged, and a container instance is reduced. t If the remaining bits are non-zero, it means that the corresponding user request is diverted to the corresponding edge cloud, otherwise they are invalid bits. t The discretization algorithm can be:

[0101] a) will a′ t Each bit of is clamped to [-θ,θ];

[0102] b) For a′ t 1st to nth e If a′ t [i]>0.5 and Then a t [i] = 1; if Then a t [i] = -1, other cases a t [i] = 0;

[0103] c) For a′ t The nth e +1 to Bit,

[0104] d) For a′ t No. To nth e +n q bit, a t [i]=0

[0105] e) Return to a t

[0106] In this way, we get the action a of the current time slot t .

[0107] Among them, the a′ t The θ in [-θ,θ] where each bit is clamped is an empirical value. Different values ​​are taken according to different problems. In this example, 1.5 is taken. There is no restriction here.

[0108] In point b), it is explained that a t The calculation process of [i], the a t The calculation of [i] is associated with the number of containers in the current time slot and the user request;

[0109] In point c), it means that the formula is used to calculate a t [i] The obtained value is rounded and mapped to the corresponding edge cloud.

[0110] In each time slot, action a tThe immediate reward obtained after acting on the network is w t -c t ,in, represents the reward value brought by the user request whose delay satisfaction rate reaches the set delay satisfaction rate threshold in each time slot t, and w0 is the reward value when a single user request meets the set delay satisfaction threshold; c t is the total cost of the system in the current time slot. t =C total (t-1, t), wherein the total cost of the network at the current time slot includes the operating cost of the occupied computing resources from the previous time slot to the current time slot, the container instance switch cost, and the user request default cost.

[0111] In this example, based on the above problem model, the A2C algorithm is trained in multiple rounds. Based on the action results obtained in each round of training, the immediate reward is calculated, and the parameters in the deep reinforcement learning model are trained by backpropagation based on the immediate reward. After the total number of training rounds reaches the preset number of training rounds, the training process of the A2C algorithm is completed. In the cross-domain edge network, the A2C algorithm can be applied, and the results obtained can be applied to the diversion control of user requests and the adjustment of the number of container instances.

[0112] It can be seen that the embodiments of the present application take into account both the perspectives of network operators and users, and propose a scheduling method for computing offload and the number of container instances that minimizes the average user request cost per time slot based on a deep reinforcement learning algorithm when edge cloud computing resources in a cross-domain edge network are limited.

[0113] In another embodiment of the present application, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions, which, when executed by a processor, cause the processor to execute the joint control method for adjusting the number of container instances and diverting user requests in the aforementioned embodiment.

[0114] Figure 4 A schematic diagram of an electronic device provided by another embodiment of the present application. Figure 4 As shown, another embodiment of the present application further provides an electronic device, which may include a processor 401, wherein the processor 401 is used to execute the steps of the above-mentioned cross-wide area network application management and control method. Figure 4 It can also be seen that the electronic device provided in the above embodiment also includes a non-transitory computer-readable storage medium 402, on which a computer program is stored. When the computer program is executed by the processor 401, the steps of the above-mentioned joint control method for adjusting the number of container instances and user request diversion are executed.

[0115] Specifically, the non-transitory computer-readable storage medium 402 can be a general storage medium, such as a mobile disk, a hard disk, FLASH, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), or a portable compact disk read-only memory (CD-ROM), etc. When the computer program on the non-transitory computer-readable storage medium 402 is executed by the processor 401, it can trigger the processor 401 to execute the various steps of the above-mentioned joint control method for adjusting the number of container instances and diverting user requests.

[0116] In practical applications, the non-transitory computer-readable storage medium 402 may be included in the device / apparatus / system described in the above embodiment, or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, each step in the above joint control method for adjusting the number of container instances and diverting user requests can be executed.

[0117] Yet another embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements each step in the above-mentioned joint control method for adjusting the number of container instances and diverting user requests.

[0118] The flow chart and block diagram in the accompanying drawings of the present application show the possible architecture, function and operation of the system, method and computer program product according to the various embodiments disclosed in the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in the order of the standards in different figures. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in the present application.

[0120] Specific embodiments are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation methods and application scopes according to the ideas, spirits and principles of the present application, and any modifications, equivalent substitutions, improvements, etc. made by them should be included in the scope of protection of the present application.

Claims

1. A joint control method for adjusting the number of container instances and user request diversion, characterized in that: The method comprises: With the goal of minimizing the average user request cost per time slot of the cross-domain edge network, a Markov decision method is used to generate a problem model for problem modeling. The state in the problem model is set based on the number of container instances that can be started by the network, the number of container instances running in the current time slot, and the user request state information of the current time slot; the action in the problem model is set based on the container instance control information of each edge cloud in the current time slot and the diversion information of the user request in the current time slot; the instant reward in the problem model is set based on the strategy of minimizing the cost of each user request in the current time slot; Using a deep reinforcement learning model to solve the problem model to train the deep reinforcement learning model; Input the number of container instances that can currently be started in the network, the number of container instances running in the current time slot, and the user request status information of the current time slot into the trained deep reinforcement learning model, and output the increase or decrease information of the number of container instances required in the current time slot and the user request diversion information of the current time slot; Among them, the user request status information includes: the number of user requests that are in service status and have not yet ended in the current time slot; and the number of user requests received by each edge cloud in the current time slot; the container instance control information includes: the container instance control information of each edge cloud includes adding a container instance information, reducing a container instance information and the container instance remains unchanged information.

2. The method according to claim 1, characterized in that The problem is to minimize the average user request cost per time slot of the network, including: In the network, an upper limit value of the number of container instances in each edge cloud is set, and the number of container instances in the edge cloud is dynamically adjusted and user request diversion information is calculated in each time slot, so that the average cost for each user request in each time slot is minimized.

3. The method according to claim 1, characterized in that The state in the problem model is set based on the number of container instances that can be started by the network, the number of container instances running in the current time slot, and the user request state information of the current time slot, including: The upper limit M of the number of container instances that the network can start i ; The number of container instances running in the current time slot The number of user requests that are in service and have not yet been completed in the current time slot; and the number of user requests received by each edge cloud in the current time slot.

4. The method according to claim 1, characterized in that The actions in the problem model include: setting the container instance control information of each edge cloud in the current time slot and the diversion information requested by the user in the current time slot: Each action a in the action space A in the problem model i Contains the container instance control information of each edge cloud in the current time slot and the diversion information of the user request in the current time slot; For a first number n e edge clouds and correspondingly have a set second number n q The network requested by a user, action a i Expressed as the first number n e The container instance control information of the edge cloud and the second number n q A vector of diversion information requested by a user; The action a i The implementation is: multiple consecutive actions a′ i After discretization, the discretized action a is obtained i , where the discretized action a i The first number n in e The vector represents the capacity instance control information corresponding to each edge cloud, and the second number n q A vector represents the information of whether the corresponding user request is diverted to the edge cloud.

5. The method according to claim 1, characterized in that The instant reward is based on the strategy setting with the lowest cost for each user request in the current time slot, including: The immediate reward is defined as: w t -c t ,in, represents the reward value brought by the user request whose delay satisfaction rate reaches the set delay satisfaction rate threshold in the current time slot t, and w0 is the reward value when a single user request meets the set delay satisfaction threshold; c t is the total cost of the network at the current time slot, including the operating cost of the occupied computing resources from the previous time slot to the current time slot, the container instance switch cost, and the user request default cost; is the number of user requests in time slot t; is the delay satisfaction rate of user request x in time slot t; in, is the agreed minimum delay satisfaction rate.

6. The method according to claim 1 or 2, characterized in that: The deep reinforcement learning model is implemented using the dominant action-critic algorithm A2C.

7. The method according to claim 1 or 2, characterized in that: The method further comprises: In the next time slot, the core network of the network increases or decreases the number of corresponding container instances in each edge cloud in the network according to the increase or decrease information of the number of container instances required in the current time slot; In the next time slot, the core network of the network establishes a protocol data unit session to the corresponding user, and diverts the user request to the corresponding edge cloud according to the user request diversion information of the current time slot.

8. A joint control device for adjusting the number of container instances and user request diversion, characterized in that: include: Problem modeling unit, model training unit and model application unit, among which, A problem modeling unit is used to minimize the average user request cost per time slot of the cross-domain edge network, and use the Markov decision method to perform problem modeling to generate a problem model. The state in the problem model is set based on the number of container instances that can be started by the network, the number of container instances running in the current time slot, and the user request state information of the current time slot; the action in the problem model is set based on the container instance control information of each edge cloud in the current time slot and the diversion information of the user request in the current time slot; the instant reward in the problem model is set based on the strategy of the lowest cost per user request in the current time slot; wherein the user request state information includes: the number of user requests that are in service and have not yet ended in the current time slot; and the number of user requests received by each edge cloud in the current time slot; the container instance control information includes: the container instance control information of each edge cloud includes information on adding a container instance, reducing a container instance, and information on keeping the container instance unchanged; The model training unit is used to solve the problem model using a deep reinforcement learning model to train the deep reinforcement learning model until the immediate reward in the problem model reaches a set evaluation value; The model application unit is used to input the number of container instances that can currently be started in the network, the number of container instances running in the current time slot, and the user request status information of the current time slot into the trained deep reinforcement learning model, and output the increase or decrease information of the number of container instances required for the current time slot and the user request diversion information of the current time slot.

9. The device according to claim 8, characterized in that The device also includes: A control unit is used to increase or decrease the number of corresponding container instances in each edge cloud in the network in the next time slot according to the increase or decrease information of the number of container instances required in the current time slot by the core network of the network; in the next time slot, the core network of the network establishes a protocol data unit session to the corresponding user, and according to the user request diversion information of the current time slot, diverts the user request to the corresponding edge cloud.

10. An electronic device, characterized in that: include: processor; A memory stores a program, wherein the program is configured to implement the joint control method for adjusting the number of container instances and diverting user requests as described in any one of claims 1 to 7 when executed by the processor.

Images