Container scheduling network construction method based on reinforcement learning, edge server container scheduling method and system
By building a container scheduling network in an edge server and collaborative training using reinforcement learning and attention mechanisms, the communication overhead problem of container scheduling in heterogeneous edge networks is solved, and efficient container scheduling and fast adaptation are achieved.
Patent Information
- Application Number
- CN202510554681.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-28
- Publication Date
- 2025-08-08
AI Technical Summary
The existing deep learning-based container scheduling methods need to be retrained in heterogeneous edge networks, resulting in large communication overhead, unable to directly deploy on edge servers, and unable to achieve efficient container scheduling.
Set up a status monitor in N edge servers to build a container scheduling network, including the Actor network layer and the Critic network layer, container scheduling is performed through reinforcement learning, and information from other edge servers is aggregated through attention mechanism for collaborative training to optimize decision accuracy.
Through multi-agent collaboration, it significantly reduces communication overhead, improves decision-making accuracy, supports rapid start-stop containers and microservice architectures, and realizes lightweight container adaptation.
Smart Images
Figure CN120455458A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of edge server container scheduling, and in particular to a container scheduling network construction method based on reinforcement learning, an edge server container scheduling method and system. Background Art
[0002] In edge computing scenarios, the containerized deployment of edge servers on edge devices for applications such as real-time data analysis and smart terminal services is currently a rapidly developing direction.
[0003] Edge servers are equipped with deep learning-based intelligent agents that learn optimal policies through interaction with the environment. In container scheduling, reinforcement learning can select the optimal scheduling decision based on the current system state (such as resource utilization and task queue length), thereby minimizing task execution time and resource waste.
[0004] Existing deep learning-based container scheduling methods have shortcomings in the edge computing field. Most existing deep learning-based container scheduling methods are deployed on X86-based servers, while edge device architectures in IoT edge networks range from ARM to MIPS. Different edge LANs exhibit significant heterogeneity.
[0005] Therefore, the existing deep learning-based container scheduling method deployed in the edge server requires retraining the container scheduling strategy. However, the computing power of the edge server is obviously unable to complete the retraining process. It needs to be trained in the cloud and then deployed, resulting in large communication overhead.
[0006] In order to enable collaborative training of edge servers and other edge networks with the same architecture, the present application provides a container scheduling network construction method based on reinforcement learning, an edge server container scheduling method and system. Summary of the Invention
[0007] To overcome the problems existing in the related art, the present application provides a first aspect of a method for constructing a container scheduling network based on reinforcement learning, comprising:
[0008] A status monitor is set in each of N edge servers, where N is an integer equal to 2;
[0009] Constructing a container scheduling network in each of the N edge servers, wherein the container scheduling network includes an Actor network layer and a Critic network layer;
[0010] Performing container scheduling in N edge servers through the container scheduling network;
[0011] Collecting an experience data set from the N edge servers through the status monitor, wherein the experience data set includes a decision-value data set of the N edge servers;
[0012] The container scheduling network in the N edge servers is trained respectively using the experience data set.
[0013] In one embodiment, performing container scheduling in N edge servers through the container scheduling network specifically includes:
[0014] Obtain task requests from edge terminal devices;
[0015] Determine task queue information according to the task request;
[0016] Determine container scheduling decisions based on the task queue information through the Actor network layer;
[0017] The action value of the container scheduling decision is determined by the Critic network layer.
[0018] In one embodiment, collecting the experience data set in N edge servers by the state monitor specifically includes:
[0019] encoding the container scheduling decision and the action value into a decision-value data set;
[0020] The decision-value data set is stored as the experience data set.
[0021] In one embodiment, determining the action value of the container scheduling decision through the critic network layer specifically includes:
[0022] Obtaining the current container scheduling decision;
[0023] The action value of the current container scheduling decision is determined according to the critic value function; the critic value function is:
[0024] Q i (O, A) = f i (g i (o i , a i ), x i )
[0025] Among them, Q i (O, A) is the action value, g i (o i , a i ) is the current decision-value data set, x i To aggregate information from other edge servers through the attention mechanism.
[0026] In one embodiment, the critic value function further includes:
[0027]
[0028] Among them, h j (o j ,a j ) is the state observation feature of other edge servers j, α ij is the attention weight.
[0029] In one embodiment, the critic value function further includes:
[0030]
[0031] Among them, e ij is the correlation coefficient between edge servers i and j.
[0032] In one embodiment, the critic value function specifically includes:
[0033] e ij =LeakyReLU(W q g i (o i , a i )+W k □ j (o j , a j ))
[0034] Among them, W q and W k is the weight matrix.
[0035] A second aspect of the present application provides an edge server container scheduling method based on reinforcement learning, which determines a container scheduling strategy based on the constructed container scheduling network, including the following steps:
[0036] Obtain task requests from M edge terminal devices; M is an integer greater than or equal to 1;
[0037] determining a task queue according to the task request;
[0038] Get the state vector of the edge server;
[0039] Determining a container scheduling strategy according to the state vector and the task queue through the container scheduling network;
[0040] Execute container scheduling actions according to the container scheduling policy.
[0041] A third aspect of the present application provides an edge server container scheduling system based on reinforcement learning, which performs container scheduling based on the steps in the edge server container scheduling method described in the third aspect of the present application, including N edge servers with communication connections;
[0042] N edge servers are each equipped with a status monitor and a container scheduling network;
[0043] The state monitor is used to collect the experience data set;
[0044] The container scheduling network is used to determine the container scheduling strategy, and the container scheduling network performs reinforcement learning through the experience data set.
[0045] The technical solution provided by this application may have the following beneficial effects:
[0046] (1) Multi-agent collaboration: Compared with traditional single-agent scheduling methods, this application significantly reduces communication overhead through multi-agent collaboration;
[0047] (2) Dynamic attention mechanism: An edge server in the network selectively focuses on the experience datasets from other edge servers through the attention mechanism for collaborative training, that is, adaptively adjusts the information aggregation weight according to the node resource correlation to improve decision accuracy;
[0048] (3) Lightweight container adaptation: Optimize action space design to support rapid container start and stop and microservice architecture.
[0049] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.
[0051] Figure 1 This is a flow chart of a method for constructing a container scheduling network as shown in an embodiment of the present application;
[0052] Figure 2 This is a flow chart of the container scheduling method shown in an embodiment of the present application;
[0053] Figure 3 This is a data flow diagram of an edge server shown in an embodiment of the present application;
[0054] Figure 4 This is a schematic diagram of the network structure of the container scheduling system shown in an embodiment of the present application;
[0055] Figure 5 This is a schematic diagram of container scheduling for an edge server shown in an embodiment of the present application. DETAILED DESCRIPTION
[0056] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.
[0057] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
[0058] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0059] Example 1
[0060] A container scheduling network is deployed on each of N edge servers. Each edge server is connected to at least one edge terminal device, which serves as an edge node. In an edge network, the edge server pushes Docker images to the edge node and starts the container according to the container scheduling policy.
[0061] The edge server of the embodiment of the present application generates a container scheduling policy through a container scheduling network. The container scheduling network performs reinforcement learning based on data from N edge servers.
[0062] In order to enable the constructed container scheduling network to combine multiple edge servers for reinforcement learning, the embodiment of the present application provides a container scheduling network construction method based on reinforcement learning, such as Figure 1 As shown, the following steps are included:
[0063] S1. Set up status monitors in each of the N edge servers.
[0064] S2. Build a container scheduling network in each of the N edge servers.
[0065] S3. Schedule containers in N edge servers through the container scheduling network;
[0066] S4. Collecting an experience data set in N edge servers through the status monitor;
[0067] S5. Train the container scheduling network in the N edge servers respectively using the experience data set.
[0068] In steps S1 to S3, the data flow structure of the edge server is as follows: Figure 3 The empirical data set includes a decision-value data set of N edge servers.
[0069] It can be understood that N is an integer equal to 2.
[0070] Specifically, the container scheduling network includes an actor network layer and a critic network layer. The actor network is used to output scheduling action strategies, and the critic network is used to evaluate the value of actions generated by the actor network.
[0071] Furthermore, S3 specifically includes:
[0072] S301, obtaining a task request from an edge terminal device;
[0073] S302, determining task queue information according to the task request;
[0074] S303: Determine a container scheduling decision based on the task queue information through the Actor network layer;
[0075] S304: Determine the action value of the container scheduling decision through the Critic network layer.
[0076] Specifically, S303 includes:
[0077] S3031. Obtain the current container scheduling decision;
[0078] S3032: Determine the action value of the current container scheduling decision according to the Critic value function.
[0079] Specifically, the critic value function is:
[0080] Q i (O, A) = f i (gi (o i , a i ), x i )
[0081] Among them, Q i (O, A) is the action value, g i (o i , a i ) is the current decision-value data set, x i To aggregate information from other edge servers through the attention mechanism.
[0082]
[0083] Among them, h j (o j ,a j ) is the state observation feature of other edge servers j, α ij is the attention weight.
[0084]
[0085] Among them, e ij is the correlation coefficient between edge servers i and j.
[0086] e ij =LeakyReLU(W q g i (o i , a i )+W k h j (o j , a j ))
[0087] Among them, W q and W k is the weight matrix.
[0088] In an embodiment of the present application, when evaluating the value of an action, the critic value function aggregates information from other edge servers through an attention mechanism and performs a weighted sum of the state observation features of N edge servers.
[0089] The role of attention weight is mainly to query the observation values of other edge servers during the training process of each edge server. j and action a jThis information is incorporated into the estimated critic value function, helping edge servers selectively filter out important environmental information and ignore unimportant information. The target network predicts the Q value of the container's action and selects one Q value as the output. A loss function is constructed based on the Q values predicted by the main network and the target network, and the main network parameters are optimized based on the loss function.
[0090] Furthermore, S4 specifically includes:
[0091] S401, encoding the container scheduling decision and the action value into a decision-value data set;
[0092] S402: Store the decision-value data set as the experience data set.
[0093] In an embodiment of the present application, a status monitor within an edge server collects information from the task request queue, the container's operating load, and container resource requirements. The status monitor within the edge server collects information from the task request queue, the container's operating load, and container resource requirements as an environmental observation. The monitor runs on each container and periodically publishes container resource usage information to a message broker, recording the operating status and performance statistics of application data in a unified format.
[0094] In S5, the edge server encodes the states received from other edge servers and its own state into local observations, converting them into high-dimensional feature vectors. A batch of samples are randomly drawn from the experience replay pool to train the actor and critic networks until convergence. Finally, the actor network is used to generate the optimal action strategy.
[0095] Example 2
[0096] Based on the container scheduling network constructed in Example 1, this embodiment of the application provides an edge server container scheduling method based on reinforcement learning, such as Figure 2 As shown, the following steps are included:
[0097] S100: Obtain task requests from M edge terminal devices;
[0098] S200, determining a task queue according to the task request;
[0099] S300: Obtain a state vector of the edge server;
[0100] S400, determining a container scheduling strategy according to the state vector and the task queue through the container scheduling network;
[0101] S500: Execute a container scheduling action according to the container scheduling policy.
[0102] It is understood that N is an integer greater than or equal to 1;
[0103] After the container scheduling network is trained, the edge server outputs the probability distribution of each action through the actor network in the container scheduling network. It selects the action with the highest probability as the final decision and sends it to the container execution module. After receiving the action decision, the container scheduling module performs the corresponding operation based on the action type, including deploying the container to an edge node, migrating the container from the current node to another node, adjusting the container's resource quota, and terminating or restarting the container.
[0104] After the network deployment is completed, the edge server takes the corresponding environment observation state Ot, action decision A and reward R, and the environment state Ot+1 at the next moment as four-element sample data, and stores the sample data as an experience dataset.
[0105] Example 3
[0106] Based on the construction method shown in Example 1 and the scheduling method shown in Example 2, the embodiment of the present application provides an edge server container scheduling system based on reinforcement learning.
[0107] The system is as Figure 4 As shown, it includes N edge servers with communication connections.
[0108] Among them, Figure 5 As shown, each of the N edge servers is provided with a state monitor and a container scheduling network. The state monitor is used to collect the experience data set; the container scheduling network is used to determine the container scheduling strategy, and the container scheduling network performs reinforcement learning based on the experience data set.
[0109] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.
[0110] The solution of the present application has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own focus. For parts not described in detail in a particular embodiment, please refer to the relevant description of other embodiments. Those skilled in the art should also be aware that the actions and modules mentioned in the description are not necessarily required for this application.
[0111] In addition, it can be understood that the steps in the method of the embodiment of the present application can be adjusted in order, merged and deleted according to actual needs, and the modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.
[0112] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.
[0113] Alternatively, the present application can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.
[0114] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the application herein may be implemented as electronic hardware, computer software, or combinations of both.
[0115] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems and methods according to multiple embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0116] The embodiments of the present application have been described above. The above description is illustrative and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.
Claims
1. A container scheduling network construction method based on reinforcement learning, characterized in that: include: A status monitor is set in each of N edge servers, where N is an integer equal to 2; Constructing a container scheduling network in each of the N edge servers, wherein the container scheduling network includes an Actor network layer and a Critic network layer; Performing container scheduling in N edge servers through the container scheduling network; Collecting an experience data set from the N edge servers through the status monitor, wherein the experience data set includes a decision-value data set of the N edge servers; The container scheduling network in the N edge servers is trained respectively using the experience data set.
2. The method for constructing a container scheduling network based on reinforcement learning according to claim 1, characterized in that: Container scheduling is performed in N edge servers through the container scheduling network, specifically including: Obtain task requests from edge terminal devices; Determine task queue information according to the task request; Determine container scheduling decisions based on the task queue information through the Actor network layer; The action value of the container scheduling decision is determined by the Critic network layer.
3. The method for constructing a container scheduling network based on reinforcement learning according to claim 2, characterized in that: Collecting an experience data set from N edge servers through the state monitor specifically includes: encoding the container scheduling decision and the action value into a decision-value data set; The decision-value data set is stored as the experience data set.
4. The method for constructing a container scheduling network based on reinforcement learning according to claim 2, characterized in that: The action value of the container scheduling decision is determined by the Critic network layer, specifically including: Obtaining the current container scheduling decision; The action value of the current container scheduling decision is determined according to the critic value function; the critic value function is: Q i (O,A)=f i (g i (o i ,a i ),x i ) Among them, Q i (O, A) is the action value, g i (o i , a i ) is the current decision-value data set, x i To aggregate information from other edge servers through the attention mechanism.
5. The method for constructing a container scheduling network based on reinforcement learning according to claim 4, characterized in that: The Critic value function also includes: Among them, h j (o j ,a j ) is the state observation feature of other edge servers j, α ij is the attention weight.
6. The method for constructing a container scheduling network based on reinforcement learning according to claim 5, characterized in that: The Critic value function also includes: Among them, e ij is the correlation coefficient between edge servers i and j.
7. The method for constructing a container scheduling network based on reinforcement learning according to claim 6, characterized in that: The critic value function specifically includes: e ij =LeakyReLU(W q g i (about i ,and i )+W k h j (about j ,and j )) Among them, W q and W k is the weight matrix.
8. A method for scheduling edge server containers based on reinforcement learning, characterized in that: Determining a container scheduling strategy based on the container scheduling network constructed according to any one of claims 1 to 7 comprises the following steps: Obtain task requests from M edge terminal devices; M is an integer greater than or equal to 1; determining a task queue according to the task request; Get the state vector of the edge server; Determining a container scheduling strategy according to the state vector and the task queue through the container scheduling network; Execute container scheduling actions according to the container scheduling policy.
9. An edge server container scheduling system based on reinforcement learning, characterized in that: Performing container scheduling based on the steps in the edge server container scheduling method of claim 8, including N edge servers with communication connections; N edge servers are each equipped with a status monitor and a container scheduling network; The state monitor is used to collect the experience data set; The container scheduling network is used to determine the container scheduling strategy, and the container scheduling network performs reinforcement learning through the experience data set.