Satellite network micro-service deployment method and device based on multi-agent reinforcement learning

Abstract

The embodiment of the application provides a satellite network micro-service deployment method and device based on multi-agent reinforcement learning, which comprises the following steps: obtaining resource requirement information of a micro-service; determining resource utilization rate information and time delay information of a satellite node according to a pre-established resource utilization rate model and time delay model and configuration information of the satellite node; in the case that the resource utilization rate information is less than a first preset value or the time delay information is less than a second preset value, a pre-trained multi-agent strategy deployment model is used to determine a deployment strategy of the satellite node corresponding to the resource requirement information of the micro-service, and the server terminal is configured according to the deployment strategy of the satellite node, so that the resource configuration of each satellite node can be performed according to the resource requirement of the server of the micro-service and the resource surplus of each satellite node, the resource utilization balance of the satellite node is improved, the calling time delay is reduced, and the configuration efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for deploying satellite network microservices based on multi-agent reinforcement learning. Background Technology

[0002] As software architecture and operation methods continue to evolve, traditional centralized architectures, due to their lack of flexibility, difficulty in expansion and migration, cannot meet the needs of various applications, and have gradually evolved into distributed microservice architectures. Microservice architecture breaks down complex applications into several relatively independent smaller applications according to their logical relationships. These microservices can be developed, updated, expanded, and deployed independently without affecting each other, using lightweight protocols for communication, and can be deployed to different satellite edge nodes. Based on microservice architecture, engineering projects have higher scalability, reliability, and flexible distributed deployment capabilities.

[0003] Currently, with the ever-increasing demand for communication services, the shortcomings of terrestrial communication networks, such as insufficient spectrum resources and small coverage area, have become apparent. Compared with terrestrial communication, satellite communication has unique advantages, such as wide coverage, high system reliability, large communication capacity, and immunity to natural disasters such as earthquakes. However, each satellite has different resource availability, different microservers have different resource requests, and the remaining resource availability of each satellite is also different. How to deploy different microservers on various satellites to achieve reasonable allocation of satellite resources is an urgent problem to be solved. Summary of the Invention

[0004] The purpose of some embodiments of this application is to provide a method and apparatus for deploying microservices in a satellite network based on multi-agent reinforcement learning. Through the technical solutions of the embodiments of this application, resource requirement information of microservices is obtained; resource utilization information and latency information of satellite nodes are determined based on pre-established resource utilization and latency models, as well as configuration information of satellite nodes; when the resource utilization information is less than a first preset value, or the latency information is less than a second preset value, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite nodes corresponding to the resource requirement information of the microservices. The pre-trained multi-agent strategy deployment model is developed using a multi-agent deep policy gradient algorithm to deploy the microservice network. The parameters of the network model are obtained after training. The server terminal is configured according to the deployment strategy of the satellite nodes. In this embodiment, a resource utilization model and a latency model are established based on a microservice architecture. The resource utilization information and latency information of the satellite nodes are determined according to the configuration information of the satellite nodes. Then, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite nodes corresponding to the resource demand information of the microservice. The server terminal is configured according to the deployment strategy of the satellite nodes. In this way, the resources of each satellite node can be configured according to the resource demand of the microservice on the server and the remaining resources of each satellite node, thereby improving the resource utilization balance of the satellite nodes, reducing the call latency, and improving the configuration efficiency.

[0005] Firstly, some embodiments of this application provide a method for deploying satellite network microservices based on multi-agent reinforcement learning, including:

[0006] Obtain resource requirements information for microservices;

[0007] Based on the pre-established resource utilization model and latency model, as well as the configuration information of the satellite nodes, the resource utilization information and latency information of the satellite nodes are determined.

[0008] When the resource utilization information is less than a first preset value, or the latency information is less than a second preset value, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice. The pre-trained multi-agent strategy deployment model is obtained by training each parameter of the agent network model using a multi-agent deep deterministic strategy gradient algorithm.

[0009] Configure the server terminal according to the deployment strategy of the satellite nodes.

[0010] Some embodiments of this application establish resource utilization and latency models. Based on the configuration information of satellite nodes, they determine the resource utilization and latency information of satellite nodes. Then, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of satellite nodes corresponding to the resource requirements of the microservices. The server terminal is configured according to the deployment strategy of the satellite nodes. In this way, resource configuration of each satellite node can be performed according to the resource requirements of the microservices and the remaining resources of each satellite node. That is, microservices with different resource requirements are deployed to appropriate satellite nodes, improving the resource utilization balance of satellite nodes, reducing call latency, and improving configuration efficiency.

[0011] Optionally, the multi-agent policy deployment model is obtained in the following manner:

[0012] Obtain agent sample parameters, wherein the agent sample parameters include at least the agent's observation environment:

[0013] Obtain an intelligent network model, wherein the intelligent network model comprises at least an actor network model and a critic network model;

[0014] The agent's observation environment is input into the actor network model, and the agent's deployment action is output.

[0015] The deployment actions and global state of the agent are input into the critic network model, and action evaluation values ​​are output.

[0016] A replay pool is established based on the agent's current state information, action information, reward information, and state information at the next moment;

[0017] A multi-agent deep deterministic policy gradient algorithm is used to update the network parameters in the actor network model and the critic network model based on the current state information, action information, reward information and the state information at the next moment obtained from the replay pool.

[0018] If the actor network model and the critic network model converge, the converged actor network model and the critic network model are identified as the multi-agent policy deployment model.

[0019] Some embodiments of this application transform the microservice deployment problem into a partially observable Markov decision process and solve it using a multi-agent reinforcement learning method. The method employs centralized training and distributed execution. During the training phase, the container instances of the microservices, acting as agents, need to acquire global information to obtain the optimal deployment scheme. During the execution phase, the microservices can complete the deployment using only their own observation space, which greatly reduces the communication overhead between microservices.

[0020] Optionally, updating the network parameters in the actor network model and the critic network model includes:

[0021] Obtain the first loss function of the actor network model and the second loss function of the critic network model;

[0022] Gradient calculations are performed on the first loss function and the second loss function, respectively.

[0023] The network parameters in the aforementioned actor network model and the aforementioned critic network model are updated using the gradient descent method.

[0024] Some embodiments of this application employ a fixed network method, which fixes the target network and transmits the original network parameters to the target network at regular intervals to avoid continuous changes in the updated target and ensure the stability of training.

[0025] Optionally, the configuration information of the satellite nodes includes at least the number of satellite nodes, the total number of resource types, and the capacity of heterogeneous resources.

[0026] Optionally, the resource utilization model is obtained in the following manner:

[0027] Obtain resource balance information of the first resource utilization model for different types of resources on the same satellite node, and node balance information of the second resource utilization model for the same type of resources on different satellite nodes;

[0028] The resource utilization model is determined based on the resource balance information and the weight value corresponding to the resource balance information, as well as the node balance information and the weight value corresponding to the node balance information.

[0029] Optionally, the delay model includes at least a transmission delay sub-model, a propagation delay sub-model, and a migration delay sub-model.

[0030] Some embodiments of this application establish resource utilization models and latency models, minimize resource utilization variance and latency, and represent the microservice deployment problem as a multi-objective optimization problem.

[0031] Secondly, some embodiments of this application provide a satellite network microservice deployment apparatus based on multi-agent reinforcement learning, comprising:

[0032] The acquisition module is used to acquire resource requirement information for microservices;

[0033] The first determining module is used to determine the resource utilization information and latency information of satellite nodes based on the pre-established resource utilization model and latency model, as well as the configuration information of satellite nodes.

[0034] The second determining module is used to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice by using a pre-trained multi-agent strategy deployment model when the resource utilization information is less than a first preset value or the latency information is less than a second preset value. The pre-trained multi-agent strategy deployment model is obtained by training each parameter of the agent network model using a multi-agent deep determination strategy gradient algorithm.

[0035] The configuration module is used to configure the server terminal according to the deployment strategy of the satellite nodes.

[0036] Some embodiments of this application establish resource utilization and latency models based on a microservice architecture. Based on the configuration information of satellite nodes, the resource utilization and latency information of each satellite node are determined. Then, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite nodes corresponding to the resource requirements of the microservices. The server terminal is configured according to this deployment strategy. In this way, resource configuration can be performed on each satellite node based on the server resource requirements of the microservices and the remaining resources of each satellite node. This means that microservices with different resource requirements are deployed to appropriate satellite nodes, improving the resource utilization balance of satellite nodes, reducing call latency, and improving configuration efficiency.

[0037] Optionally, the apparatus further includes a model training module, the model training module being used for:

[0038] Obtain agent sample parameters, wherein the agent sample parameters include at least the agent's observation environment:

[0039] Obtain an intelligent network model, wherein the intelligent network model comprises at least an actor network model and a critic network model;

[0040] The agent's observation environment is input into the actor network model, and the agent's deployment action is output.

[0041] The deployment actions and global state of the agent are input into the critic network model, and action evaluation values ​​are output.

[0042] A replay pool is established based on the agent's current state information, action information, reward information, and state information at the next moment;

[0043] A multi-agent deep deterministic policy gradient algorithm is used to update the network parameters in the actor network model and the critic network model based on the current state information, action information, reward information and the state information at the next moment obtained from the replay pool.

[0044] If the actor network model and the critic network model converge, the converged actor network model and the critic network model are identified as the multi-agent policy deployment model.

[0045] Some embodiments of this application transform the microservice deployment problem into a partially observable Markov decision process and solve it using a multi-agent reinforcement learning method. The method employs centralized training and distributed execution. During the training phase, the container instances of the microservices, acting as agents, need to acquire global information to obtain the optimal deployment scheme. During the execution phase, the microservices can complete the deployment using only their own observation space, which greatly reduces the communication overhead between microservices.

[0046] Optionally, the model training module is used for:

[0047] Obtain the first loss function of the actor network model and the second loss function of the critic network model;

[0048] Gradient calculations are performed on the first loss function and the second loss function, respectively.

[0049] The network parameters in the aforementioned actor network model and the aforementioned critic network model are updated using the gradient descent method.

[0050] Some embodiments of this application employ a fixed network method, which fixes the target network and transmits the original network parameters to the target network at regular intervals to avoid continuous changes in the updated target and ensure the stability of training.

[0051] Optionally, the configuration information of the satellite nodes includes at least the number of satellite nodes, the total number of resource types, and the capacity of heterogeneous resources.

[0052] Optionally, the model training module is used for:

[0053] Obtain resource balance information of the first resource utilization model for different types of resources on the same satellite node, and node balance information of the second resource utilization model for the same type of resources on different satellite nodes;

[0054] The resource utilization model is determined based on the resource balance information and the weight value corresponding to the resource balance information, as well as the node balance information and the weight value corresponding to the node balance information.

[0055] Optionally, the delay model includes at least a transmission delay sub-model, a propagation delay sub-model, and a migration delay sub-model.

[0056] Some embodiments of this application establish resource utilization models and latency models, minimize resource utilization variance and latency, and represent the microservice deployment problem as a multi-objective optimization problem.

[0057] Thirdly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can implement the satellite network microservice deployment method based on multi-agent reinforcement learning as described in any embodiment of the first aspect.

[0058] Fourthly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the satellite network microservice deployment method based on multi-agent reinforcement learning as described in any embodiment of the first aspect.

[0059] Fifthly, some embodiments of this application provide a computer program product, the computer program product including a computer program, wherein, when the computer program is executed by a processor, it can implement the satellite network microservice deployment method based on multi-agent reinforcement learning as described in any embodiment of the first aspect. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A flowchart illustrating a satellite network microservice deployment method based on multi-agent reinforcement learning, provided as an embodiment of this application;

[0062] Figure 2 A flowchart illustrating another satellite network microservice deployment method based on multi-agent reinforcement learning provided in this application embodiment;

[0063] Figure 3 This is a schematic diagram of a microservice deployment scenario provided in an embodiment of this application;

[0064] Figure 4 The network structure diagram for model training provided in the embodiments of this application;

[0065] Figure 5 A schematic diagram illustrating the model training process provided in this application embodiment;

[0066] Figure 6 A schematic diagram of the structure of a satellite network microservice deployment device based on multi-agent reinforcement learning, provided in an embodiment of this application;

[0067] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0068] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0069] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0070] As software architecture and operation methods continue to evolve, traditional centralized architectures, due to their lack of flexibility, difficulty in expansion and migration, cannot meet the needs of various applications, and have gradually evolved into distributed microservice architectures. Microservice architecture breaks down complex applications into several relatively independent smaller applications according to their logical relationships. These microservices can be developed, updated, expanded, and deployed independently without affecting each other, using lightweight protocols for communication, and can be deployed to different satellite edge nodes. Based on microservice architecture, engineering projects have higher scalability, reliability, and flexible distributed deployment capabilities.

[0071] Currently, with the increasing demand for communication services, the shortcomings of terrestrial communication networks, such as insufficient spectrum resources and small coverage area, have become apparent. Compared with terrestrial communication, satellite communication has unique advantages, such as wide coverage, high system reliability, large communication capacity, and immunity to natural disasters such as earthquakes. However, each satellite has different resource availability, different microservers have different resource requests, and the remaining resource availability of each satellite is also different. In view of this, some embodiments of this application provide a satellite network microservice deployment method based on multi-agent reinforcement learning. This method includes obtaining resource requirement information of microservices; determining resource utilization information and latency information of satellite nodes based on pre-established resource utilization models and latency models, as well as the configuration information of satellite nodes; and when the resource utilization information is less than a first preset value, or the latency information is less than a second preset value, using a pre-trained multi-agent strategy deployment model to determine the deployment strategy of satellite nodes corresponding to the resource requirement information of the microservices. The multi-agent strategy deployment model is obtained by training the various parameters of the agent network model using a multi-agent deep deterministic policy gradient algorithm. The server terminal is configured according to the deployment strategy of the satellite nodes. In this embodiment, a resource utilization model and a latency model are established based on a microservice architecture. Based on the configuration information of the satellite nodes, the resource utilization information and latency information of the satellite nodes are determined. Then, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite nodes corresponding to the resource requirements of the microservices. The server terminal is configured according to the deployment strategy of the satellite nodes. In this way, resource configuration can be performed on each satellite node according to the server resource requirements of the microservices and the remaining resources of each satellite node. That is, microservices with different resource requirements are deployed to appropriate satellite nodes, improving the resource utilization balance of satellite nodes, reducing call latency, and improving configuration efficiency.

[0072] like Figure 1 As shown, embodiments of this application provide a method for deploying satellite network microservices based on multi-agent reinforcement learning, the method comprising:

[0073] S101. Obtain resource requirement information for microservices;

[0074] Server terminals are used to execute microservices. Each microservice corresponds to at least one container instance. Container technology is a lightweight resource virtualization technology that abstracts, transforms, and partitions computing resources to present one or more computing resources. Among them, Docker is currently the most popular container technology, widely used in microservice deployments and cloud computing platforms.

[0075] The scheduling platform obtains resource requirement information for each microservice, such as which microservice needs to obtain the resources of which satellite node, such as CPU, memory, and disk I / O.

[0076] S102. Based on the pre-established resource utilization model and latency model, as well as the configuration information of the satellite nodes, determine the resource utilization information and latency information of the satellite nodes.

[0077] The configuration information of satellite nodes includes at least the number of satellite nodes, the total amount of resource types, and the capacity of heterogeneous resources.

[0078] Specifically, the scheduling platform pre-establishes resource utilization and latency models. The resource utilization model is represented by variance and is divided into two categories. One category is the variance of different types of resources on the same node, to prevent an overabundance of one type of resource, which could lead to a bottleneck effect and resource waste. The other category is the variance of the same type of resource on different nodes, to prevent satellite node resources from being idle.

[0079] Latency is categorized into transmission latency, propagation latency, and migration latency. Transmission latency can be expressed as the quotient of data size and transmission rate, which can be calculated using Shannon's formula. Propagation latency is directly proportional to the physical distance between nodes. Migration latency is determined by the migration frequency of microservices.

[0080] The scheduling platform determines the resource utilization and latency information of satellite nodes based on pre-established resource utilization and latency models, as well as the configuration information of satellite nodes.

[0081] S103. When the resource utilization information is less than the first preset value or the latency information is less than the second preset value, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice. The pre-trained multi-agent strategy deployment model is obtained by training each parameter of the agent network model using the multi-agent deep determination strategy gradient algorithm.

[0082] Specifically, the scheduling platform pre-acquires global state information, including the resource occupancy of satellite nodes, satellite location information, and container deployment status. As satellite positions change over time, agents take corresponding actions based on these changes, leading to state changes. To minimize resource utilization variance and latency, a reward function is established. Based on the above information, an agent network model is built. Then, the Multi-agent Deep Deterministic Policy Gradient (MADDPG) algorithm is used to train the various parameters of the agent network model, resulting in a multi-agent policy deployment model.

[0083] If the resource utilization information is less than the first preset value, or the latency information is less than the second preset value, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of satellite nodes corresponding to the resource demand information of the microservice.

[0084] S104. Configure the server terminal according to the satellite node deployment strategy.

[0085] Specifically, the scheduling platform configures the deployment strategy of each satellite node for each microservice. In this way, the satellite nodes execute the deployment strategy, which improves the resource utilization balance of the satellite nodes, reduces call latency, and improves configuration efficiency.

[0086] Some embodiments of this application establish resource utilization and latency models based on a microservice architecture. Based on the configuration information of satellite nodes, the resource utilization and latency information of each satellite node are determined. Then, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite nodes corresponding to the resource requirements of the microservices. The server terminal is configured according to this deployment strategy. In this way, resource configuration can be performed on each satellite node based on the server resource requirements of the microservices and the remaining resources of each satellite node. This means that microservices with different resource requirements are deployed to appropriate satellite nodes, improving the resource utilization balance of satellite nodes, reducing call latency, and improving configuration efficiency.

[0087] Another embodiment of this application further supplements the description of the satellite network microservice deployment method based on multi-agent reinforcement learning provided in the above embodiments.

[0088] Figure 2 A flowchart illustrating another satellite network microservice deployment method based on multi-agent reinforcement learning provided in this application embodiment is shown below. Figure 2 As shown, the satellite network microservice deployment method based on multi-agent reinforcement learning includes:

[0089] Step 1: Build a microservice deployment model;

[0090] Specifically, the network structure, number of satellite nodes, and types of resources of the satellite edge computing system are determined, and a microservice deployment model is constructed, such as... Figure 3 As shown, steps 101 to 104 are included, as follows:

[0091] Step 101: Consider a satellite edge computing scenario, such as... Figure 1 As shown, it contains a set of satellite edge nodes S = {s1, s2, ..., s} N}, where N is the number of satellite nodes. In edge computing scenarios, the total number of resource types is R (CPU, memory, disk I / O, etc.), represented as R = {r1, r2, ..., r}. R For node s i Its heterogeneous resource capacity is represented by the vector V. i ={V i 1 V i 2 ,...,V i R}, where V i j Represents node s i Up Resources r j Available capacity.

[0092] Step 102: The set of microservices for the target application deployed in the satellite edge computing platform is MS = {ms1, ms2, ..., ms}. M} where M is the number of microservices, and the microservices are deployed as containers on edge nodes. Resource requests from different microservices are denoted as vectors. in Indicates microservices (ms) i For resource r j The number of requests.

[0093] Step 103: Each microservice can have multiple replicas, i.e., multiple containers deployed on different nodes. Let the number of containers for each microservice be Q = {q1, q2, ..., q}. M}, where q i Indicates microservices (ms) i The number of container replicas. Therefore, the total number of containers to be deployed is ∑Q, which can be represented as a set. Indicates microservices (ms) i The corresponding j-th container replica. Define container instance scheduling decision variables. when Time represents container instance Deployed on node s i If it is above, then the value is 0; otherwise, the value is 0.

[0094] Step 104: The call relationship between microservices is represented by a directed acyclic graph, and the adjacency matrix Y represents the call relationship. ij =1 indicates microservice milliseconds (ms) i The next microservice to be called is ms j Otherwise, the value is 0.

[0095]

[0096] Step 2: Optimize the problem representation, i.e., establish a resource utilization model and a latency model. Minimize the resource utilization variance and latency, representing the microservice deployment problem as a multi-objective optimization problem.

[0097] Alternatively, the resource utilization model can be obtained as follows:

[0098] Obtain resource balance information of the first resource utilization model for different types of resources on the same satellite node, and node balance information of the second resource utilization model for the same type of resources on different satellite nodes;

[0099] Based on the resource balance information and the corresponding weight values, as well as the node balance information and the corresponding weight values, the resource utilization model is determined.

[0100] In this step, the resource utilization model is represented by variance, which is divided into two categories. One category is the variance of different types of resources at the same node, to prevent an overabundance of one type of resource, which could lead to a bottleneck effect and resource waste. The other category is the variance of the same type of resource at different nodes, to prevent satellite node resources from being idle.

[0101] Step 201: Establish a satellite node resource utilization model.

[0102] Nodes s i Up Resources r j utilization rate u i.j It can be represented as:

[0103]

[0104] Resource utilization models are divided into two categories. One category considers the resource utilization of different types of resources on the same node. Each microservice has different resource type requests. When multiple microservices with the same resource type are deployed to the same node, other microservices may be unable to deploy on that node, creating a "bottleneck effect" and resulting in resource waste. Resource balance is represented by the standard deviation, where s is the number of nodes. i The degree of balance ε of all resources on i It can be represented as:

[0105]

[0106] Another category is the resource utilization of the same type of resource across different nodes. As the number of microservices increases, it's desirable to utilize all satellite edge nodes to prevent resource idleness and waste. (Resource r) j Equilibrium at different nodes It can be represented as:

[0107]

[0108] Therefore, the formula for calculating the cluster's resource utilization rate U is: where α is the weighting factor:

[0109]

[0110] Optionally, the delay model includes at least a transmission delay sub-model, a propagation delay sub-model, and a migration delay sub-model.

[0111] Latency is categorized into transmission latency, propagation latency, and migration latency. Transmission latency can be expressed as the quotient of data size and transmission rate, which can be calculated using Shannon's formula. Propagation latency is directly proportional to the physical distance between nodes. Migration latency is determined by the migration frequency of microservices.

[0112] Step 202: Establish a delay model, which is divided into transmission delay, propagation delay and migration delay.

[0113] Specifically, depending on the service deployment scenario, it is necessary to consider the communication links between the ground station and satellite nodes, as well as the communication links between satellite nodes. According to Shannon's theorem, the data transmission rate from the ground station to the destination satellite node can be expressed as:

[0114]

[0115] Among them, W g_s p is the channel bandwidth. g For the transmission power of the ground station, g g_s The channel gain between the ground station and the target satellite is N0, where N0 represents the background noise. This represents the sum of other noise interference power from the ground station to the satellite.

[0116] set up For node s i and s j Inter-satellite link channel bandwidth, Given the signal-to-noise ratio between the two nodes, the data transmission rate between the two satellite nodes is:

[0117]

[0118] Based on the adjacency matrix, the complete scheduling chain data transmission delay can be calculated as follows:

[0119]

[0120] Where, d g_s d represents the amount of data transmitted from the ground station to the satellite. i,j Indicates satellite node s i and s j The amount of data transmitted.

[0121] The information propagation speed is the speed of light c, and the node s i and s j The distance between them is represented as dis i,j The propagation delay can then be expressed as:

[0122]

[0123] When a satellite moves out of the service cell's visibility range, a migration action is required. This migration action incurs a migration delay. The agent's migration action is modeled as a directed weighted graph, where the weights represent the migration delay from the origin satellite to the destination satellite. The overall migration cost is the sum of these weights, expressed as...

[0124]

[0125] Tr represents the entire migration path under the agent's actions within this time period, w i,j Indicates the weight.

[0126] The overall time delay D can be calculated as follows:

[0127] D = τ trans +τ prop +τ mig

[0128] Some embodiments of this application establish resource utilization models and latency models, minimize resource utilization variance and latency, and represent the microservice deployment problem as a multi-objective optimization problem.

[0129] Step 203: Optimize the problem representation.

[0130] Based on the above model, a joint optimization problem can be established, minimizing the standard deviation of resource utilization U and minimizing the overall time delay D, expressed as:

[0131] P:min(U),min(D)

[0132]

[0133]

[0134] Where C1 means that each microservice must deploy at least one container instance to implement the function of the microservice, and C2 means that the amount of data requested by the microservice cannot exceed the maximum resource capacity of the node.

[0135] Step 3: Represent the microservice deployment problem as a partially observable Markov decision process and solve it using a multi-agent reinforcement learning method.

[0136] like Figure 4As shown, since the agent cannot obtain all the state information, the problem is partially observable. Each agent has its own observation space. As time changes, the relative position of the satellite changes, which in turn affects the communication delay. Therefore, the environmental state undergoes state transition with time and the actions of the agent.

[0137] Optionally, the multi-agent policy deployment model is obtained in the following way:

[0138] Obtain agent sample parameters, wherein the agent sample parameters include at least the agent's observation environment:

[0139] Acquire intelligent network models, which include at least actor network models and critic network models;

[0140] The agent's observation environment is input into the agent network model, and the agent's deployment actions are output.

[0141] The agent's deployment actions and global state are input into the critic network model, which outputs action evaluation values.

[0142] A replay pool is established based on the agent's current state information, action information, reward information, and state information at the next moment;

[0143] A multi-agent deep deterministic policy gradient algorithm is adopted to update the network parameters in the actor network model and the critic network model based on the current state information, action information, reward information and the state information of the next time step obtained from the replay pool.

[0144] If the actor network model and the critic network model converge, the converged actor network model and the critic network model are identified as the multi-agent policy deployment model.

[0145] Some embodiments of this application transform the microservice deployment problem into a partially observable Markov decision process and solve it using a multi-agent reinforcement learning method. The method employs centralized training and distributed execution. During the training phase, the container instances of the microservices, acting as agents, need to acquire global information to obtain the optimal deployment scheme. During the execution phase, the microservices can complete the deployment using only their own observation space, which greatly reduces the communication overhead between microservices.

[0146] Optionally, the network parameters in the actor network model and the critic network model are updated, including:

[0147] Obtain the first loss function of the actor network model and the second loss function of the critic network model;

[0148] Calculate the gradients of the first loss function and the second loss function respectively;

[0149] The gradient descent method is used to update the network parameters in the actor network model and the critic network model.

[0150] Some embodiments of this application employ a fixed network method, which fixes the target network and transmits the original network parameters to the target network at regular intervals to avoid continuous changes in the updated target and ensure the stability of training.

[0151] Specifically, in the embodiments of this application, some observable Markov decision processes are represented as node pre-selection;

[0152] Step 301: State Space Representation. Global state information includes the resource usage of satellite nodes, satellite location information, and container deployment status, which can be represented as s = [u, p, c], where:

[0153] u = [u 1,1 ,u 1,2 ,...,u 1,R ,u 2,1 ,u 2,2 ,...,u 2,R ,...,u N,1 ,u N,2 ,...,u N,R ]

[0154] p = [x1, y1, z1, x2, y2, z2, ..., x N ,y N ,z N ]

[0155]

[0156] u i,j For node s i Up Resources r j utilization rate, [x i ,y i ,z i ] is node s i Location coordinates, The index number of the node deployed for the container.

[0157] As intelligent agents, containers cannot obtain global state information during deployment. The observation space of container instance j on microservice i can be represented as follows:

[0158] Step 302: Action Space Representation. Based on its own observation space, the action space of container instance j on microservice i is represented as follows: k is the number of nodes in the observation space that meet the resource requirements, when The value is 0 if the container is deployed to that node, otherwise it is 0.

[0159] Step 303: State Transition Function Representation. The satellite's position changes continuously over time. The intelligent agent performs corresponding actions based on these changes, leading to a change in its state. The state transition function can be represented as follows:

[0160]

[0161] Step 304: Reward Function Representation. We aim to minimize resource utilization variance and latency; therefore, the reward function can be expressed as follows:

[0162] reward=-(βU+(1-β)D)

[0163] Step 4: Use the Multi-Agent Deep Deterministic Strategy Gradient Algorithm (MADDPG) to train the agent network model for each parameter, including building the neural network and updating the network parameters.

[0164] These include:

[0165] Building an intelligent agent network, such as Figure 4 As shown. The container is considered as an agent, and each agent contains four networks: the Actor network μ(o i ;θ i ), Target actor network t_μ(o i ;θ i ) and the Critic network c(s,a;ω i ), Targetcritic network t_c(s,a;ω i This includes steps 401 to 402. A fixed network method is used, where the target network is fixed and the parameters of the original network are passed to the target network at regular intervals to avoid the target from changing and to ensure the stability of training.

[0166] Step 401: Setting up the two networks. The input to the Actor network model is the local observation information of the current agent. i The input to the Critic Network model is the action and global state output by the Actor Network, i.e., global state information s and action a. The output is the corresponding Q-value, which is used to evaluate the quality of the action performed by the agent in the current state.

[0167] Step 402: Network parameter transfer process. During parameter updates, if the target network keeps changing, updates become difficult. Therefore, a fixed network method is used: the parameters of the target network are fixed, and the original network parameters are transferred to the target network at intervals to ensure training stability.

[0168] Step 5: Build an experience replay pool D. The agent randomly takes a deployment action based on the noise settings, generating a four-tuple that records the state, the agent's action, the next state, and the reward, denoted as (s). t ,a t ,r t ,s t+1 Since the MADDPG algorithm is a heterogeneous strategy, it can utilize an experience replay pool to eliminate the correlation of historical experience and break it down. When training the neural network, a batch of experience data can be randomly selected to improve the training of the neural network.

[0169] Step Six: Execute the MADDPG algorithm to update network parameters and perform centralized training. Randomly select quadruplets from the replay pool to update the Actor and Critic network parameters until convergence.

[0170] This is mainly reflected in the update process of two types of networks, including steps 601 to 602.

[0171] Step 601: Update the Actor network parameters. The loss function of the Actor network is -Q, which is obtained by inputting the output action of the Actor network into the current Critic network. A smaller -Q is better. The observation space o of agent i in the replay pool is... i The input to the Actor network μ(o) of this agent i ;θ i In the process, the deployment action a is obtained. i Then, the global state information s and a i The input is fed into the Critic network to obtain the Q-value of the action. The network parameters θ are then updated using gradient descent with -Q as the loss function. i Specifically, the loss function can be expressed as:

[0172]

[0173] Where x = (o1, o2, ..., o N ) represents the observation space of all agents, a i This indicates that agent i, in its policy μ i The following action. According to the chain rule, its gradient can be expressed as:

[0174]

[0175] Use gradient descent to update the parameter θ. i

[0176]

[0177] Step 602: Update the Critic network parameters. The Critic network needs to predict Q values ​​as accurately as possible; therefore, its loss function is the output Q(s0, a0; ω) of the Critic network. i The sum of the predicted value (r1+γQ(s1,a1;ω)) and the output Q-value of the Targetcritic network with the reward is r1+γQ(s1,a1;ω). i The difference between the actual value and the network parameter ω is considered, and the smaller the difference, the better. This difference can be represented by MSE, and the network parameter ω is updated using gradient descent. i Specifically, the loss function can be expressed as:

[0178]

[0179]

[0180] Calculate gradient

[0181]

[0182] Update parameter ω using gradient descent. i

[0183]

[0184] The two networks calculate their gradients based on their respective loss functions and use gradient descent to update the network parameters.

[0185] Figure 5 This is a schematic diagram of the model training process provided in the embodiments of this application, such as... Figure 5 As shown, it includes:

[0186] 1) Initialize the network and satellite nodes to obtain a set of candidate nodes;

[0187] 2) The container agent randomly generates actions, resulting in a quadruple;

[0188] 3) Store the quadruple in the experience replay pool;

[0189] 4) Randomly select quaternions;

[0190] 5) Update the Actor network and Critic network according to the LOSS function;

[0191] 6) Update the Target network parameters;

[0192] 7) Output the trained policy network, i.e., the multi-agent policy deployment model.

[0193] Step 5: Deploy the policy network.

[0194] By deploying the trained policy network onto container agents, microservices can make optimal decisions independently based on local observations.

[0195] This application provides a satellite network microservice deployment method based on multi-agent reinforcement learning to address the microservice deployment problem. The neural network portion employs a fixed-network approach, consisting of an original network and a target network. The target network is fixed first, and the parameters of the original network are periodically passed to the target network. This avoids update difficulties caused by constantly changing update targets, ensuring training stability. After training, the agents only need to utilize their own observation space and the policy network to make optimal actions, reducing the overhead of frequent interactions between microservices.

[0196] It should be noted that each of the implementable methods in this embodiment can be implemented individually or in any combination without conflict. This application does not limit this.

[0197] Another embodiment of this application provides a satellite network microservice deployment apparatus based on multi-agent reinforcement learning, used to execute the satellite network microservice deployment method based on multi-agent reinforcement learning provided in the above embodiments.

[0198] like Figure 6 The diagram shown is a structural schematic of a satellite network microservice deployment device based on multi-agent reinforcement learning provided in an embodiment of this application. This satellite network microservice deployment device based on multi-agent reinforcement learning includes an acquisition module 601, a first determination module 602, a second determination module 603, and a configuration module 604, wherein:

[0199] Module 601 is used to obtain resource requirement information for microservices;

[0200] The first determining module 602 is used to determine the resource utilization information and latency information of the satellite node based on the pre-established resource utilization model and latency model, as well as the configuration information of the satellite node.

[0201] The second determining module 603 is used to determine the deployment strategy of satellite nodes corresponding to the resource demand information of microservices by using a pre-trained multi-agent strategy deployment model when the resource utilization information is less than the first preset value or the latency information is less than the second preset value. The pre-trained multi-agent strategy deployment model is obtained by training each parameter of the agent network model using a multi-agent deep determination strategy gradient algorithm.

[0202] Configuration module 604 is used to configure the server terminal according to the deployment strategy of the satellite nodes.

[0203] Regarding the apparatus in this embodiment, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0204] Some embodiments of this application establish resource utilization and latency models based on a microservice architecture. Based on the configuration information of satellite nodes, the resource utilization and latency information of each satellite node are determined. Then, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite nodes corresponding to the resource requirements of the microservices. The server terminal is configured according to this deployment strategy. In this way, resource configuration can be performed on each satellite node based on the server resource requirements of the microservices and the remaining resources of each satellite node. This means that microservices with different resource requirements are deployed to appropriate satellite nodes, improving the resource utilization balance of satellite nodes, reducing call latency, and improving configuration efficiency.

[0205] This application provides another embodiment to further illustrate the satellite network microservice deployment device based on multi-agent reinforcement learning provided in the above embodiments.

[0206] Optionally, the device further includes a model training module, which is used for:

[0207] Obtain agent sample parameters, wherein the agent sample parameters include at least the agent's observation environment:

[0208] Acquire intelligent network models, which include at least actor network models and critic network models;

[0209] The agent's observation environment is input into the agent network model, and the agent's deployment actions are output.

[0210] The agent's deployment actions and global state are input into the critic network model, which outputs action evaluation values.

[0211] A replay pool is established based on the agent's current state information, action information, reward information, and state information at the next moment;

[0212] A multi-agent deep deterministic policy gradient algorithm is adopted to update the network parameters in the actor network model and the critic network model based on the current state information, action information, reward information and the state information of the next time step obtained from the replay pool.

[0213] If the actor network model and the critic network model converge, the converged actor network model and the critic network model are identified as the multi-agent policy deployment model.

[0214] Some embodiments of this application transform the microservice deployment problem into a partially observable Markov decision process and solve it using a multi-agent reinforcement learning method. The method employs centralized training and distributed execution. During the training phase, the container instances of the microservices, acting as agents, need to acquire global information to obtain the optimal deployment scheme. During the execution phase, the microservices can complete the deployment using only their own observation space, which greatly reduces the communication overhead between microservices.

[0215] Optionally, the model training module is used for:

[0216] Obtain the first loss function of the actor network model and the second loss function of the critic network model;

[0217] Calculate the gradients of the first loss function and the second loss function respectively;

[0218] The gradient descent method is used to update the network parameters in the actor network model and the critic network model.

[0219] Some embodiments of this application employ a fixed network method, which fixes the target network and transmits the original network parameters to the target network at regular intervals to avoid continuous changes in the updated target and ensure the stability of training.

[0220] Optionally, the configuration information of satellite nodes may include at least the number of satellite nodes, the total number of resource types, and the capacity of heterogeneous resources.

[0221] Optionally, the model training module is used for:

[0222] Obtain resource balance information of the first resource utilization model for different types of resources on the same satellite node, and node balance information of the second resource utilization model for the same type of resources on different satellite nodes;

[0223] Based on the resource balance information and the corresponding weight values, as well as the node balance information and the corresponding weight values, the resource utilization model is determined.

[0224] Optionally, the delay model includes at least a transmission delay sub-model, a propagation delay sub-model, and a migration delay sub-model.

[0225] Some embodiments of this application establish resource utilization models and latency models, minimize resource utilization variance and latency, and represent the microservice deployment problem as a multi-objective optimization problem.

[0226] Regarding the apparatus in this embodiment, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0227] It should be noted that each of the implementable methods in this embodiment can be implemented individually or in any combination without conflict. This application does not limit this.

[0228] This application also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it can implement the operation of any of the methods corresponding to the embodiments of the satellite network microservice deployment method based on multi-agent reinforcement learning provided in the above embodiments.

[0229] This application also provides a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the embodiments of the satellite network microservice deployment method based on multi-agent reinforcement learning provided in the above embodiments.

[0230] like Figure 7 As shown, some embodiments of this application provide an electronic device 700, which includes: a memory 710, a processor 720, and a computer program stored on the memory 710 and executable on the processor 720. When the processor 720 reads the program from the memory 710 and executes the program via a bus 730, it can implement any of the methods included in the above-described satellite network microservice deployment method based on multi-agent reinforcement learning.

[0231] Processor 720 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 720 can be a microprocessor.

[0232] The memory 710 can be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 720 of this disclosure embodiment can be used to execute the instructions in the memory 710 to implement the methods shown above. The memory 710 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0233] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0234] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0235] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for deploying microservices in satellite networks based on multi-agent reinforcement learning, characterized in that, The method includes: Obtain resource requirements information for microservices; Based on the pre-established resource utilization model and latency model, as well as the configuration information of the satellite nodes, the resource utilization information and latency information of the satellite nodes are determined; the configuration information of the satellite nodes includes at least the number of satellite nodes, the total amount of resource types, and the capacity of heterogeneous resources; the latency model includes at least the transmission latency sub-model, the propagation latency sub-model, and the migration latency sub-model. When the resource utilization information is less than a first preset value, or the latency information is less than a second preset value, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice. The pre-trained multi-agent strategy deployment model is obtained by training each parameter of the agent network model using a multi-agent deep deterministic strategy gradient algorithm. Configure the server terminal according to the deployment strategy of the satellite nodes; in, The resource utilization model is obtained in the following way: Obtain resource balance information of the first resource utilization model for different types of resources on the same satellite node, and node balance information of the second resource utilization model for the same type of resources on different satellite nodes; The resource utilization model is determined based on the resource balance information and the weight value corresponding to the resource balance information, as well as the node balance information and the weight value corresponding to the node balance information.

2. The satellite network microservice deployment method based on multi-agent reinforcement learning according to claim 1, characterized in that, The multi-agent policy deployment model is obtained through the following method: Obtain agent sample parameters, wherein the agent sample parameters include at least the agent's observation environment: Obtain an intelligent network model, which includes at least an actor network model and a critic network model; The agent's observation environment is input into the actor network model, and the agent's deployment action is output. The deployment actions and global state of the agent are input into the critic network model, and action evaluation values ​​are output. A replay pool is established based on the agent's current state information, action information, reward information, and state information at the next moment; A multi-agent deep deterministic policy gradient algorithm is used to update the network parameters in the actor network model and the critic network model based on the current state information, action information, reward information and the state information at the next moment obtained from the replay pool. If the actor network model and the critic network model converge, the converged actor network model and the critic network model are identified as the multi-agent policy deployment model.

3. The satellite network microservice deployment method based on multi-agent reinforcement learning according to claim 2, characterized in that, The updating of network parameters in the actor network model and the critic network model includes: Obtain the first loss function of the actor network model and the second loss function of the critic network model; Gradient calculations are performed on the first loss function and the second loss function, respectively. The network parameters in the aforementioned actor network model and the aforementioned critic network model are updated using the gradient descent method.

4. A satellite network microservice deployment device based on multi-agent reinforcement learning, characterized in that, The device includes: The acquisition module is used to acquire resource requirement information for microservices; The first determining module is used to determine the resource utilization information and latency information of satellite nodes based on a pre-established resource utilization model and latency model, as well as the configuration information of satellite nodes; the configuration information of satellite nodes includes at least the number of satellite nodes, the total amount of resource types, and the capacity of heterogeneous resources; the latency model includes at least a transmission latency sub-model, a propagation latency sub-model, and a migration latency sub-model. The second determining module is used to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice by using a pre-trained multi-agent strategy deployment model when the resource utilization information is less than a first preset value or the latency information is less than a second preset value. The pre-trained multi-agent strategy deployment model is obtained by training each parameter of the agent network model using a multi-agent deep determination strategy gradient algorithm. The configuration module is used to configure the server terminal according to the deployment strategy of the satellite nodes; in, The resource utilization model is obtained in the following way: Obtain resource balance information of the first resource utilization model for different types of resources on the same satellite node, and node balance information of the second resource utilization model for the same type of resources on different satellite nodes; The resource utilization model is determined based on the resource balance information and the weight value corresponding to the resource balance information, as well as the node balance information and the weight value corresponding to the node balance information.

5. The satellite network microservice deployment device based on multi-agent reinforcement learning according to claim 4, characterized in that, The device further includes a model training module, which is used for: Obtain agent sample parameters, wherein the agent sample parameters include at least the agent's observation environment: Obtain an intelligent network model, which includes at least an actor network model and a critic network model; The agent's observation environment is input into the actor network model, and the agent's deployment action is output. The deployment actions and global state of the agent are input into the critic network model, and action evaluation values ​​are output. A replay pool is established based on the agent's current state information, action information, reward information, and state information at the next moment; A multi-agent deep deterministic policy gradient algorithm is used to update the network parameters in the actor network model and the critic network model based on the current state information, action information, reward information and the state information at the next moment obtained from the replay pool. If the actor network model and the critic network model converge, the converged actor network model and the critic network model are identified as the multi-agent policy deployment model.

6. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can implement the satellite network microservice deployment method based on multi-agent reinforcement learning as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, characterized in that, when the program is executed by a processor, it can implement the satellite network microservice deployment method based on multi-agent reinforcement learning as described in any one of claims 1-3.

Images