A deep reinforcement learning-based on-board micro-service deployment method

Abstract

This invention relates to the fields of on-board microservice deployment and satellite communication, specifically to an on-board microservice deployment method based on deep reinforcement learning. The method includes: assigning a unique service requirement attribute to each microservice to quantify its resource requirements; defining microservices as virtual network request nodes and satellite nodes as physical network nodes, and establishing network models for both virtual and physical networks; extracting network resource attributes from the physical network, constructing a feature matrix, inputting it into a policy network built using basic neural network elements, performing training to obtain a set of available satellite nodes and mapping probabilities, and using the communication latency between all microservices as a reward signal; using the satellite nodes and mapping probabilities to sequentially place microservices, and adjusting the microservice placement strategy based on the reward signal. This invention provides a refined and efficient solution to the design challenges of microservice deployment methods, possessing strong practical application value and scalability.

Description

Technical Field

[0001] This invention relates to the fields of onboard microservice deployment and satellite communication, specifically to an onboard microservice deployment method based on deep reinforcement learning. Background Technology

[0002] In recent years, satellite technology has developed rapidly, and satellite communication is being applied in an increasing number of fields. Deploying microservices on satellites to achieve different levels of Quality of Service (QoS) for different users offers numerous benefits. First, it improves application reliability because each service unit operates independently; even if one service unit fails, others can still function normally, ensuring the overall availability of the application. Second, it enhances application scalability because each service unit can be scaled independently to meet diverse application needs. Finally, it reduces application complexity and cost because each service unit is independent and can utilize different programming languages ​​and development tools, making the development process simpler and more efficient. However, placing instances of various microservice categories on suitable satellites to achieve uninterrupted microservice provision or minimize latency in microservice chain calls remains a challenge.

[0003] Deep reinforcement learning can effectively solve the microservice deployment problem. Compared with other deployment methods, applying deep reinforcement learning techniques has the following advantages:

[0004] Highly adaptive: Deep reinforcement learning algorithms can adaptively change their strategies according to changes in the environment, thus better adapting to the dynamic orchestration needs of microservice chains.

[0005] Good predictability: Deep reinforcement learning algorithms can learn and predict using historical data, thus better predicting future needs and changes, thereby better optimizing microservice chain orchestration.

[0006] Good optimization performance: Deep reinforcement learning algorithms can find the global optimum or local optimum in a relatively short time.

[0007] Highly scalable: Deep reinforcement learning algorithms can be extended to large-scale microservice deployment problems and support complex microservice chain orchestration requirements.

[0008] Therefore, deep reinforcement learning techniques can be used to provide a reliable on-board microservice placement strategy and solve the deployment problem of on-board microservices. Summary of the Invention

[0009] To address the challenges in designing placement strategies for microservice instances, and with the goal of providing uninterrupted microservices or minimizing call latency in the microservice chain, this paper proposes an on-board microservice deployment method based on deep reinforcement learning.

[0010] The technical solution adopted in this invention is as follows:

[0011] A method for deploying on-board microservices based on deep reinforcement learning includes the following process:

[0012] (1) Divide the types of microservice instances and set a unique service requirement attribute for each microservice to quantify the resource requirements of the microservice;

[0013] (2) Define microservices as virtual network request nodes and satellite nodes as physical network nodes, and establish network models for virtual network requests and physical networks using network graph method and undirected weighted graph method respectively;

[0014] (3) Extract network resource attributes from the physical network, construct a feature matrix, input the policy network built using the basic elements of the neural network, perform learning and training, obtain a set of available satellite nodes and mapping probabilities, and use the communication latency between all microservices as a reward signal.

[0015] (4) Using satellite nodes and mapping probabilities, microservice instances are placed sequentially, and the microservice placement strategy is adjusted according to the reward signal.

[0016] Furthermore, the network model established in step (2) is specifically as follows:

[0017] Virtual network request modeling is VNR = {Catagory(n v ),CPU(n v ),STO(n v )}, where Catagory(n v ) represents microservice n v The type of CPU(n) v ) represents microservice n v The computational resource requirements, STO(n) v ) represents microservice n v Storage resource requirements;

[0018] The physical network is modeled as an undirected weighted graph G = {N} S E S}, where N S E represents the set of satellite nodes. S N represents the set of satellite links; S ={CPU(n s ),STO(n sCPU(n) s ) represents satellite node n s Available computing resources, STO(n) s ) represents satellite node n s The amount of available storage resources; Represents physical nodes With physical nodes The time delay between them.

[0019] Furthermore, step (3) specifically involves:

[0020] Employing deep reinforcement learning, the following elements are defined: the intelligent agent is a policy network built using basic neural network elements; the agent training environment is a feature matrix composed of network resource attributes extracted from the physical network, where network resource attributes include computing resources (CPU), storage resources (STO), the degree (DEG) of satellite nodes, and the sum of link delays (Delay) connecting to satellite nodes; the action is the mapping process of virtual network request nodes, i.e., the specific placement process of microservice instances; the state is the resource change status of the physical network; and the reward signal is the communication delay between all microservices, with a larger reward signal for a smaller delay value.

[0021] For each satellite node, a feature vector is established. For the i-th satellite node n s In this context, the feature vector is represented as:

[0022]

[0023] The feature vectors of all k satellite nodes in the current time slice are combined into a feature matrix, which is represented as:

[0024]

[0025] The feature matrix is ​​input into a policy network built using basic elements of a neural network. The intelligent agent interacts with the environment to perceive the real network resource situation, obtains a set of available satellite nodes and mapping probabilities, and uses the communication latency between all microservices as a reward signal.

[0026] Furthermore, from the network resource attributes extracted from the physical network:

[0027] The degree DEG of a satellite node represents the number of links directly connected to the satellite node within a certain time slice;

[0028] The sum of the link delays connected to the satellite node, Delay, represents the sum of the delays of the links directly connected to the satellite within a certain time slice;

[0029] Storage resources (STO) represent the storage resources possessed by a single satellite node;

[0030] Computing resources (CPU) represent the computing power of a single satellite node.

[0031] Furthermore, the policy network includes an extraction layer, a convolutional layer, a softmax layer, a filtering layer, and an output layer;

[0032] The extraction layer is used to extract the feature matrix of the network at the current time from the satellite network as input to the policy network;

[0033] Convolutional layers are used to perform convolution operations on each eigenvector of the feature matrix to obtain the usable resource form of each eigenvector;

[0034] The softmax layer is used to apply the softmax function to each feature vector, calculate a probability for each satellite node, and sum the probabilities of all satellite nodes to 1.

[0035] The filtering layer is used to filter out satellite nodes with insufficient resource capacity based on the actual computing and storage resource requirements of the microservice instance;

[0036] The output layer is used to output a set of available satellite nodes and their mapping probabilities.

[0037] Compared to traditional microservice placement strategies, the method of this invention considers the specific functions (load balancing, circuit breaking, etc.) and resource requirements of microservice instances, resulting in more refined and efficient microservice orchestration. Furthermore, this method employs deep reinforcement learning, enabling the placement strategy to self-optimize and adapt, demonstrating good practicality and feasibility.

[0038] This invention takes into account the specific functions and resource requirements of microservice instances, and uses deep reinforcement learning as a foundation to solve the design challenges of microservice placement strategies in a refined and efficient manner. It has strong practical application value and scalability. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of microservice deployment and orchestration provided for an example of the present invention.

[0041] Figure 2 This invention provides a classification of microservice instance types and resource requirements for a specific time slice in an example of the invention.

[0042] Figure 3A network model diagram provided for an example of the present invention.

[0043] Figure 4 The policy network model diagram provided for an example of the present invention.

[0044] Figure 5 The overall process of microservice placement strategy provided for the example of this invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] In this microservice deployment method, instances of various microservice categories are placed on appropriate satellite constellations to achieve uninterrupted microservice provision or ensure the shortest possible latency in microservice chain calls. The overall process can be divided into four steps:

[0047] ① Classification of microservice instance types;

[0048] ②System modeling;

[0049] ③ Satellite node selection;

[0050] ④ Microservice placement.

[0051] Defining microservice placement as a candidate physical node selection problem, microservices as virtual network request nodes, and satellite nodes as physical network nodes, the selection of satellite nodes is essentially the process of selecting candidate physical nodes. The overall process can be divided into:

[0052] ① Extract the network feature matrix at the current moment from the satellite network;

[0053] ② Resource vectors are obtained by extracting local spatial features of satellite nodes in the satellite network through convolutional layers;

[0054] ③ Calculate the probability of a satellite node being placed based on available resources using a probability layer;

[0055] ④ Filter out satellite nodes with insufficient resource capacity through the filtering layer;

[0056] ⑤ Obtain a set of available satellite nodes and their mapping probabilities from the output.

[0057] like Figure 1 As shown.

[0058] (1) Classification of Microservice Instances

[0059] Within each defined time slice, ensure that at least one instance of each type of microservice is accessible. This aims to avoid excessively long-distance interactions between microservice instances due to satellite movement, and is a prerequisite for implementing a load-balanced placement strategy.

[0060] Taking OODA as an example, a unique service requirement attribute is set for each microservice, such as... Figure 2 The nodes at different depths represent microservice instances with different service requirements. Since different microservices have varying resource needs, the most suitable satellite constellation needs to be selected based on the specific functions implemented by the microservices. Therefore, it is necessary to quantify the resource requirements of the microservices. For example... Figure 1 As shown, the computing resource (CPU) and storage resource (STO) requirements of each microservice instance are marked next to it.

[0061] (2) System Modeling

[0062] The placement of microservices is defined as a candidate physical node selection problem. Microservices are defined as virtual network request nodes, and satellite nodes are defined as physical network nodes. Network models for virtual network requests and physical networks are established using network graph methods and undirected weighted graph methods, respectively. Figure 3 As shown.

[0063] Virtual network request modeling is VNR = {Catagory(n v ),CPU(n v ),STO(n v )}, where Catagory(n v ) represents microservice n v The type of membership, taking OODA (Boyd's loop) as an example, can be represented as n v ∈{Observe|Orient|Decide|Act}, representing virtual node n v In observation, judgment, decision-making, and action, only one behavior can be changed; CPU(n v ) represents microservice n v The computational resource requirements of STO(n) v ) represents microservice n v The storage resource requirements. The physical network is modeled as an undirected weighted graph G = {N}. S E S}, where N S Represents physical (satellite) node n s The set, E S This represents the set of physical (satellite) links. Specifically, N S ={CPU(n s ),STO(ns CPU(n) s ) represents physical node n s Available computing resources, STO(n) s ) represents physical node n s The amount of available storage resources; Represents physical nodes With physical nodes The time delay between them.

[0064] (3) Satellite node selection

[0065] Satellite nodes are defined as physical network nodes, and the entire satellite network topology is regarded as a physical network. The selection of satellite nodes is essentially the selection process of candidate physical nodes.

[0066] This phase proposes to employ deep reinforcement learning. First, the key elements of deep reinforcement learning are established: Agent – ​​a policy network built using basic neural network elements; Environment – ​​a feature matrix composed of network resource attributes extracted from the physical network; Action – the mapping process of virtual request nodes, i.e., the specific placement process of microservice instances; State – the resource changes in the physical network; Reward – the communication latency between all microservices, i.e., f = ∑ A-B The smaller the value of t(A,B), the stronger the reward signal.

[0067] Deep reinforcement learning agents are trained in an environment composed of physical network resource attributes. By interacting with the environment, the agents perceive the real underlying network resource conditions and make optimal network node selection decisions.

[0068] To enable the agent to fully perceive the physical network environment, the following four network attributes are proposed to be extracted from the physical network to build the agent training environment:

[0069] Computing resources (CPU): Each satellite node has a certain computing capacity. Satellite nodes need to be selected based on the actual computing resource requirements of the microservice instance to avoid insufficient computing capacity of satellite nodes or excessive fragmentation of computing resources.

[0070] Storage Resources (STO): Each satellite node has limited storage resources. Satellite nodes need to be selected based on the actual storage resource requirements of the microservice instance to avoid insufficient storage capacity of satellite nodes or excessive storage resource fragmentation.

[0071] Satellite node degree (DEG): The number of links directly connected to this satellite node within a given time slice. A higher degree indicates a greater number of other satellites directly connected to that node. Selecting this satellite node increases the likelihood of short-range interactions with other satellites.

[0072] Sum of link delays to satellite nodes (Delay): The sum of the delays of links directly connected to the satellite within a certain time slice. The smaller this value, the more likely it is that the selected node will be able to achieve short-latency communication with other satellite nodes.

[0073] The above four network attributes consider both local physical network attributes (CPU, STO) and global attributes (DEG, Delay), making them highly representative. Within a given time slice, the satellite network topology is relatively fixed, allowing the extraction of these four attributes for each satellite node. Thus, each satellite node has a unique corresponding feature vector. For the i-th satellite node, its feature vector is represented as...

[0074]

[0075] The feature vectors of all k satellite nodes in the current time slice are combined into a feature matrix, which is represented as follows:

[0076]

[0077] Building such using basic elements of neural networks Figure 4 The policy network shown uses the aforementioned feature matrix as input to the feature matrix.

[0078] The specific functions of each layer in the policy network are as follows:

[0079] Extraction layer: Extracts the network feature matrix at the current moment from the satellite network as input to the policy network.

[0080] Convolutional layer: Performs convolution operation on each feature vector of the feature matrix to obtain the usable resource form of each feature vector.

[0081] The softmax layer uses the softmax function to operate on each feature vector, calculating a probability for each satellite node. The probabilities of all satellite nodes are summed to 1. This probability indicates that mapping the virtual request node to the physical node might produce better results.

[0082] Filtering layer: Based on the actual computing and storage resource requirements of the microservice instance, filter out satellite nodes with insufficient resource capacity.

[0083] Output layer: Outputs a set of available satellite nodes and their mapping probabilities.

[0084] (4) Microservice placement

[0085] During the training phase, the intelligent agent is trained using the methods described above. To guide the intelligent agent to achieve good training results, the communication latency between all microservices is used as a reward signal, encouraging the intelligent agent to explore placement strategies with the lowest possible communication latency. In the final test, microservice instances are placed sequentially based on the obtained satellite nodes and mapping probabilities.

[0086] After selecting the satellite nodes where the microservice instances are placed, the Floyd shortest path algorithm is used to orchestrate the service function chain.

[0087] The overall process of microservice deployment strategy is as follows: Figure 5 As shown.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for deploying on-board microservices based on deep reinforcement learning, characterized in that, Includes the following processes: (1) Classify the types of microservice instances and set a unique service requirement attribute for each microservice to quantify the resource requirements of the microservice; (2) Define microservices as virtual network request nodes, define satellite nodes as physical network nodes, and use the network graph method to build a network model for virtual network requests and the undirected weighted graph method to build a network model for physical networks. (3) Extract network resource attributes from the physical network, construct a feature matrix, input the policy network built using the basic elements of the neural network, perform learning and training, obtain a set of available satellite nodes and mapping probabilities, and use the communication latency between all microservices as a reward signal; (4) Using satellite nodes and mapping probabilities, place microservice instances sequentially, and adjust the microservice placement strategy according to the reward signal; The network model established in step (2) is specifically as follows: Virtual network request modeling as ,in Represents microservices The types to which they belong, Represents microservices The computing resource requirements, Represents microservices Storage resource requirements; The physical network is modeled as an undirected weighted graph. ,in Represents the set of satellite nodes. Represents a set of satellite links; , Representative satellite node The amount of available computing resources, Representative satellite node The amount of available storage resources; , representing physical nodes With physical nodes The time delay between; Specifically, step (3) is as follows: Employing deep reinforcement learning, the following elements are defined: the intelligent agent is a policy network built using basic neural network elements; the agent training environment is a feature matrix composed of network resource attributes extracted from the physical network, where network resource attributes include computing resources (CPU), storage resources (STO), the degree (DEG) of satellite nodes, and the sum of link delays (Delay) connecting to satellite nodes; the action is the mapping process of virtual network request nodes, i.e., the specific placement process of microservice instances; the state is the resource change status of the physical network; and the reward signal is the communication delay between all microservices, with a larger reward signal for a smaller delay value. Each satellite node establishes a feature vector; for the ... n satellite nodes s In this context, the feature vector is represented as: All within the current time slice The feature vectors of each satellite node are combined into a feature matrix, represented as follows: The feature matrix is ​​input into a policy network built using basic elements of a neural network. The intelligent agent interacts with the environment to perceive the real network resource situation, obtains a set of available satellite nodes and mapping probabilities, and uses the communication latency between all microservices as a reward signal.

2. The on-board microservice deployment method based on deep reinforcement learning according to claim 1, characterized in that, Network resource attributes extracted from the physical network: The degree DEG of a satellite node represents the number of links directly connected to the satellite node within a certain time slice; The sum of the link delays connected to the satellite node, Delay, represents the sum of the delays of the links directly connected to the satellite within a certain time slice; Storage resources (STO) represent the storage resources possessed by a single satellite node; Computing resources (CPU) represent the computing power of a single satellite node.

3. The on-board microservice deployment method based on deep reinforcement learning according to claim 1, characterized in that, The policy network consists of an extraction layer, a convolutional layer, a softmax layer, a filtering layer, and an output layer; The extraction layer is used to extract the feature matrix of the network at the current time from the satellite network as input to the policy network; Convolutional layers are used to perform convolution operations on each eigenvector of the feature matrix to obtain the usable resource form of each eigenvector; The softmax layer is used to apply the softmax function to each feature vector, calculate a probability for each satellite node, and sum the probabilities of all satellite nodes to 1. The filtering layer is used to filter out satellite nodes with insufficient resource capacity based on the actual computing and storage resource requirements of the microservice instance; The output layer is used to output a set of available satellite nodes and their mapping probabilities.

Images