Edge delay-aware microservice instance selection method based on graph reinforcement learning

Abstract

The application discloses an edge delay perception microservice instance selection method based on graph reinforcement learning, belongs to the field of Internet of Things and edge computing, and comprises the following steps: first, edge computing environment state perception and graph structure feature extraction of an edge node instance deployment graph and a microservice request calling graph; second, microservice instance selection action decision generation based on graph features; third, edge computing environment interaction and classified experience sample recording; and fourth, edge microservice instance selection graph reinforcement learning model training and iterative optimization based on multiple groups of experiences, so that stable iterative optimization of the model is realized. The application realizes efficient selection of microservice instances in an Internet of Things edge computing environment, and takes into account request delay and instance load balancing.

Description

Technical Field

[0001] This invention belongs to the fields of Internet of Things and edge computing, and specifically relates to a method for selecting microservice instances with edge latency awareness based on graph reinforcement learning. Background Technology

[0002] With the rapid rise of the Internet of Things (IoT) industry, massive numbers of devices have enabled connections between people and things, and between things themselves, driving the digitalization of the physical world. IoT applications have widely penetrated into various fields such as smart elevators, smart healthcare, and smart transportation, requiring the collection and processing of massive amounts of device data every day. These applications are often computationally intensive and extremely sensitive to response latency. While cloud computing can provide powerful computing and storage capabilities, the long distance between cloud data centers and IoT devices leads to high latency, becoming a prominent pain point.

[0003] Against this backdrop, edge computing has emerged as an extension paradigm of cloud computing, deploying services near edge nodes to effectively reduce access latency. Meanwhile, IoT application development has shifted from monolithic architectures to microservice architectures, decomposing applications into multiple independent microservices, improving flexibility, scalability, and maintainability. However, while microservice architecture decouples the system, it also brings challenges to coordinating cross-service request business processes.

[0004] Microservice orchestration, as a declarative centralized coordination pattern, explicitly manages business logic through metadata-driven process definitions, supporting distributed transactions, data consistency, and process visualization. In dynamic edge computing environments, microservices are deployed on different edge nodes and the load is distributed across multiple instances. IoT requests require the collaboration of multiple interdependent microservices. Therefore, efficiently orchestrating edge network requests, selecting appropriate microservice instances for each request to meet service quality requirements, and maintaining load balancing among instances become key challenges.

[0005] In recent years, deep reinforcement learning (DRL) has achieved great success in many fields, leveraging the powerful understanding of perception problems by deep neural networks and the excellent decision-making capabilities of reinforcement learning agents. This provides strong model support for handling complex choices in dynamic edge environments. Graph neural networks (GNNs), especially their spatial methods, intuitively provide a way to process graph-structured data from the perspective of node spatial relationships, demonstrating their remarkable potential in handling dependencies between microservices and network topologies between edge nodes.

[0006] Existing methods, including traditional load balancing strategies such as heuristics, metaheuristics, and game theory-based approaches, are largely designed for cloud computing environments. They fail to adequately consider latency constraints in edge scenarios and are prone to getting trapped in local optima. Most DRL methods do not fully exploit the graph structure features of inter-service dependencies and edge node network topology. For example, Khansari et al.'s distributed DRL algorithm only perceives edge nodes and their first-order neighbors, failing to abstract microservice dependencies and edge network topology into a graph structure for feature extraction, thus failing to fully explore the cascading effects of service dependencies and the association information of node connections. Boudie et al. only modeled microservice dependencies as acyclic graphs, without combining edge network topology for graph structure feature learning, failing to capture the impact of global associations on performance. Guo et al.'s improved DDPG algorithm does not consider the graph structure features of microservice dependencies and edge topology, modeling only with a single latency feature, lacking global structural association awareness, and having limited adaptability to dynamic environments. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies and achieve efficient selection of microservice instances in IoT edge computing environments while balancing request latency and instance load balancing, this invention proposes an edge latency-aware microservice instance selection method based on graph reinforcement learning. This method first performs graph structure modeling and feature extraction on the edge computing environment state perception, edge node instance deployment graph, and microservice request call graph. Then, it generates efficient microservice instance selection decisions based on graph features. Next, it executes actions and interactions in the edge environment, designs a composite reward that integrates latency and load balancing, and stores effective experience categorized by request type. Finally, it uses the categorized experience to train the model, achieving stable iterative optimization.

[0008] The technical solution adopted by this invention to solve its technical problem is: An edge latency-aware microservice instance selection method based on graph reinforcement learning includes the following steps: The first step is to extract graph structure features of edge computing environment state perception and edge node instance deployment graph and microservice request call graph: Graph reinforcement learning-driven edge microservice instance selection has high requirements for the accuracy of environment state representation. This step constructs a two-layer graph structure of edge node instance deployment and request call, and applies the feature-enhanced graph attention mechanism to the feature extraction of edge computing platform topology, microservice instance deployment and IoT request call dependencies, and integrates graph-level features and non-graph structure state information to form a unified high-dimensional vector.

[0009] The second step is to generate microservice instance selection action decisions based on graph features: In order to meet the efficiency and legality requirements of microservice instance selection in edge computing scenarios, this step is based on the extracted fusion graph features and is geared towards the instance allocation scenario of IoT requests. It combines the Duel DQN architecture with the factorized noise network to optimize the action exploration method for microservice instance selection. The total action space is split according to the microservice dimension and a masking mechanism is designed. Adaptive multi-dimensional instance selection action vectors are generated through independent optimization of subspaces and action concatenation.

[0010] The third step is to record experience samples of interaction and classification in the edge computing environment: The efficiency of experience learning in graph reinforcement learning directly affects the training effect of the edge microservice instance selection model. In this step, a composite reward function that integrates request latency constraints and instance load balancing is designed in the interaction between the dynamic environment of edge computing and the microservice instance selection action. At the same time, a type priority experience replay pool is built based on the IoT request category and an effectiveness flag is added to improve the utilization efficiency of experience samples and the model's learning of scheduling rules for different types of requests.

[0011] The fourth step involves training and iteratively optimizing a reinforcement learning model for edge microservice instance selection based on multiple sets of experience. Parallel training of multi-class request experience can easily lead to gradient conflicts, affecting the convergence stability of the edge microservice instance selection model. This step utilizes categorized experience samples for model training, introduces gradient surgery techniques to correct conflicting gradients, and combines the DDQN algorithm with the target network soft update mechanism to suppress Q-value overestimation. Through the process design of batch sampling, loss calculation, parameter update, and soft update, the microservice instance selection decision logic of the agent is continuously optimized.

[0012] Furthermore, the process of the first step is as follows: Step (1.1) Define environmental status information; Edge computing platform Model it as an undirected graph: ,in, Represents a finite set of edge computing nodes, totaling One node; It is a finite set of edges defined by different node pairs, representing the network topology relationships between nodes, totaling... Edge; definition For IoT applications, each request consists of a triple. express, express The request call graph, Define a set of microservice tasks to be invoked for the node set; Define a set of directed edges to represent the call dependencies between microservices; Indicates the maximum tolerable delay for the request. Indicates the request category; definition For the IoT user equipment that issued the request, where, This indicates where to unload IoT requests, i.e., following the shortest path algorithm to select the nearest edge computing node for unloading; It refers to the time period during which the uninstallation request was made.

[0013] Furthermore, the first step also includes: Step (1.2) Intelligent Agent Perceive each time step Current environmental state The state information is then input into the Q network, which uses a neural network to fit the Q-value function, taking the environmental state as input and outputting the Q-value of each action in that state. Include Node topology and instance deployment status IoT requests for unloading Requester and a given time period Arrival rate within ; Step (1.3) for The included graph data describes the undirected graph of the edge computing platform. And the request call diagram describing the request. Modeling; Step (1.4) Graph attention feature extraction; Step (1.5) concatenates the ECP and RCG graph feature vectors with the remaining information to obtain the state feature vector representation.

[0014] Furthermore, the second step is as follows: Step (2.1) involves applying the state features obtained in step (1.5). The input is fed into a noisy network based on a duel-style DQN architecture. Decompose the Q-network architecture into a state value function. and action advantage function The two streams are then combined to output the Q-value; during each forward propagation, noise causes fluctuations in the Q-value estimate, thereby encouraging... Explore more movements, and the amount of noise injected will be continuously adjusted during training to conduct a more context-specific movement exploration. The fully connected layers in the Q-network are replaced with noisy fully connected layers. The noise is generated using a factorized noise generation method. The calculation formula for the noisy network is as follows: ,in, These are learnable parameters; It is random noise; Let the Hadamard product be represented, where the method for generating factorized noise refers to a... Input and The output fully connected layer, after factoring the noise, will generate two independent noise vectors, let the input noise be... The output noise is Then the parameter noise is: ,in, The noise value in the i-th row and j-th column of the weight matrix. Let j be the noise value of the j-th element of the bias vector. The signed square root function is defined as follows: This is used to map the original noise variable to noise components that meet the distribution requirements, in order to achieve low-rank decomposition of weighted noise, where... It is a symbolic function.

[0015] Preferably, the second step further includes: Step (2.2) The state-value stream results are output after processing by a noisy fully connected layer. and action advantage flow results ,in, ; Step (2.3) takes the result obtained in step (2.2) and The Q-values ​​are merged into the output Q-value matrix, where they are now randomized and represent the state. Next action The action value Q is equal to the value of the action Q. The state value calculated after extracting state features Add current action Advantage function The difference from the average of all available action advantage functions; Step (2.4) defines the action space. , The instance selection strategy for the request is a variable of length . The vector, For a by An IoT application composed of microservices; each microservice consists of a tuple. express, express The base number of exposures is The task set, express The set of instances and their relative positions on the edge computing platform, where, microservices Examples Placed on edge computing nodes superior, microservices The selection strategy is based on whether the current request involves the microservice. A value of 0 indicates that no instance is selected; otherwise, let Indicates the selection of instances ; Each microservice has independent action options, and the total action space is the Cartesian product of all microservice action options. Therefore, the size of the action space is... .

[0016] In a further preferred embodiment, the second step also includes: Step (2.5) defines the total space as follows: The action space is divided into subspaces: the total action space is decomposed according to the microservice dimension. Subspace ,Right now ， Each subspace The instance selection strategy corresponding to a single microservice, that is The element sub-actions Equivalent to The value of will Defined as not selecting any instance, the remaining sub-actions correspond sequentially to each instance of the microservice; Step (2.6) Action masking mechanism: Based on the microservice subset of the current request, generate a matrix with the same dimension as the Q-value matrix. Vector, if the request contains microservices ,make This indicates that the corresponding instance can be selected. In this case, the sub-actions in the sub-action space will be selected. This is used to prevent illegal situations where microservice instances are not selected; conversely, if it is not involved... Then let and only retain To prevent the selection of illegal instances; Step (2.7) Based on the action masking mechanism described in step (2.6), the Q-value matrix obtained in step (2.3) is compared with... Vectors are fused element by element, setting the Q-value of invalid actions to negative infinity, thus masking the Q-value of invalid actions and retaining the Q-value of valid actions; Step (2.8) involves selecting service instances within each subspace, with each subspace executing independently on the Q-value matrix retained in step (2.7). Operation: Select the sub-action with the largest Q value. ; Step (2.9) selects all sub-actions in the subspace. The outputs are combined sequentially to form the final multi-dimensional action vector, i.e., the action. Each subspace selects only one action, and the action space is a concatenation of these sub-actions. Therefore, the size of the action space becomes .

[0017] The process of the third step is as follows: Step (3.1) The agent will select the desired action. Execute in the environment; Step (3.2) Environment based on action The execution result returns an immediate reward. and new status ; Step (3.3) The experience from this interaction will be categorized according to the request type. Stored in the corresponding experience replay pool , ,in, This indicates that a priority experience replay pool was used, meaning the priority of the experience is determined by the timing difference error. The absolute value is determined, the larger the error, the higher the empirical priority, and the greater the sampling probability; It is a Boolean value that indicates the validity of the sampling; if If you can get an instant reward, then... Otherwise, set it to ; Update only after the request is completed; Step (3.4) Repeat steps (3.1)-(3.3) until all grouped experience storage pools have stored enough experience.

[0018] The process of the fourth step is as follows: Step (4.1) Batch sampling: from the experience replay pool for each request type A small batch of valid experiences were sampled from each sample, namely experience; Step (4.2) Calculate the objective and loss: Calculate the objective Q-value using the objective network and the DDQN algorithm. The target network is a neural network with the same structure as the Q network, and it is responsible for calculating a stable target Q value during model training. The training loss function is calculated based on the temporal difference error. Timing difference error This represents the difference between the current policy network's estimate of the Q-value of the state-action pair and the target Q-value. The training loss function is the expectation of the squared batch temporal difference error sampled from the experience replay pool. Step (4.3) Perform gradient surgery to correct gradient conflicts: Calculate the gradients of the loss functions from different request groups, i.e., different experience pools. If two gradients and If the cosine similarity is negative, then they are considered to be in conflict, and the gradient of a request group is projected onto an orthogonal plane of the gradients of any conflicting request groups. Step (4.4) updates the parameters of the Q-network using stochastic gradient descent. ; Step (4.5) Soft update of the target network: After each iteration, the parameters of the target network will receive a small portion of the changes in the Q network parameters. Soft update stabilizes the calculation of the target Q value by slowly synchronizing the target network with the current Q network parameters. Step (4.6) Repeat steps (4.1)-(4.5) of the training process until the model converges.

[0019] The beneficial effects of this invention are mainly reflected in: 1. Through the graph attention mechanism enhanced by edge features, the graph structure data such as edge node network topology, microservice dependencies, instance deployment status and real-time load are efficiently extracted. This captures the spatial relationship between nodes and incorporates key information such as edge network latency, making the system status representation more comprehensive and accurate. This provides a more reliable basis for instance selection and achieves collaborative optimization of request latency and instance load balancing. 2. By adopting a strategy of splitting the total action space according to the microservice dimension, the high-dimensional combined action space is decomposed into multiple low-dimensional subspaces, significantly reducing computational complexity; combined with The shielding mechanism accurately filters out invalid actions to avoid illegal decisions and invalid calculations, enabling the method to flexibly adapt to large-scale edge computing scenarios with a large number of microservices and a wide distribution of instances. 3. Introduce gradient surgery technique to correct gradient conflicts caused by different request groups through gradient projection, avoid parameter update oscillations during training, and ensure the consistency of model convergence; use a noisy network to replace the traditional The exploration mechanism enables more context-specific action exploration through parameter noise reduction, allowing intelligent agents to better adapt to the dynamics of edge environments. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the architecture of an edge latency-aware microservice instance selection model based on graph reinforcement learning.

[0021] Figure 2 This is a diagram illustrating subspace partitioning and illegal action blocking. Detailed Implementation

[0022] The present invention will now be further described with reference to the accompanying drawings.

[0023] Reference Figure 1 The overall architecture of the edge-latency-aware microservice instance selection method based on graph reinforcement learning proposed in this invention comprises five core components: a Q-Network, a Target Network, Replay Buffers, a Noisy Network, and a Gradient Surgery module. The model training process is as follows: First, the model observes the environment to obtain the current state and inputs it into the Q-Network (step ①). Then, it uses the output of the Q-Network to obtain an action (step ②). After executing this action, the model can obtain a new state (Next state) and a reward from the environmental feedback (step ③), and store them in the Replay Buffer as experience tuples (step ④). This interaction process is repeated until a sufficient number of experience tuples are available. Then, the model begins training while continuing the above interaction process. First, it samples a batch of samples from the Replay Buffer (step ⑤), then calculates the loss using the outputs of the Q-Network and the Target Network, and updates the parameters of the Q-Network after performing gradient surgery (step ⑥). Finally, after the Q network has been updated a certain number of times, the target network parameters are updated using a soft update mechanism (step ⑦). The environment consists of a system platform, i.e., an edge computing platform. It consists of microservice applications, IoT users (UDs), and IoT requests. (In the experience replay pool) This refers to the request category, as opposed to the patent. They have the same meaning.

[0024] Reference Figure 2 Assuming a scenario of abnormal monitoring requests for an IoT elevator, the microservices work in the order of Ms1-Ms2-Ms3-Ms6. The diagram shows 5 edge computing nodes, each deploying a different microservice instance, and each instance is labeled with its load latency. Includes 6 actions arrive Each action represents the selection of a microservice instance. For instance selection corresponding to Ms1, For Ms2 instance selection, For Ms3 instance selection, For Ms4 instance selection, For Ms5 instance selection, The instance selection corresponds to Ms6. As shown in the diagram, the request does not involve microservices Ms4 and Ms5, therefore... and The value is 0. Ms1 is deployed on ECn1, ECn2, and ECn5 respectively; Ms2 is deployed on ECn3 and ECn4; Ms3 is deployed on ECn2 and ECn4; and Ms6 is deployed on ECn2 and ECn5. Instance numbers are assigned according to the deployment order shown in the diagram, i.e., from left to right, they are numbered as instance 1, instance 2, ... Therefore, the first instance selection strategy... The corresponding action is The action corresponding to the second instance selection strategy is .

[0025] by Figure 2 Taking the IoT elevator request scenario shown below as an example, the edge latency-aware microservice instance selection method based on graph reinforcement learning in this embodiment includes the following steps: The first step, extracting graph structure features from the edge computing environment state awareness and edge node instance deployment graph, and the microservice request call graph, is as follows: Step (1.1) Define environmental status information; edge computing platform Model it as an undirected graph: , representing a finite set of edge computing nodes, totaling One node; It is a finite set of edges defined by different node pairs, representing the network topology relationships between nodes, totaling... Edge; definition This is a collection of requests for IoT applications. Each request consists of a triple. express, express The request call graph, For the node set, a set of microservice tasks to be invoked are defined; A set of directed edges is defined to represent the call dependencies between microservices. Indicates the maximum tolerable delay for the request. Indicates the request category; definition For the IoT user equipment that issued the request. This indicates where to unload IoT requests, i.e., following the shortest path algorithm to select the nearest edge computing node for unloading; This refers to the time period during which the unloading request is processed. For example, in an IoT elevator scenario, an anomaly detection request can be made by IoT user devices such as speed sensors, load sensors, or video surveillance equipment inside the elevator car. The requests generated by these devices need to sequentially call microservices such as data acquisition, anomaly detection, scheduling decision-making, and result feedback to complete the processing.

[0026] Step (1.2) Perceive each time step Current environmental state The state information is then input into the Q-network. The Q-network is a core learning-based neural network in reinforcement learning; essentially, it uses a neural network to fit a Q-value function, taking the environment state as input and outputting the Q-value of each action in that state. Include Node topology and instance deployment IoT requests for unloading Requester and a given time period Arrival rate within ( (Following the Poisson distribution) The formula is as follows: .

[0027] Step (1.3) for The included graph data describes the undirected graph of the edge computing platform. And the request call diagram describing the request. The modeling process is as follows; Step (1.3.1) integrates edge node topology, node features, edge features, and adjacency relationships to model the edge node topology and microservice instance deployment relationships as an undirected graph. Defined as... This is further referred to as the Instance Deployment Graph (IDG). For the set of node features, It is an adjacency matrix. Represents a node and nodes The value of 0 indicates that the network is not connected. This is a set of edge features. For example, in a city-level IoT elevator platform, a multi-level edge computing node cluster distributed across multiple business districts, residential areas, and transportation hubs can be abstracted as an edge computing node. These nodes are interconnected through a network to form a large-scale, complex, heterogeneous network topology, i.e., an instance deployment graph. Step (1.3.2) Construction Node characteristics: Represents the set of node feature vectors. The feature vector representing a node. This represents the node's feature count. This vector reflects the deployment and real-time load of the microservice instances on the current node. Specifically, Defined as a set containing multiple subsets, each subset using... This means that each subset contains several elements, denoted as . ,in It is the number of elements in the j-th subset. The entire... Includes from the first to the... All elements of a subset. It is a set The total number of elements is equal to the total number of all subsets. The sum of the number of elements. For element Corresponding Deployed on edge nodes When, Values ,otherwise The value is 0. The formula is as follows: ; Step (1.3.3) Construction Edge features: Let f(x) represent the set of edge features, where f(x) Represents a node and nodes The edge feature vectors between them It mainly reflects the network latency between nodes. If a node returns to itself, =0; when the edge node and The two sides belong to the edge set. When the edge is In other cases, the value is positive infinity, as shown in the following formula: ; Step (1.3.4) will process a single IoT request. The microservice dependency relationships are modeled as a directed graph, fully defined as a request-call graph. .in, Represents the set of node feature vectors. This represents the adjacency matrix. For example, in this embodiment, the IoT elevator anomaly detection request only includes four microservices: Ms1, Ms2, Ms3, and Ms6. Therefore, the request call graph only includes these four nodes and the call edges from Ms1 to Ms2 to Ms3 to Ms6, while Ms4 and Ms5 do not appear in the request call graph. Step (1.3.5) Construction Node characteristics: Represents the set of node feature vectors. Represents a node eigenvectors, This represents the number of node features. This vector reflects... The request chain includes the microservice task attributes and execution status. Node feature construction and steps (1.3.2) The nodes have the same feature structure.

[0028] Step (1.4) Graph attention feature extraction, the process is as follows: Step (1.4.1) Based on the graph convolutional network (GCN), the edge feature enhanced graph attention mechanism (EGAT) is introduced to extract features from ECP. The steps are as follows; Step (1.4.1.1) uses a learnable weight matrix. Node features Sum of edge features Perform a linear transformation, that is and Multiplication and Multiply to get and ; Step (1.4.1.2) is performed via... Activation function and attention vector compute nodes with neighbors Normalized attention weights ; Specifically, first, the nodes Transformed features ,node Transformed features And the transformed features of the edges connecting them. Perform a concatenation operation to obtain a long vector, then combine this long vector with the attention vector. Perform the dot product; then input the dot product result. Activation function, performing nonlinear transformation on nodes For all first-order neighbor nodes (including itself), calculate the results of the first two steps and take the exponent; finally, use the node... To the node The exponential result divided by the node The sum of the exponents of all first-order neighbor nodes yields the final normalized attention weights. The formula is as follows: ; in, Represents the attention weight vector. This indicates a splicing operation. Represents a non-linear activation function. Represents a node First-order neighbor nodes, including nodes itself.

[0029] Step (1.4.1.3) aggregates neighbor information based on attention weights to generate updated node features. Specifically, it is based on the calculated attention weights. For nodes All neighboring nodes The features are weighted and summed, and then passed through a nonlinear activation function. Introducing nonlinearity to generate updated node features The formula is as follows: ; Step (1.4.1.4) is a repeat of step (1.4.1.3). Layered convolution propagation, finally through global graph pooling. Node-level features Transform into graph-level features. ; ; Global graph pooling uses mean pooling, and the formula is as follows: , The total number of nodes; Step (1.4.2) uses the Graph Attention (GAT) mechanism to extract the RCG, and the steps are as follows: Step (1.4.2.1) uses the weight matrix. Transform node features to obtain ,Right now ; Step (1.4.2.2) is performed via... Activation function and attention vector compute nodes with neighbors Normalized attention weights Compared to the attention weight calculation in step (1.4.1.2), only the edge features are missing. The formula is as follows: ; Step (1.4.2.3) is a repeat of step (1.4.2.2). Layered convolution propagation, finally through global graph mean pooling. Node-level features Transform into graph-level feature vectors ,Right now .

[0030] Step (1.5) will and The graph feature vector is concatenated with the remaining information to obtain the state feature vector representation, as shown in the following formula: ; Among them, the remaining information This refers to information in step (1.2) that does not include graph data. Specifically: .

[0031] The second step, generating action decisions for microservice instance selection based on graph features, involves the following process: Step (2.1) inputs the state features obtained in step (1.5) into the duel-based system. In noisy networks with different architectures, the Q-network architecture is decomposed into state-value functions. and action advantage function Two streams are combined to output the Q-value. A noisy network is an exploration strategy based on parameter noise reduction. The exploration in a noisy network is embedded within the network. During each forward propagation, noise causes fluctuations in the Q-value estimate, thus encouraging... Explore more actions. The amount of noise injected is continuously adjusted during training, thus enabling more context-specific action exploration. In standard DQN, this is typically achieved using... ,by The probability of choosing the action with the highest Q value or the action with the highest Q value Actions are randomly selected with probability to generate training data. However, in microservice instance selection scenarios, due to the complexity and dynamism of the environment, It doesn't seem quite appropriate; This invention replaces the fully connected layers in the Q-network with noisy fully connected layers, and the noise is generated using a factorized noise generation method. The calculation formula for the noisy network is as follows: .in These are learnable parameters; It is random noise; This represents the Hadamard product. The method for generating factorized noise refers to the process of generating noise for a given input. and output In a fully connected layer, factoring the noise will generate two independent noise vectors, let the input noise be... The output noise is Then the parameter noise is: .in The noise value in the i-th row and j-th column of the weight matrix. Let j be the noise value of the j-th element of the bias vector. The signed square root function is defined as follows: This is used to map the original noise variable to noise components that meet distribution requirements, thereby achieving low-rank decomposition of weighted noise. It is a symbolic function.

[0032] Step (2.2) The state-value stream results are output after processing by a noisy fully connected layer (Noisy MLP). In scalar form, and action-dominant flow results It is in matrix form. .

[0033] Step (2.3) takes the result obtained in step (2.2) and The results are merged into the output Q-value matrix. The Q-value of this invention refers to the expected reward, or action value, of performing an action according to a strategy in a given state. This Q-value contains a degree of randomness. In the state... Next action The action value Q is equal to the value of the action Q. The state value calculated after extracting state features Add current action Advantage function The difference between this value and the average of the advantage functions of all available actions. The formula is as follows: ; Step (2.4) defines the action space. The instance selection strategy for the request is a variable of length . The vector. For a by An IoT application composed of microservices. Each microservice consists of a tuple. express, express The base number of exposures is The task set, express The set of instances and their relative locations on the edge computing platform, where microservices Examples Placed on edge computing nodes superior. microservices The selection strategy is based on whether the current request involves the microservice. A value of 0 indicates that no instance is selected; otherwise, let , indicating the selection of instances . The formula is as follows: ; Each microservice has independent action options, and the total action space is the Cartesian product of all microservice action options. Therefore, the size of the action space is... .

[0034] Step (2.5) defines the total space as follows: The action space is divided into subspaces: the total action space is decomposed according to the microservice dimension. Subspace ,Right now Each subspace The instance selection strategy corresponding to a single microservice, that is The element Sub-action Equivalent to The value to be taken. Defined as not selecting any instance, the remaining sub-actions correspond sequentially to each instance of the microservice. In this embodiment, the action space is divided into 6 subspaces, corresponding to Ms1-Ms6 respectively, and each subspace independently completes instance selection.

[0035] Step (2.6) Action Shielding Mechanism: To avoid the selection of illegal instances, this invention designs an action shielding mechanism. Specifically, based on the microservice subset of the current request, an action shielding matrix with the same dimension as the Q-value matrix is ​​generated. Vector. If the request contains microservices. ,make This indicates that the corresponding instance can be selected. In this case, the sub-actions in the sub-action space will be selected. This is to prevent illegal situations where microservice instances are not selected. Conversely, if it is not involved... Then let and only retain This is to prevent the selection of illegal instances. Its definition is as follows: ; In this embodiment, the IoT elevator anomaly detection request does not involve Ms4 and Ms5. Therefore, only action 0 is retained in the corresponding two subspaces, and all other actions are masked.

[0036] Step (2.7) According to the action masking mechanism described in step (2.6), the Q-value matrix and vector obtained in step (2.3) are fused element by element, the Q-value of invalid actions is set to negative infinity, the Q-value of invalid actions is masked, and the Q-value of valid actions is retained.

[0037] Step (2.8) involves selecting service instances within each subspace, with each subspace executing independently on the Q-value matrix retained in step (2.7). Operation: Select the sub-action with the largest Q value. .

[0038] Step (2.9) selects all sub-actions in the subspace. The outputs are combined sequentially to form the final multi-dimensional action vector, i.e., the action. Each subspace selects only one action, and the action space is a concatenation of these sub-actions. Therefore, the size of the action space becomes... .

[0039] The third step involves recording experience samples from edge computing environment interactions and classification. The process is as follows: Step (3.1) The agent will select the desired action. Execution occurs within the environment. For example, the agent selects actions based on the generated instance, scheduling IoT elevator anomaly detection requests to the corresponding microservice instances on the edge nodes for execution.

[0040] Step (3.2) Environment based on action The execution result returns an immediate reward. and new status ; Defined as timeout reward and load balancing rewards The composite function, binary variables in Used to measure whether request latency is satisfied. constraint. The constraint is that the request must be processed within the specified deadline. hour If the request is deemed a failure, a negative reward is given; otherwise, a positive reward is given. Defined as when the action is performed, If the value decreases, a positive reward is given; otherwise, a negative reward is given. To request end-to-end delay, It consists of three parts . This indicates the access latency of IoT user devices to nearby edge computing nodes; This represents the routing latency caused by microservice instances crossing nodes during request processing, which is limited by the dependencies between microservice instances and the geographical location of their respective nodes. Specifically, if the selected entry microservice instance is not on the node unloading the request, or if the collaborating microservice instances are not on the same node, additional routing latency will be introduced. This latency is determined by the routing protocol based on the shortest path algorithm; otherwise, the relevant latency can be ignored. This indicates the task processing latency of a microservice instance; The load balancing score is calculated as follows: First, for each microservice instance, the total number of load requests it handles is counted, and based on the specific load values ​​of these requests, the average load of that instance is calculated. Then, using this average as a benchmark, the variance of the load values ​​of all load requests for that instance is calculated, and the square root of the variance is taken to obtain the standard deviation of the instance's load, reflecting the degree of load fluctuation within a single instance. Next, the standard deviations of the loads of all microservice instances are summed, and then divided by the total number of instances to obtain the average standard deviation of the load. This final result is the load balancing score.

[0041] The formula is as follows: ; ; in, for of Resource utilization rate for The average resource utilization of all instances reflects the overall load balancing of the system. The smaller the value, the better the load distribution among microservice instances. express and The balance factor can be adjusted to suit the different preferences of IoT applications.

[0042] Step (3.3) The experience from this interaction will be categorized according to the request type. Stored in the corresponding experience replay pool . ,in This indicates that a priority experience replay pool was used, meaning the priority of the experience is determined by the timing difference error. The absolute value is determined, and the larger the error, the higher the empirical priority and the greater the sampling probability. This is a Boolean value indicating the validity of the sampling. If the agent can obtain an immediate reward, then let... Otherwise, set it to ; The update continues until the request is completed. For example, after the IoT elevator anomaly detection request is processed, its status, action, reward, and next status together form an experience sample, which is then stored in the corresponding experience replay pool according to the request category.

[0043] Step (3.4) Repeat steps (3.1)-(3.3) until all grouped experience storage pools have stored enough experience.

[0044] The fourth step involves training and iteratively optimizing a graph reinforcement learning model for edge microservice instance selection based on multiple sets of experience. The process is as follows: Step (4.1) Batch sampling: from the experience replay pool for each request type A small batch of valid experiences were sampled from each sample, namely Experience.

[0045] Step (4.2) Calculate the target and loss: Use the target network and the Double Deep Q-Network (DDQN) algorithm. The target network is a neural network with the same structure as the Q-network, responsible for calculating a stable target Q-value during model training. The DDQN algorithm is a reinforcement learning algorithm that uses two independent Q-networks to handle "action selection" and "value estimation" respectively. Through decoupling optimization, it solves the overestimation problem of the basic DQN, making model training more stable and decision-making more reliable. Calculate the target Q-value. The formula is as follows: ; in, This indicates that the action with the largest Q value is selected using the Q-network, and the target network is used. Re-evaluate the Q-value of the action. Temporal difference error represents the difference between the current policy network's estimate of the Q-value of the state-action pair and the target Q-value, used to measure the degree of bias in the model's prediction. The training loss function is the expectation of the squared batch temporal difference errors sampled from the experience replay pool. The formulas for temporal difference error and training loss are as follows: .

[0046] Step (4.3) Perform gradient surgery to correct gradient conflicts: Calculate the gradients of the loss functions from different request groups, i.e., different experience pools. If two gradients and If the cosine similarity of the requesting groups is negative, they are considered to be in conflict. To mitigate this conflict, the gradient of a requesting group is projected onto an orthogonal plane of the gradients of any conflicting requesting groups. This projection effectively neutralizes the conflicts in scale and direction, making training more balanced and stable. The projection formula is as follows: ; in, After requesting grouping, retrieve from the experience replay pool. Training loss from mid-sample playback This indicates the gradient corrected by gradient surgery.

[0047] Step (4.4) updates the parameters of the Q-network using stochastic gradient descent. : ; Step (4.5) Soft update of the target network: After each iteration, the parameters of the target network will receive a small change in the Q-network parameters. Soft update stabilizes the calculation of the target Q-value by slowly synchronizing the target network with the current Q-network parameters, as shown in the following formula: ; in, It updates the parameters; Step (4.6) Repeat steps (4.1)-(4.5) of the training process until the model converges.

[0048] This embodiment analyzes the actual effects of the present invention using specific data, as follows: 1) Experimental setup First, this invention uses IFogSim2 to build an edge computing simulation platform. IFogSim2 is a Java-based toolkit for modeling and simulating resource management technologies in IoT, edge, and fog computing environments. Building upon the existing IFogSim simulation toolkit, IFogSim2 further integrates multiple simulation models related to service migration, dynamic distributed cluster formation, and microservice orchestration in edge and fog computing, making it highly suitable for the experiments presented in this invention. Secondly, IFogSim2 supports the integration of real-world datasets, such as the EUA dataset; This invention organizes seven different request structures and sizes, encompassing eight microservices, to simulate IoT applications. In request simulation, this invention follows a arrival rate... The Poisson distribution is observed. Secondly, to establish appropriate deadlines for various requests within the edge computing platform, this invention examines the average service latency for each request. and multiply it by the appropriate deadline factor. To establish a reasonable deadline ; Finally, the model was implemented using PyTorch and integrated into the platform. All experiments were conducted on a server running Windows 11, equipped with an AMD R9-7945HX CPU workstation with a frequency of 2.50 GHz, 16 GB of memory, and an NVIDIA GeForce RTX 4060 GPU.

[0049] After completing the above configuration, the next step is to optimize the compound reward balance factor. Five different Conduct training, and take respectively Multiple rounds of comparative experiments were conducted to verify the different values ​​of 0, 0.2, 0.5, 0.8, and 1. Value effect.

[0050] Table 1 shows the experimental results;

[0051] The data analysis above shows that when At this time, the model can only perceive the feedback from load balancing rewards, which reduces the success rate of requests because the model doesn't know whether the selected low-load instance group can meet the request's deadline constraint. In a scenario of distributed instance deployment, only perceiving load signals can lead to significant routing latency, resulting in request failures. Conversely, when... At this point, the request success rate is highest because the model knows which decisions will lead to a successful request, but it doesn't further perceive load signals. For example, when multiple feasible decisions exist, the model doesn't tend to choose the low-load instance group, causing a decrease in system load balancing. When the value is between 0 and 1, the model can simultaneously sense both signals and make comprehensive optimization considerations. The larger the value, the more the model will favor latency; The smaller the value, the more it favors load balancing.

[0052] Based on the above experimental setup and reward factor configuration, further hyperparameter experiments were conducted to optimize the model's convergence effect and runtime performance.

[0053] Table 2 shows the different numbers of convolutional layers. Table showing the impact on model request success rate and load balancing.

[0054] This affects the representation of the state feature vector; rich feature information helps the agent make better and more accurate decisions. However, as... As the number of nodes increases, the computational overhead will increase, and excessive neighborhood information will also cause the node representation to lose locality, leading to the problem of over-smoothing.

[0055] The table shows that the request success rate is... When it reaches a stable state, as Increasing the value to 3 or 4 did not result in a significant performance improvement; in fact, it led to... There is a certain degree of decrease, which may be a result of over-smoothing of node feature representations. When When the load balancing metric is increased from 1 to 2, there is a significant improvement, and then as... The increase in [something] exhibits certain fluctuations. Ultimately, considering both performance and computational cost, this invention selected [something]. This is the optimal setting for the experiment.

[0056] 2) Compare to baseline To better verify the advancement of the proposed method, this invention first proposes two basic algorithms, which greedily consider both distance and load: Greedy distance algorithm: For each microservice involved, select the microservice instance that is closest to it. If there are multiple candidate instances, then consider selecting the instance with the lowest load.

[0057] Greedy workload algorithm: For each microservice involved, select the microservice instance with the lowest load. If there are multiple candidate instances, then consider selecting the instance that is closest to the target microservice.

[0058] Next, the present invention selects the following existing methods, which solve similar problems to those of the present invention: MSS (Microservice Service Selection): This algorithm addresses the service selection problem in microservice systems by proposing a list-based scheduling approach and a strategy for resolving competition among multiple composite services (CSs) in workflow scheduling. Firstly, before selecting a service, the algorithm calculates the urgency of tasks to arrange the execution order of tasks within the CSs. Secondly, when selecting a service instance for each task, the algorithm estimates the QoS provided by the instance and discards timed-out tasks and their CSs based on the task's sub-deadline. Simultaneously, the algorithm continuously updates the task urgency and searches for idle time intervals to adjust the task execution order.

[0059] MS_DDPG (Microservice Selection Based on Deep Deterministic Policy Gradient): To address the microservice selection and scheduling optimization problem in cloud-edge-device environments, an improved DDPG algorithm is proposed to learn microservice selection strategies to minimize user service access latency. Simultaneously, this algorithm also establishes an experience replay pool for better training.

[0060] MISP (Microservice Instance Selection Problem): An improved DDQN algorithm used to select appropriate physical service plans, i.e., instance composition, for microservice-based IoT requests in dynamic fog environments, to simultaneously optimize request latency and load balancing. Secondly, for action selection, the algorithm employs both Q-Mapping and Q-Masking methods to address the variability of the action space.

[0061] DRLMCF (Deep Reinforcement Learning for Microservice Composition in Fog Computing) is an improved DRL-based fog layer microservice chaining algorithm for enhancing the composition of distributed chained microservices in relevant IoT applications, thereby minimizing resource utilization and latency in serverless architectures based on fog computing. This algorithm does not rely entirely on a central controller; fog nodes also participate in decision-making, making it scalable in IoT environments.

[0062] DDRLMSE (Distributed Deep Reinforcement Learning-Based Microservice Selection in Edge Computing) is a distributed DRL algorithm based on A3C (Application-Specific Logic Buffer) used to allocate suitable available microservice instances to a microservice chain for IoT applications in a serverless cloud-edge environment, minimizing resource utilization and latency. The algorithm trains multiple agents, each of which is responsible for sensing information about the current edge computing node and its first-order neighbors, and passing a minimal subset of the actual data from each interval to the global network to update the model.

[0063] GRL-MIS (Graph Reinforcement Learning-Based Edge Delay-Aware Microservice Instance Selection): The method of this invention.

[0064] 3) Experimental Results The performance experiment results are shown in Table 3, with the best results highlighted in bold. The table clearly shows that the distance-first greedy algorithm and the load-first greedy algorithm use relatively simple rules to greedily select microservice instances, resulting in the worst performance. The MSS method considers deadlines and urgency, carefully scheduling microservices, making its results better than the Greedy algorithm. However, compared to DRL-type algorithms, their performance is inferior because DRL algorithms can interact with the environment and learn knowledge, making them more suitable for dynamic and heterogeneous IoT environments. Thanks to the powerful perception and decision-making capabilities of DRL, the four methods MS_DDPG, MISP, DRLMCF, and DDRLMSE all achieved better results. MS_DDPG only considers service latency and does not consider the dependencies between microservices, resulting in inferior performance compared to the other three. Compared to MISP, the DRLMCF method does not rely entirely on a central controller; fog nodes also participate in decision-making, improving its performance. DDRLMSE, based on DRLMCF, performs distributed modifications to the agents, constructing a multi-agent collaborative distributed DRL model, further improving the model's performance. However, these methods are all inferior to the GRL-MIS method proposed in this invention.

[0065] Table 3 shows the performance test results;

[0066] Table 4 shows the ablation experiments of GRL-MIS. GRL-MIS-1 removed the attention module from the GCN; GRL-MIS-2 removed the action space partitioning; GRL-MIS-3 removed the gradient surgery module; and GRL-MIS-4 removed the noisy network module. The experimental results are shown in the table below, with the optimal results highlighted in bold. The parameter values ​​of the Q-network in the GRL-MIS-2 and GRL-MIS models are specifically listed for reference.

[0067] Table 4 shows the ablation experiments of GRL-MIS;

[0068] Table 4 shows that GRL-MIS-1, by removing the graph attention module, experienced a significant performance decrease. This is because treating all neighbors equally ignores the importance of different neighbors to the central node, resulting in a loss of accuracy in the node's feature representation. Furthermore, these errors are further amplified with convolution. While GRL-MIS-2's performance is similar to GRL-MIS, GRL-MIS improves the action space explosion problem, reduces the number of parameters and model size, and achieves faster convergence, proving that action space partitioning is a beneficial improvement. The introduction of gradient surgery optimizes the model's convergence process. By correcting conflicting gradients, it ensures optimization consistency, thereby achieving better performance. Therefore, after removing gradient surgery, the performance of GRL-MIS-3 decreased. GRL-MIS-4 removed the noisy network and reused... The greedy algorithm, when used for action exploration and utilization, suffers from performance degradation. The advantage of noisy networks lies in their ability to automatically adjust the amount of noise injected into the network parameters, thus providing a more targeted action exploration mechanism that adjusts based on training.

[0069] Table 5 presents the robustness experiments of the model parameters for GRL-MIS and GRL-MIS-4. Specifically, Gaussian noise was added to the parameters of each fully connected layer during testing, where the parameters were fixed. The value is 0, and the values ​​are used to simulate different intensities of noise injection.

[0070] Table 5 shows the robustness experiments of the model parameters for GRL-MIS and GRL-MIS-4;

[0071] As can be seen from Table 5, in At this point, the model's performance degradation reached its maximum. Compared to the case without added noise, the request success rates of GRL-MIS and GRL-MIS-4 decreased by 10.57% and 62.96%, respectively, with GRL-MIS-4 experiencing a significant 52.39% decrease. Secondly, it was found that after the noise intensity exceeded a certain threshold, such as... The model's performance no longer declines but remains at a certain level. This may be because the model has reached its limit under extreme noise conditions, meaning it cannot extract any useful information from the noise and can only output random or meaningless results. Even so, GRL-MIS can maintain a high request success rate after using a noisy network.

[0072] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for edge-latency-aware microservice instance selection based on graph reinforcement learning, characterized in that, The method includes the following steps: The first step involves constructing a two-layer graph structure for edge node instance deployment and request invocation, applying a feature-enhanced graph attention mechanism to the feature extraction of edge computing platform topology, microservice instance deployment, and IoT request invocation dependencies, and fusing graph-level features with non-graph structure state information to form a unified high-dimensional vector. The second step is to combine the Duel DQN architecture with the factorized noise network to optimize the action exploration method for microservice instance selection, based on the extracted fusion graph features and for the instance allocation scenario of IoT requests. The total action space is split according to the microservice dimension and a mask shielding mechanism is designed. Adapted multi-dimensional instance selection action vectors are generated by independent optimization of subspaces and action splicing. The third step involves designing a composite reward function that integrates request latency constraints and instance load balancing in the interaction between the dynamic environment of edge computing and the microservice instance selection action. At the same time, a categorized priority experience replay pool is built based on IoT request categories, and an effectiveness flag is added. The fourth step involves using categorized experience samples to train the model, introducing gradient surgery techniques to correct conflicting gradients, and combining the DDQN algorithm with the target network soft update mechanism to suppress Q-value overestimation. Through the process design of batch sampling, loss calculation, parameter update and soft update, the microservice instance selection decision logic of the agent is continuously optimized.

2. The edge latency-aware microservice instance selection method based on graph reinforcement learning as described in claim 1, characterized in that, The process of the first step is as follows: Step (1.1) Define environmental status information; Edge computing platform Model it as an undirected graph: ,in, Represents a finite set of edge computing nodes, totaling One node; It is a finite set of edges defined by different node pairs, representing the network topology relationships between nodes, totaling... Edge; definition For IoT applications, each request consists of a triple. express, express The request call graph, Define a set of microservice tasks to be invoked for the node set; Define a set of directed edges to represent the call dependencies between microservices; Indicates the maximum tolerable delay for the request. Indicates the request category; definition For the IoT user equipment that issued the request, where, This indicates where to unload IoT requests, i.e., following the shortest path algorithm to select the nearest edge computing node for unloading; It refers to the time period during which the uninstallation request was made.

3. The edge latency-aware microservice instance selection method based on graph reinforcement learning as described in claim 2, characterized in that, The first step also includes: Step (1.2) Intelligent Agent Perceive each time step Current environmental state The state information is then input into the Q network, which uses a neural network to fit the Q-value function, taking the environmental state as input and outputting the Q-value of each action in that state. Include Node topology and instance deployment IoT requests for unloading Requester and a given time period Arrival rate within ; Step (1.3) for The included graph data describes the undirected graph of the edge computing platform. And the request call diagram describing the request. Modeling; Step (1.4) Graph attention feature extraction; Step (1.5) concatenates the ECP and RCG graph feature vectors with the remaining information to obtain the state feature vector representation.

4. The edge latency-aware microservice instance selection method based on graph reinforcement learning as described in claim 3, characterized in that, The process of step (1.3) is as follows: Step (1.3.1) integrates edge node topology, node features, edge features, and adjacency relationships to model the edge node topology and microservice instance deployment relationship as an undirected graph: defined as... This is further referred to as the Instance Deployment Graph (IDG), where... For the set of node features, It is an adjacency matrix. Represents a node and nodes For China Unicom, a value of 0 indicates no connection. It is a set of edge features; Step (1.3.2) Construction Node characteristics: Represents the set of node feature vectors. The feature vector representing a node. The number of node features; Step (1.3.3) Construction Edge features: Let f(x) represent the set of edge features, where f(x) Represents a node and nodes The edge feature vectors between them Reflects network latency between nodes ; Step (1.3.4) will process a single IoT request. The microservice dependency relationships are modeled as a directed graph, fully defined as a request-call graph. ,in, Represents the set of node feature vectors. Represents the adjacency matrix; Step (1.3.5) Construction Node characteristics: Represents the set of node feature vectors. Represents a node eigenvectors, Let be the number of node features, where Node feature construction and steps (1.3.2) The nodes have the same feature structure.

5. The edge latency-aware microservice instance selection method based on graph reinforcement learning as described in claim 3, characterized in that, The process of step (1.4) is as follows: Step (1.4.1) uses the graph attention mechanism EGAT, which enhances edge features based on the graph convolutional network GCN, to extract features from ECP. The steps are as follows: Step (1.4.1.1) uses a learnable weight matrix. Node features Sum of edge features Perform a linear transformation, that is and Multiplication and Multiply to get and ; Step (1.4.1.2) is performed via... Activation function and attention vector compute nodes with neighbors Normalized attention weights ; Step (1.4.1.3) aggregates neighbor information based on attention weights to generate updated node features. Based on the calculated attention weights For nodes All neighboring nodes The features are weighted and summed, and then passed through a nonlinear activation function. Introducing nonlinearity to generate updated node features ; Step (1.4.1.4) is a repeat of step (1.4.1.3). Layered convolution propagation enables node features to be fused with information from higher-order neighbors, and finally global graph pooling is applied. Node-level features Transform into graph-level features ; Step (1.4.2) uses the graph attention mechanism GAT to extract the RCG, and the steps are as follows: Step (1.4.2.1) uses the weight matrix. Transform node features to obtain ; Step (1.4.2.2) is performed via... Activation function and attention vector compute nodes with neighbors Normalized attention weights ; Step (1.4.2.3) is a repeat of step (1.4.2.2). Layered convolution propagation, finally through global graph mean pooling. Node-level feature vectors Transform into graph-level features ,Right now .

6. The edge latency-aware microservice instance selection method based on graph reinforcement learning as described in any one of claims 2 to 5, characterized in that, The second step is as follows: Step (2.1) involves applying the state features obtained in step (1.5). The input is fed into a noisy network based on a duel-style DQN architecture. Decompose the Q-network architecture into a state value function. and action advantage function The two streams are then combined to output the Q-value; during each forward propagation, noise causes fluctuations in the Q-value estimate, thereby encouraging... Explore more movements, and the amount of noise injected will be continuously adjusted during training to conduct a more context-specific movement exploration. The fully connected layers in the Q-network are replaced with noisy fully connected layers. The noise is generated using a factorized noise generation method. The calculation formula for the noisy network is as follows: ,in, These are learnable parameters; It is random noise; Let the Hadamard product be represented, where the method for generating factorized noise refers to a... Input and The output fully connected layer, after factoring the noise, will generate two independent noise vectors, let the input noise be... The output noise is Then the parameter noise is: ,in, The noise value in the i-th row and j-th column of the weight matrix. Let j be the noise value of the j-th element of the bias vector. The signed square root function is defined as follows: This is used to map the original noise variable to noise components that meet the distribution requirements, in order to achieve low-rank decomposition of weighted noise, where... It is a symbolic function.

7. The edge latency-aware microservice instance selection method based on graph reinforcement learning as described in claim 6, characterized in that, The second step also includes: Step (2.2) The state-value stream results are output after processing by a noisy fully connected layer. and action advantage flow results ,in, ; Step (2.3) takes the result obtained in step (2.2) and The Q-values ​​are merged into the output Q-value matrix, where they are now randomized and represent the state. Next action The action value Q is equal to the value of the action Q. The state value calculated after extracting state features Add current action Advantage function The difference from the average of all available action advantage functions; Step (2.4) defines the action space. , The instance selection strategy for the request is a variable of length . The vector, For a by An IoT application composed of microservices; each microservice consists of a tuple. express, express The base number of exposures is The task set, express The set of instances and their relative positions on the edge computing platform, where, microservices Examples Placed on edge computing nodes superior, microservices The selection strategy is based on whether the current request involves the microservice. A value of 0 indicates that no instance is selected; otherwise, let Indicates the selection of instances ; Each microservice has independent action options, and the total action space is the Cartesian product of all microservice action options. Therefore, the size of the action space is... .

8. The edge latency-aware microservice instance selection method based on graph reinforcement learning as described in claim 7, characterized in that, The second step also includes: Step (2.5) defines the total space as follows: The action space is divided into subspaces: the total action space is decomposed according to the microservice dimension. Subspace ,Right now ， Each subspace The instance selection strategy corresponding to a single microservice, that is The element sub-actions Equivalent to The value of will Defined as not selecting any instance, the remaining sub-actions correspond sequentially to each instance of the microservice; Step (2.6) Action masking mechanism: Based on the microservice subset of the current request, generate a matrix with the same dimension as the Q-value matrix. Vector, if the request contains microservices ,make This indicates that the corresponding instance can be selected. In this case, the sub-actions in the sub-action space will be selected. This is used to prevent illegal situations where microservice instances are not selected; conversely, if it is not involved... Then let and only retain To prevent the selection of illegal instances; Step (2.7) Based on the action masking mechanism described in step (2.6), the Q-value matrix obtained in step (2.3) is compared with... Vectors are fused element by element, setting the Q-value of invalid actions to negative infinity, thus masking the Q-value of invalid actions and retaining the Q-value of valid actions; Step (2.8) involves selecting service instances within each subspace, with each subspace executing independently on the Q-value matrix retained in step (2.7). Operation: Select the sub-action with the largest Q value. ; Step (2.9) selects all sub-actions in the subspace. The outputs are combined sequentially to form the final multi-dimensional action vector, i.e., the action. Each subspace selects only one action, and the action space is a concatenation of these sub-actions. Therefore, the size of the action space becomes .

9. The edge latency-aware microservice instance selection method based on graph reinforcement learning as described in any one of claims 1 to 5, characterized in that, The process of the third step is as follows: Step (3.1) The agent will select the desired action. Execute in the environment; Step (3.2) Environment based on action The execution result returns an immediate reward. and new status ; Step (3.3) The experience gained from this interaction will be categorized according to the request type. Stored in the corresponding experience replay pool , ,in, This indicates that a priority experience replay pool was used, meaning the priority of the experience is determined by the timing difference error. The absolute value is determined, the larger the error, the higher the empirical priority, and the greater the sampling probability; It is a Boolean value that indicates the validity of the sampling; if If you can get an instant reward, then... Otherwise, set it to ; Update only after the request is completed; Step (3.4) Repeat steps (3.1)-(3.3) until all grouped experience storage pools have stored enough experience.

10. The edge latency-aware microservice instance selection method based on graph reinforcement learning as described in any one of claims 1 to 5, characterized in that, The process of the fourth step is as follows: Step (4.1) Batch sampling: from the experience replay pool for each request type A small batch of valid experiences were sampled from each sample, namely experience; Step (4.2) Calculate the objective and loss: Calculate the objective Q-value using the objective network and the DDQN algorithm. The target network is a neural network with the same structure as the Q network, and it is responsible for calculating a stable target Q value during model training. The training loss function is calculated based on the temporal difference error. Timing difference error This represents the difference between the current policy network's estimate of the Q-value of the state-action pair and the target Q-value. The training loss function is the expectation of the squared batch temporal difference error sampled from the experience replay pool. Step (4.3) Perform gradient surgery to correct gradient conflicts: Calculate the gradients of the loss functions from different request groups, i.e., different experience pools. If two gradients and If the cosine similarity is negative, then they are considered to be in conflict, and the gradient of a request group is projected onto an orthogonal plane of the gradients of any conflicting request groups. Step (4.4) updates the parameters of the Q-network using stochastic gradient descent. ; Step (4.5) Soft update of the target network: After each iteration, the parameters of the target network will receive a small portion of the changes in the Q network parameters. Soft update stabilizes the calculation of the target Q value by slowly synchronizing the target network with the current Q network parameters. Step (4.6) Repeat steps (4.1)-(4.5) of the training process until the model converges.

Images