A knowledge graph enhanced service deployment and request distribution method in a cloud-edge environment

Abstract

This invention proposes a method for service deployment and request distribution with knowledge graph enhancement in a cloud-edge environment. First, it constructs an edge service deployment model, a request distribution model, and a service processing latency model, and then uses a weighted combination of latency and cost to construct a quantitative representation model of the overall system overhead. Second, under constraints such as storage capacity and distribution consistency, it constructs a joint optimization mathematical model for service deployment and request distribution, focusing on the joint optimization of service latency and deployment cost. It also constructs an edge computing multi-agent knowledge graph (EC-MAKG), solidifying the inherent attributes and prior constraints of servers, services, and requests through triples. Finally, it transforms the optimization problem into a knowledge graph enhancement multi-agent restricted Markov decision process, designing a state space for knowledge enhancement, an action space for knowledge constraints, and reward and constraint functions for knowledge enhancement. This invention achieves efficient joint decision-making for service deployment and request distribution with knowledge enhancement in edge computing scenarios.

Description

Technical Field

[0001] This invention relates to mobile edge computing technology, multi-agent deep reinforcement learning, and knowledge graphs, specifically to a method for service deployment and request distribution of knowledge graph enhancement in a cloud-edge environment. Background Technology

[0002] Edge computing, by deploying computing resources at the network edge, can effectively reduce service processing latency and improve user experience quality. In multi-agent edge computing scenarios, edge servers can distribute requests to alleviate single-point resource constraints and uneven system load. However, optimizing service deployment and request distribution to ensure service quality and resource utilization efficiency still faces multiple challenges. First, achieving joint optimization of service deployment and request distribution is difficult under conditions of heterogeneous and limited storage and computing resources on edge servers and dynamic arrival of service requests. It should be noted that user requests can only be processed by edge servers with corresponding service instances deployed, and the limited storage capacity of each server cannot deploy all service types simultaneously. If service deployment decisions are inappropriate, not only will some requests not be processed because the target server does not have the required service deployed, but it may also result in wasted resources due to deployed services but no requests arriving. In addition, there is a significant cost difference between service cold starts and warm starts. Frequent deployment migrations will introduce high initialization overhead, while excessively long residency may increase maintenance resource consumption. How to strike a balance between the two profoundly affects the overall system performance. Secondly, in multi-agent decision-making, each edge server acts as an independent decision-making entity, with a strong coupling between its service deployment and request distribution actions. Specifically, the feasibility of the request distribution strategy directly depends on whether the target server has already deployed the corresponding service, while the effectiveness of the deployment strategy depends on the actual scheduling effect of subsequent request distribution. This coupling characteristic easily leads to policy oscillations and ineffective exploration in dynamic environments, significantly increasing the difficulty of collaborative optimization. Finally, under multiple constraints such as storage capacity, computing power, and distribution consistency, jointly solving discrete service deployment decisions and continuous request distribution decisions is highly complex. This problem is essentially a mixed-integer nonlinear programming problem, and traditional optimization methods struggle to efficiently obtain the global optimum. Existing research has focused on minimizing service latency or optimizing resource allocation. However, these works mostly focus on solitary subproblems or single-agent decision-making scenarios, failing to effectively address the low exploration efficiency and policy instability caused by the lack of structured cognition in multi-agent environments. While deep reinforcement learning has shown potential in this field, traditional multi-agent reinforcement learning methods lack structured modeling of entity associations and collaborative experiences in the environment, making it difficult to achieve efficient policy convergence while satisfying complex constraints. During training, problems such as slow convergence and frequent constraint violations are likely to occur. Summary of the Invention

[0003] This invention addresses the challenges faced by joint optimization of service deployment and request distribution in edge computing multi-agent collaborative scenarios, including dynamic requests, resource constraints, and multi-agent decision-making. Traditional multi-agent reinforcement learning methods suffer from frequent ineffective explorations, policy oscillations, and slow convergence due to a lack of explicit modeling of structured relational information in the environment. Considering the heterogeneous and limited resources of edge servers, the dynamic diversity of service requests, and the significant differences in service cold / hot start costs in edge computing networks, this invention constructs an edge computing multi-agent knowledge graph EC-MAKG to structurally represent the inherent attributes and collaborative experience of servers, services, and requests. It then utilizes a relational graph convolutional network R-GCN to extract knowledge embedding features and action masking. Based on this, and combined with secure deep reinforcement learning, a knowledge graph-driven multi-agent security-enhanced near-end policy optimization algorithm, KG-S-MAPPO, is proposed. The algorithm adopts a centralized training-distributed execution framework. In the distributed execution phase, each edge server agent autonomously generates service deployment and request distribution decisions based on its local state and knowledge embedding features. In the centralized training phase, collaborative strategy optimization is achieved through a knowledge-enhanced global Critic network and a Lagrange augmented objective constraint optimization mechanism. This ensures that, under the premise of meeting system hard constraints such as storage capacity, computing power, and distribution consistency, the weighted combination of service latency and deployment cost is minimized, thus realizing knowledge-enhanced multi-agent decision-making for service deployment and request distribution in edge computing scenarios.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] Step S1: Considering the characteristics of heterogeneous and limited server resources, dynamic arrival of service requests, and differences in service cold / hot start costs in edge computing networks, construct an edge service deployment model, a service request distribution model, and a service processing latency model; with the goal of minimizing the weighted combination of system service latency and deployment costs, construct a quantitative representation model of overall system performance.

[0006] Step S2: For edge server storage capacity constraints, computing power constraints, and request distribution consistency constraints, construct a joint optimization mathematical model for service deployment and request distribution that is jointly optimized for service latency and deployment cost. This model is a mixed integer nonlinear programming problem.

[0007] Step S3: Construct an edge computing multi-agent knowledge graph EC-MAKG, which solidifies the inherent attributes and prior constraints of servers, services, and requests through triples, and dynamically accumulates effective deployment and distribution experience in the multi-agent collaboration process; use the relational graph convolutional network R-GCN to embed local knowledge into the knowledge graph and extract the knowledge feature vectors of each edge server agent.

[0008] Step S4: Remodel the joint optimization mathematical model into a knowledge graph-enhanced multi-agent restricted Markov decision process (KG-MA-CMDP), defining the knowledge-enhanced state space, the knowledge-constrained action space, the knowledge-enhanced reward function, and the constraint function, respectively.

[0009] Step S5: Based on the centralized training-distributed execution framework, a knowledge graph-driven multi-agent security enhancement near-end policy optimization algorithm KG-S-MAPPO is designed for solving the problem. In the centralized training phase, the global state and knowledge features are integrated through a knowledge-enhanced global Critic network. In the distributed execution phase, each edge server agent autonomously generates service deployment and request distribution decisions based on its local state and knowledge embedding features. Through a knowledge-guided action masking mechanism and a Lagrange augmented objective constraint optimization mechanism, a joint optimization strategy for service deployment and request distribution is obtained under the premise of satisfying system resource constraints.

[0010] Furthermore, in step S1, an edge computing system model is defined: Consider a typical edge computing system, which includes a set of edge servers, denoted as... ,in This represents the total number of edge servers. These servers are deployed at the base stations within the network coverage area to provide low-latency computing services to users within their coverage area. Due to differences in actual deployment conditions (such as hardware configuration, geographical location, and network access capabilities), different edge servers exhibit significant heterogeneity in terms of computing power and storage capacity. This heterogeneity makes service deployment and task scheduling more complex, but also provides greater design flexibility for system performance optimization.

[0011] Assume the system has Class-specific services (such as real-time video analytics, automatic speech recognition, and deep learning inference) are denoted as... Each service They have different startup overheads. (Unit: CPU cycles), Storage requirements and calculation intensity (Unit: CPU cycles / bit), reflecting the computational load of the service during the initialization phase, the memory resources required for operation, and the computational complexity required for processing a unit of data, respectively.

[0012] Based on this, we model each service request as a corresponding type. The computational task has an input data size of (Unit: bit). Therefore, the total computing resources required to process this request are: CPU cycles, a metric that characterizes the computational load intensity of a single task, are crucial. It's important to note that service requests can only be processed by edge servers with the corresponding service instances deployed; this constraint plays a critical role in subsequent service deployment and task scheduling decisions.

[0013] To characterize the dynamic evolution process of the system, let Denotes the set of discrete time slots, where This represents the total number of time slots. This indicates the duration of each time slot. In each time slot... In this scenario, users initiate different types of service requests to their nearest edge servers. The arrival process of these requests exhibits significant randomness; the intensity of arrival and the type of service dynamically change over time to reflect the fluctuating characteristics of business operations in the real-world network environment. In time slot Inner edge server Received service type The number of requests is used to characterize the distribution of the system's business load in the spatiotemporal dimensions.

[0014] Service deployment model:

[0015] set up For the binary decision variables of service deployment, where Indicates time slot Service Deployed on edge servers If the value is above 0, it is 0 otherwise. This variable is used to characterize the service deployment status of the system in the spatiotemporal dimensions.

[0016] Due to limited storage resources on edge servers, it is impossible to deploy all service types simultaneously on the same server within any given time slot. Therefore, the total storage usage of deployed services on each edge server must meet capacity constraints, namely:

[0017] ,

[0018] ,

[0019] in, Represents edge server Storage capacity (unit: GB). Indicates service The above constraints ensure the feasibility of service deployment decisions within physical resource limitations.

[0020] In the service deployment decision-making process, the service's startup status has a significant impact on system performance (especially latency and cost). Specifically, it is necessary to distinguish between two typical scenarios: cold start and warm start. For newly deployed services, if they have not been loaded in the previous time slot, a complete initialization process (such as container startup, dependency loading, and cache preheating) needs to be executed, thus introducing additional startup latency. However, for services that maintain their deployment status in consecutive time slots, their runtime environment has been cached and reused, and the startup overhead is usually negligible.

[0021] Formalistically, the following two states are defined:

[0022] Cold start: If and Then the service In the time gap The transition from a non-deployed state to a deployed state requires initial loading, which corresponds to the cold start process.

[0023] Warm Start: If and If the service remains in a deployed state, its startup overhead is negligible, which corresponds to a warm start.

[0024] To uniformly characterize service startup behavior, the following indicator function is introduced:

[0025] ,

[0026] in .when The time interval indicates a cold start; otherwise, it indicates a warm start or an undeployed state. This definition concisely captures the temporal changes in service deployment status.

[0027] Based on the above definition, the additional latency introduced by a cold start can be expressed as:

[0028] ,

[0029] in, Represents edge server For service Allocated computing power (unit: CPU cycles / s). This represents the computational overhead required for service initialization. This expression quantitatively describes the relationship between cold start latency and resource allocation.

[0030] Furthermore, to comprehensively assess the impact of service deployment behavior on system resources and performance, this paper models the service deployment cost as the sum of the following two parts:

[0031] 1) Cold start configuration cost It is used to characterize the initialization overhead when a service is deployed for the first time (such as dependency loading, image pulling, and data prefetching).

[0032] 2) Hot start maintenance costs This is used to describe the system resource overhead (such as memory residency and background maintenance consumption) consumed by a service during continuous operation.

[0033] Therefore, service On the edge server During the time gap The total deployment cost can be expressed as:

[0034] ,

[0035] Generally speaking, cold start cost Significantly higher than maintenance costs Higher cold start costs discourage frequent service deployments and migrations, prompting the system to extend service dwell time to amortize initialization overhead. Conversely, higher maintenance costs drive the system to reduce the duration of continuous service deployments to minimize long-term resource consumption. Therefore, the trade-off between cold start and maintenance costs essentially determines the dynamic evolution of service deployment strategies and has a profound impact on overall system performance.

[0036] Service request distribution model:

[0037] To alleviate the problems of limited resources and uneven system load on edge servers, this paper allows collaborative request forwarding and processing among edge servers. For example, when the load on servers in the office area is high during the day, some requests can be forwarded to servers in the residential area where the load is lower; while at night, the opposite forwarding strategy can be implemented. This mechanism helps improve the overall resource utilization of the system and reduce service processing latency.

[0038] Therefore, define For edge servers In the time gap Regarding services The request distribution strategy, in which Indicates from server Forwarded to server The request ratio. This strategy must meet the following constraints:

[0039] ,

[0040] ,

[0041] ,

[0042] The above constraints respectively ensure the legality, completeness, and feasibility of the request distribution ratio. Among them, the third constraint ensures that requests can only be forwarded to edge servers where the corresponding services have been deployed, thereby avoiding ineffective scheduling.

[0043] Given a request distribution strategy, the edge server In the time gap Received services The total computational load (in CPU cycles) can be expressed as:

[0044] ,

[0045] This expression describes the redistribution process of requests from all edge servers within the system. Meanwhile, to ensure system stability and processability, the following computational capacity constraints must be met:

[0046] ,

[0047] That is, the computing load received by each edge server per unit of time must not exceed its maximum processing capacity. server Regarding services If the incoming load exceeds its processing limit, the excess will be forwarded to the cloud server. Process accordingly. Define the overload limit as:

[0048] ,

[0049] set up For time slots Inner edge server To cloud server If the effective transmission rate is 0, then the forwarding delay for transmissions exceeding the load is:

[0050] ,

[0051] Cloud computing latency is:

[0052] ,

[0053] in For cloud servers as services Allocated computing power.

[0054] Service processing latency model:

[0055] In edge computing systems, service processing latency is a crucial indicator of service quality. This paper models service processing latency into two main components: forwarding latency, computation latency, and potential cold start latency.

[0056] (1) Forwarding delay:

[0057] Additional transmission latency will occur when service requests need to be forwarded between edge servers. Define the slave server. to server Forwarding service The forwarding delay of the request is:

[0058] ,

[0059] Where the numerator represents the amount of data that needs to be transmitted per unit time, and the denominator represents the amount of data that needs to be transmitted per unit time. This indicates the effective transmission rate between servers.

[0060] In a multi-hop network environment, the transmission rate is typically determined by the bottleneck link in the path; that is, the throughput of the entire path is limited by the minimum link capacity on that path. Let... , indicating at time from arrive The set of path links. (Note: The original text contains some formatting errors and inconsistencies. A more accurate translation would require For time slots Inner Link Given the available bandwidth, the transmission rate satisfies:

[0061] ,

[0062] This modeling method can effectively reflect the bandwidth bottleneck effect in real networks. It should be noted that when the data transmission path... When selecting the shortest path based on hop count or the path based on maximum available bandwidth, the transmission rate... All are limited by the minimum (bottleneck) link bandwidth on the selected path.

[0063] Furthermore, in simpler scenarios, if and If the connection between them is a single-hop direct connection, then This can be directly taken as the time slot of the direct link. Instantaneous capacity.

[0064] (2) Calculate the time delay (M / M / 1 model):

[0065] For those assigned to edge servers The computation latency of a service request can be characterized using the classic M / M / 1 queuing model. The computation latency is defined as:

[0066] ,

[0067] in, Indicates service On the server The arrival rate. When the system load approaches its processing capacity, the computation latency will increase significantly, thus reflecting queuing congestion.

[0068] Taking into account cold start latency, forwarding latency, and computation latency, the total processing latency for a service request can be expressed as:

[0069] ,

[0070] Finally, regarding the time interval Arrival at the server services The overall service latency of a request is determined by the slowest processing path, i.e.:

[0071] ,

[0072] This modeling approach can characterize the worst service experience in the system, thus better meeting the conservative requirements of user service quality (QoS / QoE) assessment.

[0073] Furthermore, in step S2, based on the above model, this paper constructs a system with service latency... Deployment costs A joint optimization problem with weighted combination as the objective. Weighting coefficients are introduced. Then the time interval The optimization objective is:

[0074] ,

[0075] ,

[0076] ,

[0077] ,

[0078] ,

[0079] in, This represents the service deployment decision variable. This represents the request distribution strategy. The objective function comprehensively considers user experience (low latency) and system overhead (low deployment cost), reflecting the trade-off between service quality and resource utilization.

[0080] Furthermore, in step S3, in the edge computing multi-agent cooperative optimization problem, the joint decision-making of service deployment and request distribution is strongly coupled and highly complex. Although traditional multi-agent reinforcement learning methods can achieve cooperative decision-making through the centralized training-distributed execution (CTDE) framework, they are prone to problems such as policy oscillation, slow convergence, and ineffective exploration due to the lack of explicit modeling of structured relational information in the environment.

[0081] To address this, this paper introduces a Knowledge Graph (KG) to structurally model entity relationships and collaborative experiences in edge computing systems. Based on this, a Knowledge Graph-Driven Multi-Agent Security Enhancement Proximal Policy Optimization Algorithm (KG-S-MAPPO) is proposed. This method constructs an Edge Computing Multi-Agent Knowledge Graph (EC-MAKG) and utilizes a Relational Graph Convolutional Network (R-GCN) for knowledge representation learning. This deeply integrates structured knowledge into the state representation, action constraints, and policy optimization processes of multi-agent reinforcement learning, achieving knowledge-driven collaborative decision-making.

[0082] EC-MAKG Construction and Updates:

[0083] EC-MAKG is a structured semantic network that represents domain knowledge of multi-agent collaboration in edge computing, presented in the form of triples. As the basic representation unit (where Indicates the head entity. Indicates a relationship. (Represents the tail entity). This knowledge graph comprises two core layers: a prior knowledge layer and an experiential knowledge layer. It supports both offline construction and online dynamic updating modes, enabling the structured solidification of prior knowledge in the edge computing domain and the continuous accumulation of experiential knowledge through multi-agent collaboration. This provides solid semantic support for subsequent knowledge-guided multi-agent reinforcement learning decisions.

[0084] EC-MAKG's entity system covers the core participants in multi-agent collaborative scenarios in edge computing, while its relationship system encompasses the inherent relationships and dynamic behaviors between entities. Together, they constitute the core semantic foundation of the knowledge graph.

[0085] Entity set It includes three core entities, each with a unique identifier and corresponding attribute features. These attribute features correspond one-to-one with the parameters of the edge computing system model constructed in this paper. The server intelligent agent entity... Corresponding edge server intelligent agent Its attribute set is Each parameter corresponds to a core characteristic of the server agent, such as its unique identifier, storage resources, and computing resources; service entity Corresponding to various services in the edge computing system Its attribute set is These respectively represent the service's unique identifier, startup overhead, storage requirements, computational intensity, and data scale; Request entity The attribute set for each service request generated within a time slot is as follows: These correspond to the unique identifier of the request, the type of service requested, the server agent that initiated the request, and the request arrival rate within that time slot, respectively.

[0086] Relationship set Based on knowledge type, they are divided into two main categories: prior relationships and empirical relationships. Prior relationships refer to inherent, static connections between entities, determined by the inherent attributes of the edge computing system. Empirical relationships, on the other hand, refer to dynamic and effective behavioral connections generated during multi-agent collaboration, extracted from the training experience of multi-agent reinforcement learning. These two categories contain six core types, among which... The relational representation indicates that the storage resources of the edge server agent can meet the deployment requirements of a certain service, which is the basic constraint for service deployment decisions. The relationship definition defines the physical communication link topology between edge server agents, which determines the reachability of request distribution and resource coordination; Clearly defining the relationship between service requests and corresponding service types provides a semantic basis for accurate request processing and scheduling; Relationships reflect the dynamic deployment behavior of edge server agents towards services and are one of the core behaviors of multi-agent collaborative decision-making; Relationships ensure that request distribution behavior meets system feasibility constraints, guaranteeing the rationality and effectiveness of distribution decisions; The relationship describes the request distribution strategy between edge server agents, reflecting the collaborative scheduling logic of multiple agents.

[0087] EC-MAKG employs a layered construction strategy. First, it builds a prior knowledge layer offline based on the inherent attributes and static associations of the edge computing system. Then, it builds an experience knowledge layer online based on the training experience of multi-agent reinforcement learning.

[0088] The core of the prior knowledge layer is to provide basic constraints for multi-agent collaboration. It is constructed offline, and three types of prior relations are built based on system constraints and static associations.

[0089] when At that time, establish ;

[0090] when At that time, establish ;

[0091] Based on the request type, establish .

[0092] After the system is built, the prior knowledge is stored in a structured manner using an RDF graph model.

[0093] The experiential knowledge layer is constructed online based on multi-agent reinforcement learning training experience. It filters effective samples that meet system constraints and offer high rewards from the experience replay buffer, extracts dynamic relationships from these samples, and generates experiential triples.

[0094] Deploy services, establish ;

[0095] Satisfying constraints ,Establish ;

[0096] Forward the request and establish .

[0097] When integrating experiential triples with prior knowledge layers, weights are assigned to experiential relationships based on reward values ​​and constraint satisfaction. At the same time, a knowledge pruning mechanism is introduced to periodically remove low-weight and invalid triples to ensure the effectiveness of the knowledge graph.

[0098] To adapt to the dynamic nature of edge computing systems (such as changes in request arrival rates and fluctuations in server deployment status), EC-MAKG has the capability for online dynamic updates, ensuring that the knowledge graph can reflect changes in system status and multi-agent experience in real time, thereby guaranteeing the timeliness and accuracy of knowledge-enhanced decision-making.

[0099] At the end of each time slot, an update operation will be performed. First, effective experience is extracted from the multi-agent behavior of the current time slot to generate new experience triples, which are then integrated into the knowledge graph to achieve real-time accumulation of experience knowledge. Second, the weights of existing experience relationships in the knowledge graph are cumulatively updated based on the reward value and constraint satisfaction of the current time slot to ensure that high-value experience has a higher weight. Finally, the attribute features of each entity are updated to ensure that the entity attributes are consistent with the actual state of the system.

[0100] Local knowledge embedding method based on R-GCN:

[0101] Structured knowledge in knowledge graphs is represented in the form of triples, which are difficult to directly participate in neural network computation. Therefore, knowledge embedding is needed to map entities and relations into vector representations. Considering the large number of many-to-many relations and multi-type relation coupling in EC-MAKG, traditional embedding methods based on the translation assumption (such as TransE) are difficult to accurately characterize complex semantic relations and are prone to representation aliasing problems. Therefore, this paper adopts R-GCN for knowledge embedding.

[0102] To balance computational efficiency and decision relevance, this paper employs an embedding strategy based on local subgraph extraction. For each edge server agent... Using the corresponding node as the center, extract its one-hop or two-hop neighborhood from EC-MAKG to form a local subgraph. This subgraph contains service nodes, request nodes, and neighbor server nodes related to the agent's decision-making, and retains multiple types of relationship connections, thereby covering the information required for service deployment and request distribution decisions.

[0103] In the time slot First, the nodes in the local subgraph are initialized based on entity attributes; then, the R-GCN with shared parameters is used in the subgraph. The information propagation and aggregation are performed at multiple levels to perceive relationships, enabling node representations to gradually integrate the structural semantic information of their multi-hop neighborhoods. After layer propagation, the embedded representation of the central node is extracted as the agent. Knowledge feature vector .

[0104] The specific algorithm flow is as follows:

[0105] Step S3-1: Set the algorithm input, including the current time slot. Global knowledge graph under Initial feature vectors of each entity node Target edge server intelligent agent node and the maximum number of convolutional layers in a graph convolutional network. .

[0106] Step S3-2: Using the target agent node Centered on the global knowledge graph Extract all nodes and corresponding relation edges within its two-hop neighborhood to form a local association subgraph. .

[0107] Step S3-3: Based on local subgraph The set of nodes contained therein is used to extract the attribute features of the corresponding entities and generate the initial feature matrix of the subgraph nodes. .

[0108] Step S3-4: Start iterative execution Layer graph convolution operation, in each convolutional layer In the context of subgraphs Each node within According to different relationship types Aggregate its neighbor nodes Based on the feature information, perform the following aggregation and update calculations:

[0109] ,

[0110] in, It is a non-linear activation function. and The first Layer-specific learnable weight matrices and self-loop weight matrices. This is the normalization constant.

[0111] Step S3-5: Completed After multiple convolutional iterations, the target agent nodes are extracted from the final generated node feature matrix. corresponding feature vector And assign it to the local knowledge embedding feature vector. .

[0112] Step S3-6: Output the target agent In the current time slot Local knowledge embedding feature vectors It is used for subsequent knowledge enhancement in the multi-agent state representation and decision-making process.

[0113] Furthermore, in step S4, knowledge embedding features are integrated into a multi-agent restricted Markov decision process (MA-CMDP) to construct a knowledge graph-driven multi-agent restricted Markov decision process (KG-MA-CMDP), achieving deep integration and collaborative optimization of structured knowledge and multi-agent dynamic decision-making processes. KG-MA-CMDP consists of octet... Formal representation, in which The constructed edge computing multi-agent knowledge graph is EC-MAKG. All other elements are redefined by incorporating knowledge embedding features on the basis of the original MA-CMDP. The core design of this model is to construct a knowledge-enhanced state space and a knowledge-constrained action space. At the same time, the reward function and constraint function are refined under knowledge guidance, so that the multi-agent decision-making process can make full use of the structured prior and experience knowledge of the knowledge graph.

[0114] Knowledge-enhanced state space:

[0115] To enable multi-agent decision-making to simultaneously rely on the system's dynamic state and the structured semantic support of knowledge graphs, this paper fuses the agents' local knowledge embedding features with their original local state features to construct a knowledge-enhanced state representation. (In time slots) Below, intelligent agents Knowledge Enhancement Local State Defined as: ,in, For the agent in the original MA-CMDP The local state vectors characterize system dynamics such as server load, remaining resources, and request arrival intensity. For agents extracted from EC-MAKG Local knowledge is embedded in the feature vector, which is obtained by analyzing the node... The local subgraph centered on the entity is obtained through R-GCN encoding and is used to represent the semantics of the multi-relation structure between entities; This represents element-wise multiplication; These are learnable gating coefficients used to adaptively adjust the fusion ratio of original state features and knowledge-embedded features during training. Based on this, the system's knowledge-enhanced global state... It is composed of the knowledge-enhanced local states of all edge server agents, and incorporates the core embedding features of the global knowledge graph to represent global knowledge associations at the system level. Its definition is... ,in It is the core embedded feature of the global knowledge graph.

[0116] Knowledge-constrained action space:

[0117] To reduce ineffective action exploration, suppress policy oscillations, and improve the effectiveness and rationality of decision-making during multi-agent reinforcement learning, this paper introduces prior relation constraints from EC-MAKG into the original MA-CMDP action space to construct a knowledge-constrained action space. Explicit, structured prior knowledge from the knowledge graph imposes hard constraints on the agent's action choices, achieving knowledge-guided action space optimization. Knowledge-constrained local action space For the original local action space A subset of , formally defined as:

[0118] ,

[0119] in, Indicates a satisfying relationship. This represents the set of prior relations in EC-MAKG.

[0120] Knowledge-enhanced reward function:

[0121] This paper directly uses the negative value of the system's global optimization objective as the reward function, in the following form:

[0122] ,

[0123] Building upon this, experiential knowledge from knowledge graphs is introduced to enhance the reward. By measuring the matching degree between joint actions and EC-MAKG, the knowledge-enhanced global reward function is obtained. :

[0124] ,

[0125] in, The weighting coefficient for the experience-knowledge matching reward is used to adaptively balance the contribution of basic decision-making reward and knowledge matching reward. For multi-agent joint action The degree of matching with EC-MAKG experiential knowledge, with a value range of [value range missing]. The higher the value, the better the current collaborative decision-making aligns with the accumulated effective experience, and the more it reflects the experience-guided value of knowledge graphs.

[0126] Constraint function design:

[0127] This paper designs constraint functions for storage resource constraints in edge computing systems, and uses the exponential reconstruction method to transform the storage constraints of multiple servers into a single global constraint function. Storage constraints Defined as:

[0128] ,

[0129] Sum the reconstructed storage constraints of all edge servers to obtain the time slot. global constraint functions under :

[0130] ,

[0131] The threshold of the global constraint function is the total number of edge servers. (Since the threshold for the constraint term is 1 for each server,) The total constraint threshold for each server is ).when At that time, all servers met the storage resource constraints; when At that time, at least one server violated the storage constraint, and The larger the value, the higher the degree of constraint violation in the system.

[0132] Furthermore, to reduce the degree of constraint violation and improve the feasibility and rationality of decision-making, this paper introduces prior information from knowledge graphs to enhance the global constraint function. Specifically, EC-MAKG prior knowledge constraints are transformed into penalty terms and integrated with the global constraint function to construct a knowledge-enhanced global constraint function. :

[0133] ,

[0134] in, This is a weighting coefficient for prior knowledge conflict penalties, used to adjust the degree of influence of knowledge conflict penalties on overall constraints; For multi-agent joint action The degree of conflict with EC-MAKG prior knowledge, with a value range of [value missing]. The higher this value, the greater the degree to which the current collaborative decision violates the inherent constraints of the system, and the greater the penalty that needs to be imposed.

[0135] Optimization goal:

[0136] This paper aims to find the optimal multi-agent joint policy to maximize the system's long-term discount accumulation global reward, while satisfying the global constraint of knowledge enhancement. Let be the joint policy of KG-MA-CMDP. For its optimal strategy, we have:

[0137] ,

[0138] in, It is a set of feasible strategies for multiple agents. Accumulated global rewards for long-term discounts are defined as follows:

[0139] ,

[0140] in, As a discount factor, This represents the total number of time slots. Representation Strategy The expected value of the following mathematical expression.

[0141] Accordingly, To accumulate global constraints for long-term discounts, namely:

[0142] ,

[0143] in, This is the global constraint threshold, whose value is an inherent threshold setting based on the system's storage resource constraints, i.e. .when When, it indicates the strategy Meeting the storage resource constraints of all edge servers during the training period is a feasible strategy.

[0144] Furthermore, in step S5, based on the centralized training-distributed execution framework, the knowledge-enhanced multi-agent network designed in this paper operates as follows in each time slot: each edge server agent interacts with the simulation environment to generate a global state containing knowledge enhancement. Joint actions Knowledge Enhancement Rewards Knowledge Enhancement Constraints and the enhanced global state in the next time slot The edge computing multi-agent knowledge graph EC-MAKG is updated online based on the latest system state, and the R-GCN encoder is used to extract the local knowledge embedding vectors of each agent. and the mask used for motion masking Subsequently, the local dynamic state of each agent... With corresponding knowledge embedding Perform weighted fusion to generate knowledge-enhanced local states. ; further all intelligent agents Combined with global knowledge embedding, it forms a knowledge-enhanced global state. The aforementioned experience samples are uniformly stored in the global experience replay buffer.

[0145] During the training phase, mini-batch experience data is sampled from the buffer. The centralized evaluation module comprises two knowledge-enhanced Critic networks to... Given input, output a reward value estimate. With constrained value estimation And calculate the reward advantage of knowledge enhancement accordingly. With constraint advantage On the distributed execution side, each agent independently maintains a dual-Actor network: a service deployment Actor network. and request distribution Actor network Finally, all network parameters are updated with gradients based on the advantage estimation of knowledge enhancement, and the empirical relation weights and entity attributes in EC-MAKG are also updated synchronously, thereby achieving collaborative optimization of knowledge representation and multi-agent policies.

[0146] Knowledge-enhanced local actor network design:

[0147] Each intelligent agent Configure two independent Actor networks to decouple service deployment and request distribution decisions. Service Deployment Actor Network Enhancing local state with knowledge As input, output the service deployment probability distribution that satisfies the prior constraints; for services not present in EC-MAKG... The probability of the service related to the relationship is directly set to zero, and the discrete deployment action is obtained through Bernoulli sampling. Request to distribute Actor network Simultaneously receive With action mask The system outputs the distribution parameters of continuously distributed actions, forces the dimensions of actions that violate constraints to zero, and finally obtains the normalized distribution ratio. Joint actions are defined as follows: Global coordinated actions Therefore, intelligent agents The local joint strategy can be expressed as:

[0148] ,

[0149] Among them, joint actions .

[0150] The global joint policy of multiple agents is composed of the product of their local policies, reflecting the independence and global coordination of distributed decision-making.

[0151] ,

[0152] Knowledge-enhanced action shielding mechanism:

[0153] To suppress ineffective exploration and policy oscillation, this paper introduces EC-MAKG prior constraints on top of the existing action masking to construct a dual filtering mechanism. Specifically, the agent... First, a priori mask is generated based on prior relations. At the same time, based on the deployment decision A deployment state mask is generated, and a distribution consistency mask is obtained through global aggregation. Fusion Mask Accurately identify and effectively distribute to target audiences, and apply them to The output forces an invalid action with a probability of zero, thus ensuring that the decision simultaneously satisfies the constraints of service deployment status, distribution feasibility, and resource feasibility.

[0154] Knowledge-enhanced global Critic network design:

[0155] The centralized evaluation module contains two independent knowledge-enhancing Critic networks: a reward value network. With constrained value network Both enhance the global state through knowledge. Given the input, output the expected long-term reward respectively. Violation of expectations with long-term constraints To accurately evaluate the merits of joint actions, generalized advantage estimation (GAE) is used to calculate the reward advantage and constraint advantage of knowledge enhancement:

[0156] ,

[0157] ,

[0158] in This is the GAE coefficient.

[0159] KG-S-MAPPO algorithm:

[0160] The KG-S-MAPPO algorithm is based on the MAPPO (Multi-Agent Proximal Policy Optimization) algorithm, integrating the structured knowledge of EC-MAKG with the constraint optimization ideas of secure deep reinforcement learning. It extends the target policy for policy update by both knowledge enhancement and security constraints, effectively suppressing oscillations and instabilities during policy training. At the same time, it incorporates the constraint optimization capabilities of secure deep reinforcement learning, ensuring that the resource constraints of the edge computing system are met. This solves the problems of low exploration efficiency and poor policy stability of the traditional MAPPO algorithm in complex constraint scenarios.

[0161] Lagrange augmentation target:

[0162] To transform the constrained optimization problem of KG-MA-CMDP into a solvable unconstrained optimization problem, this paper introduces global Lagrange multipliers. With global penalty factor Construct the multi-agent Lagrange augmented objective function, which is formally defined as follows:

[0163] ,

[0164] The first term is the negative long-term global reward for knowledge enhancement; the second term is the augmented Lagrange penalty term, which mainly penalizes policies that violate global constraints; and the third term is the regularization term, which is used to ensure the stability of the Lagrange multiplier update process, avoid policy oscillations caused by multiplier fluctuations, and improve the convergence of algorithm training.

[0165] Local Actor Network Update Target:

[0166] Combining the optimization ideas of the PPO algorithm, importance sampling, and entropy regularization, this paper proposes a method for each edge server agent. Service deployment Actor network With request distribution Actor network Design a local policy to update the target. During the algorithm training... Wheel, defining the old multi-agent joint strategy as The new alliance strategy is intelligent agent The local importance sampling ratio is defined as the ratio of the probability of the old policy to that of the new policy in the current local state, i.e. ,in For intelligent agents The set of local Actor network parameters includes parameters from two sub-networks: service deployment and request distribution. The global importance sampling ratio is the product of the local importance sampling ratios of all agents, i.e. ,in It is the set of parameters for the joint strategy of multiple agents.

[0167] Based on the aforementioned importance sampling ratio, the service deploys an Actor network. The local update target, combined with the PPO pruning operation and entropy regularization term, is defined as:

[0168]

[0169] ,

[0170] in To determine the local importance sampling ratio for deploying the Actor network, The coefficient of the entropy regularization term, This is the entropy function, used to enhance the exploration capabilities of the strategy.

[0171] Correspondingly, a request is made to distribute the Actor network. The local update target is:

[0172]

[0173] ,

[0174] in To request the distribution of the local importance sampling ratio of the Actor network, For its corresponding entropy regularization coefficient, incorporate a knowledge-guided action mask. This ensures that strategy updates comply with knowledge constraints.

[0175] The global constraint optimization objective is constructed based on the global importance sampling ratio and the global constraint advantage function. It is used to strengthen the guiding role of constraints on policy updates and is defined as follows:

[0176] ,

[0177] In summary, the intelligent agent The overall goal of local updates can be formally expressed as:

[0178]

[0179] ,

[0180] Global Lagrange multipliers and penalty factor updates:

[0181] The global Lagrange multipliers are obtained using the projection gradient ascent method. Iterative updates are conducted to strictly ensure... The constraints and their update rules are as follows:

[0182] ,

[0183] in For the first The long-term discount of the wheel accumulates global constraints. This update rule can dynamically adjust the size of the multiplier according to the degree of constraint violation. The more serious the constraint violation, the larger the multiplier, and the stronger the penalty for the policy that violates the constraint.

[0184] In addition, the global constraint violation degree is defined as:

[0185] ,

[0186] Set constraint violation threshold The penalty factor is adjusted based on the comparison between the degree of violation and the threshold. This indicates that the degree of constraint violation exceeds the acceptable range, and the penalty factor is updated to... ,in The penalty growth coefficient is used; otherwise, the penalty factor remains unchanged, i.e. .

[0187] Global Critic network update target:

[0188] Global Reward Value Assessment Network With constrained value assessment network A centralized update approach is adopted, using the mean squared error (MSE) loss function as the update objective. The core objective is to minimize the error between the network value estimate and the global target value. The loss functions are defined as follows:

[0189] ,

[0190] ,

[0191] in, and These are the target reward value evaluation network and the target constraint value evaluation network, respectively. Their parameters are a lag copy of the current corresponding Critic network parameters. This lag update mechanism can effectively reduce the fluctuation of target value during training, avoid the instability of Critic network training caused by sudden changes in target value, and thus improve the convergence stability of the entire algorithm.

[0192] The specific algorithm flow is as follows:

[0193] Step S5-1: Initialization Phase. Initialize the knowledge enhancement service deployment Actor network. With request distribution Actor network Parameters; Initialize reward value Critic network With constrained value Critic network Parameters; initialize global Lagrange multipliers Punishment factor Penalty growth coefficient and cutting factor Initialize the global experience replay buffer. Initialize the parameters of the relational graph convolutional network R-GCN.

[0194] Step S5-2: Begin outer layer training iterations. For each training iteration... Based on current knowledge, enhance multi-agent joint strategies Interact with the edge computing environment to generate an interaction trajectory that includes a knowledge-enhanced global state, joint actions, knowledge-enhanced rewards, and constraint values. .

[0195] Step S5-3: Perform online updates to the edge computing multi-agent knowledge graph EC-MAKG, extract the multi-agent behaviors of the current time slot into experience triples and integrate them into the knowledge graph, and update the attribute features and experience relationship weights of each entity.

[0196] Step S5-4: Encode the updated knowledge graph using R-GCN and extract the local knowledge embedding feature vectors of each edge server agent. And generate knowledge-guided action masking based on prior relationships and deployment status. .

[0197] Step S5-5: Enhance the knowledge experience samples generated in each time slot Store in the global experience replay buffer .

[0198] Steps S5-6: Begin the update loop. From the experience buffer. Medium-sampled small batches of data were used to calculate the knowledge-enhancing reward advantage based on generalized advantage estimation. With knowledge-enhancing constraint advantages .

[0199] Step S5-7: Update the global reward value Critic network using the mean squared error loss function. and global constraint value Critic network Minimize the error between the network value estimate and the target value.

[0200] Step S5-8: Deploy the Actor network for the computing service Local update objective function ;Calculation request distribution Actor network Local update objective function ; Calculate the global constraint optimization objective function .

[0201] Step S5-9: Integrate the above sub-objectives with the Lagrange augmentation penalty term to calculate the local update overall objective function for each agent. .

[0202] Step S5-10: Update the service deployment Actor network parameters using gradient descent. and request distribution of Actor network parameters Synchronously update R-GCN network parameters.

[0203] Step S5-11: Update the global Lagrange multipliers using the projected gradient ascent method. Calculate the constraint violation degree of the current policy. With violation threshold .

[0204] Step S5-12: Determine the degree of constraint violation. If the degree of constraint violation exceeds the acceptable range, i.e. Then the penalty factor will be updated to Otherwise, the penalty factor remains unchanged, i.e. .

[0205] Step S5-13: After the update loop ends, the weights and entity attributes of each experience relationship in the knowledge graph are updated synchronously to complete the training iteration of the current round.

[0206] Step S5-14: After reaching the preset maximum number of training rounds, save the parameters of the trained service deployment Actor network, request distribution Actor network, and each Critic network; when applied, each edge server agent autonomously generates joint decisions for service deployment and request distribution based on local observation status and knowledge embedding features through the trained network.

[0207] Compared to existing technologies, the advantages of this invention are as follows: This invention proposes an edge service deployment and request distribution method based on knowledge graph-enhanced multi-agent secure reinforcement learning. First, an edge computing multi-agent knowledge graph is constructed to structurally solidify the inherent attributes and prior constraints of servers, services, and requests, and dynamically accumulate effective experience in deployment and distribution. Then, the problem is modeled as a knowledge graph-enhanced multi-agent restricted Markov decision process, designing a knowledge-enhanced state space, a knowledge-constrained action space, and knowledge-enhanced reward and constraint functions. Finally, a knowledge graph-driven multi-agent secure reinforcement near-end policy optimization algorithm is proposed. Through a knowledge-enhanced action masking mechanism and a Lagrange augmented objective constraint optimization mechanism, efficient joint optimization of service deployment and request distribution strategies is achieved while ensuring system resource constraints are met. Attached Figure Description

[0208] Figure 1 This is a schematic diagram of the edge computing environment in an example of the present invention;

[0209] Figure 2 This is a knowledge graph structure diagram in an example of the present invention;

[0210] Figure 3 This is a diagram of the algorithm framework in an example of the present invention. Detailed Implementation

[0211] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0212] Step S1: Considering the characteristics of heterogeneous and limited server resources, dynamic arrival of service requests, and differences in service cold / hot start costs in edge computing networks, construct an edge service deployment model, a service request distribution model, and a service processing latency model; with the goal of minimizing the weighted combination of system service latency and deployment costs, construct a quantitative representation model of overall system performance.

[0213] Consider a typical edge computing system that includes a set of edge servers, denoted as . ,in This represents the total number of edge servers. These servers are deployed at the base stations within the network coverage area to provide low-latency computing services to users within their coverage area. Due to differences in actual deployment conditions (such as hardware configuration, geographical location, and network access capabilities), different edge servers exhibit significant heterogeneity in terms of computing power and storage capacity. This heterogeneity makes service deployment and task scheduling more complex, but also provides greater design flexibility for system performance optimization.

[0214] Assume the system has Class-specific services (such as real-time video analytics, automatic speech recognition, and deep learning inference) are denoted as... Each service They have different startup overheads. (Unit: CPU cycles), Storage requirements and calculation intensity (Unit: CPU cycles / bit), reflecting the computational load of the service during the initialization phase, the memory resources required for operation, and the computational complexity required for processing a unit of data, respectively.

[0215] Based on this, we model each service request as a corresponding type. The computational task has an input data size of (Unit: bit). Therefore, the total computing resources required to process this request are: CPU cycles, a metric that characterizes the computational load intensity of a single task, are crucial. It's important to note that service requests can only be processed by edge servers with the corresponding service instances deployed; this constraint plays a critical role in subsequent service deployment and task scheduling decisions.

[0216] To characterize the dynamic evolution process of the system, let Denotes the set of discrete time slots, where This represents the total number of time slots. This indicates the duration of each time slot. In each time slot... In this scenario, users initiate different types of service requests to their nearest edge servers. The arrival process of these requests exhibits significant randomness; the intensity of arrival and the type of service dynamically change over time to reflect the fluctuating characteristics of business operations in the real-world network environment. In time slot Inner edge server Received service type The number of requests is used to characterize the distribution of the system's business load in the spatiotemporal dimensions.

[0217] Service deployment model:

[0218] set up For the binary decision variables of service deployment, where Indicates time slot Service Deployed on edge servers If the value is above 0, it is 0 otherwise. This variable is used to characterize the service deployment status of the system in the spatiotemporal dimensions.

[0219] Due to limited storage resources on edge servers, it is impossible to deploy all service types simultaneously on the same server within any given time slot. Therefore, the total storage usage of deployed services on each edge server must meet capacity constraints, namely:

[0220] ,

[0221] ,

[0222] in, Represents edge server Storage capacity (unit: GB). Indicates service The above constraints ensure the feasibility of service deployment decisions within physical resource limitations.

[0223] In the service deployment decision-making process, the service's startup status has a significant impact on system performance (especially latency and cost). Specifically, it is necessary to distinguish between two typical scenarios: cold start and warm start. For newly deployed services, if they have not been loaded in the previous time slot, a complete initialization process (such as container startup, dependency loading, and cache preheating) needs to be executed, thus introducing additional startup latency. However, for services that maintain their deployment status in consecutive time slots, their runtime environment has been cached and reused, and the startup overhead is usually negligible.

[0224] Formalistically, the following two states are defined:

[0225] Cold start: If and Then the service In the time gap The transition from a non-deployed state to a deployed state requires initial loading, which corresponds to the cold start process.

[0226] Warm Start: If and If the service remains in a deployed state, its startup overhead is negligible, which corresponds to a warm start.

[0227] To uniformly characterize service startup behavior, the following indicator function is introduced:

[0228] ,

[0229] in .when The time interval indicates a cold start; otherwise, it indicates a warm start or an undeployed state. This definition concisely captures the temporal changes in service deployment status.

[0230] Based on the above definition, the additional latency introduced by a cold start can be expressed as:

[0231] ,

[0232] in, Represents edge server For service Allocated computing power (unit: CPU cycles / s). This represents the computational overhead required for service initialization. This expression quantitatively describes the relationship between cold start latency and resource allocation.

[0233] Furthermore, to comprehensively assess the impact of service deployment behavior on system resources and performance, this paper models the service deployment cost as the sum of the following two parts:

[0234] 1) Cold start configuration cost It is used to characterize the initialization overhead when a service is deployed for the first time (such as dependency loading, image pulling, and data prefetching).

[0235] 2) Hot start maintenance costs This is used to describe the system resource overhead (such as memory residency and background maintenance consumption) consumed by a service during continuous operation.

[0236] Therefore, service On the edge server During the time gap The total deployment cost can be expressed as:

[0237] ,

[0238] Generally speaking, cold start cost Significantly higher than maintenance costs Higher cold start costs discourage frequent service deployments and migrations, prompting the system to extend service dwell time to amortize initialization overhead. Conversely, higher maintenance costs drive the system to reduce the duration of continuous service deployments to minimize long-term resource consumption. Therefore, the trade-off between cold start and maintenance costs essentially determines the dynamic evolution of service deployment strategies and has a profound impact on overall system performance.

[0239] Service request distribution model:

[0240] To alleviate the problems of limited resources and uneven system load on edge servers, this paper allows collaborative request forwarding and processing among edge servers. For example, when the load on servers in the office area is high during the day, some requests can be forwarded to servers in the residential area where the load is lower; while at night, the opposite forwarding strategy can be implemented. This mechanism helps improve the overall resource utilization of the system and reduce service processing latency.

[0241] Therefore, define For edge servers In the time gap Regarding services The request distribution strategy, in which Indicates from server Forwarded to server The request ratio. This strategy must meet the following constraints:

[0242] ,

[0243] ,

[0244] ,

[0245] The above constraints respectively ensure the legality, completeness, and feasibility of the request distribution ratio. Among them, the third constraint ensures that requests can only be forwarded to edge servers where the corresponding services have been deployed, thereby avoiding ineffective scheduling.

[0246] Given a request distribution strategy, the edge server In the time gap Received services The total computational load (in CPU cycles) can be expressed as:

[0247] ,

[0248] This expression describes the redistribution process of requests from all edge servers within the system. Meanwhile, to ensure system stability and processability, the following computational capacity constraints must be met:

[0249] ,

[0250] That is, the computing load received by each edge server per unit of time must not exceed its maximum processing capacity. server Regarding services If the incoming load exceeds its processing limit, the excess will be forwarded to the cloud server. Process accordingly. Define the overload limit as:

[0251] ,

[0252] set up For time slots Inner edge server To cloud server If the effective transmission rate is 0, then the forwarding delay for transmissions exceeding the load is:

[0253] ,

[0254] Cloud computing latency is:

[0255] ,

[0256] in For cloud servers as services Allocated computing power.

[0257] Service processing latency model:

[0258] In edge computing systems, service processing latency is a crucial indicator of service quality. This paper models service processing latency into two main components: forwarding latency, computation latency, and potential cold start latency.

[0259] (1) Forwarding delay:

[0260] Additional transmission latency will occur when service requests need to be forwarded between edge servers. Define the slave server. to server Forwarding service The forwarding delay of the request is:

[0261] ,

[0262] Where the numerator represents the amount of data that needs to be transmitted per unit time, and the denominator represents the amount of data that needs to be transmitted per unit time. This indicates the effective transmission rate between servers.

[0263] In a multi-hop network environment, the transmission rate is typically determined by the bottleneck link in the path; that is, the throughput of the entire path is limited by the minimum link capacity on that path. Let... , indicating at time from arrive The set of path links. (Note: The original text contains some formatting errors and inconsistencies. A more accurate translation would require For time slots Inner Link Given the available bandwidth, the transmission rate satisfies:

[0264] ,

[0265] This modeling method can effectively reflect the bandwidth bottleneck effect in real networks. It should be noted that when the data transmission path... When selecting the shortest path based on hop count or the path based on maximum available bandwidth, the transmission rate... All are limited by the minimum (bottleneck) link bandwidth on the selected path.

[0266] Furthermore, in simpler scenarios, if and If the connection between them is a single-hop direct connection, then This can be directly taken as the time slot of the direct link. Instantaneous capacity.

[0267] (2) Calculate the time delay (M / M / 1 model):

[0268] For those assigned to edge servers The computation latency of a service request can be characterized using the classic M / M / 1 queuing model. The computation latency is defined as:

[0269] ,

[0270] in, Indicates service On the server The arrival rate. When the system load approaches its processing capacity, the computation latency will increase significantly, thus reflecting queuing congestion.

[0271] Taking into account cold start latency, forwarding latency, and computation latency, the total processing latency for a service request can be expressed as:

[0272] ,

[0273] Finally, regarding the time interval Arrival at the server services The overall service latency of a request is determined by the slowest processing path, i.e.:

[0274] ,

[0275] This modeling approach can characterize the worst service experience in the system, thus better meeting the conservative requirements of user service quality (QoS / QoE) assessment.

[0276] Step S2: For edge server storage capacity constraints, computing power constraints, and request distribution consistency constraints, construct a joint optimization mathematical model for service deployment and request distribution that is jointly optimized for service latency and deployment cost. This model is a mixed integer nonlinear programming problem.

[0277] Based on the above model, this paper constructs a system based on service latency. Deployment costs A joint optimization problem with weighted combination as the objective. Weighting coefficients are introduced. Then the time interval The optimization objective is:

[0278] ,

[0279] ,

[0280] ,

[0281] ,

[0282] ,

[0283] in, This represents the service deployment decision variable. This represents the request distribution strategy. The objective function comprehensively considers user experience (low latency) and system overhead (low deployment cost), reflecting the trade-off between service quality and resource utilization.

[0284] Step S3: Construct an edge computing multi-agent knowledge graph EC-MAKG, which solidifies the inherent attributes and prior constraints of servers, services, and requests through triples, and dynamically accumulates effective deployment and distribution experience in the multi-agent collaboration process; use the relational graph convolutional network R-GCN to embed local knowledge into the knowledge graph and extract the knowledge feature vectors of each edge server agent.

[0285] In edge computing multi-agent cooperative optimization problems, the joint decision-making of service deployment and request distribution is highly coupled and complex. Although traditional multi-agent reinforcement learning methods can achieve cooperative decision-making through a centralized training-distributed execution (CTDE) framework, they are prone to problems such as policy oscillation, slow convergence, and ineffective exploration due to the lack of explicit modeling of structured relational information in the environment.

[0286] To address this, this paper introduces a Knowledge Graph (KG) to structurally model entity relationships and collaborative experiences in edge computing systems. Based on this, a Knowledge Graph-Driven Multi-Agent Security Enhancement Proximal Policy Optimization Algorithm (KG-S-MAPPO) is proposed. This method constructs an Edge Computing Multi-Agent Knowledge Graph (EC-MAKG) and utilizes a Relational Graph Convolutional Network (R-GCN) for knowledge representation learning. This deeply integrates structured knowledge into the state representation, action constraints, and policy optimization processes of multi-agent reinforcement learning, achieving knowledge-driven collaborative decision-making.

[0287] EC-MAKG Construction and Updates:

[0288] EC-MAKG is a structured semantic network that represents domain knowledge of multi-agent collaboration in edge computing, presented in the form of triples. As the basic representation unit (where Indicates the head entity. Indicates a relationship. (Represents the tail entity). This knowledge graph comprises two core layers: a prior knowledge layer and an experiential knowledge layer. It supports both offline construction and online dynamic updating modes, enabling the structured solidification of prior knowledge in the edge computing domain and the continuous accumulation of experiential knowledge through multi-agent collaboration. This provides solid semantic support for subsequent knowledge-guided multi-agent reinforcement learning decisions.

[0289] EC-MAKG's entity system covers the core participants in multi-agent collaborative scenarios in edge computing, while its relationship system encompasses the inherent relationships and dynamic behaviors between entities. Together, they constitute the core semantic foundation of the knowledge graph.

[0290] Entity set It includes three core entities, each with a unique identifier and corresponding attribute features. These attribute features correspond one-to-one with the parameters of the edge computing system model constructed in this paper. The server intelligent agent entity... Corresponding edge server intelligent agent Its attribute set is Each parameter corresponds to a core characteristic of the server agent, such as its unique identifier, storage resources, and computing resources; service entity Corresponding to various services in the edge computing system Its attribute set is These respectively represent the service's unique identifier, startup overhead, storage requirements, computational intensity, and data scale; Request entity The attribute set for each service request generated within a time slot is as follows: These correspond to the unique identifier of the request, the type of service requested, the server agent that initiated the request, and the request arrival rate within that time slot, respectively.

[0291] Relationship set Based on knowledge type, they are divided into two main categories: prior relationships and empirical relationships. Prior relationships refer to inherent, static connections between entities, determined by the inherent attributes of the edge computing system. Empirical relationships, on the other hand, refer to dynamic and effective behavioral connections generated during multi-agent collaboration, extracted from the training experience of multi-agent reinforcement learning. These two categories contain six core types, among which... The relational representation indicates that the storage resources of the edge server agent can meet the deployment requirements of a certain service, which is the basic constraint for service deployment decisions. The relationship definition defines the physical communication link topology between edge server agents, which determines the reachability of request distribution and resource coordination; Clearly defining the relationship between service requests and corresponding service types provides a semantic basis for accurate request processing and scheduling; Relationships reflect the dynamic deployment behavior of edge server agents towards services and are one of the core behaviors of multi-agent collaborative decision-making; Relationships ensure that request distribution behavior meets system feasibility constraints, guaranteeing the rationality and effectiveness of distribution decisions; The relationship describes the request distribution strategy between edge server agents, reflecting the collaborative scheduling logic of multiple agents.

[0292] EC-MAKG employs a layered construction strategy. First, it builds a prior knowledge layer offline based on the inherent attributes and static associations of the edge computing system. Then, it builds an experience knowledge layer online based on the training experience of multi-agent reinforcement learning.

[0293] The core of the prior knowledge layer is to provide basic constraints for multi-agent collaboration. It is constructed offline, and three types of prior relations are built based on system constraints and static associations.

[0294] when At that time, establish ;

[0295] when At that time, establish ;

[0296] Based on the request type, establish .

[0297] After the system is built, the prior knowledge is stored in a structured manner using an RDF graph model.

[0298] The experiential knowledge layer is constructed online based on multi-agent reinforcement learning training experience. It filters effective samples that meet system constraints and offer high rewards from the experience replay buffer, extracts dynamic relationships from these samples, and generates experiential triples.

[0299] Deploy services, establish ;

[0300] Satisfying constraints ,Establish ;

[0301] Forward the request and establish .

[0302] When integrating experiential triples with prior knowledge layers, weights are assigned to experiential relationships based on reward values ​​and constraint satisfaction. At the same time, a knowledge pruning mechanism is introduced to periodically remove low-weight and invalid triples to ensure the effectiveness of the knowledge graph.

[0303] To adapt to the dynamic nature of edge computing systems (such as changes in request arrival rates and fluctuations in server deployment status), EC-MAKG has the capability for online dynamic updates, ensuring that the knowledge graph can reflect changes in system status and multi-agent experience in real time, thereby guaranteeing the timeliness and accuracy of knowledge-enhanced decision-making.

[0304] At the end of each time slot, an update operation will be performed. First, effective experience is extracted from the multi-agent behavior of the current time slot to generate new experience triples, which are then integrated into the knowledge graph to achieve real-time accumulation of experience knowledge. Second, the weights of existing experience relationships in the knowledge graph are cumulatively updated based on the reward value and constraint satisfaction of the current time slot to ensure that high-value experience has a higher weight. Finally, the attribute features of each entity are updated to ensure that the entity attributes are consistent with the actual state of the system.

[0305] Local knowledge embedding method based on R-GCN:

[0306] Structured knowledge in knowledge graphs is represented in the form of triples, which are difficult to directly participate in neural network computation. Therefore, knowledge embedding is needed to map entities and relations into vector representations. Considering the large number of many-to-many relations and multi-type relation coupling in EC-MAKG, traditional embedding methods based on the translation assumption (such as TransE) are difficult to accurately characterize complex semantic relations and are prone to representation aliasing problems. Therefore, this paper adopts R-GCN for knowledge embedding.

[0307] To balance computational efficiency and decision relevance, this paper employs an embedding strategy based on local subgraph extraction. For each edge server agent... Using the corresponding node as the center, extract its one-hop or two-hop neighborhood from EC-MAKG to form a local subgraph. This subgraph contains service nodes, request nodes, and neighbor server nodes related to the agent's decision-making, and retains multiple types of relationship connections, thereby covering the information required for service deployment and request distribution decisions.

[0308] In the time slot First, the nodes in the local subgraph are initialized based on entity attributes; then, the R-GCN with shared parameters is used in the subgraph. The information propagation and aggregation are performed at multiple levels to perceive relationships, enabling node representations to gradually integrate the structural semantic information of their multi-hop neighborhoods. After layer propagation, the embedded representation of the central node is extracted as the agent. Knowledge feature vector .

[0309] The specific algorithm flow is as follows:

[0310] Step S3-1: Set the algorithm input, including the current time slot. Global knowledge graph under Initial feature vectors of each entity node Target edge server intelligent agent node and the maximum number of convolutional layers in a graph convolutional network. .

[0311] Step S3-2: Using the target agent node Centered on the global knowledge graph Extract all nodes and corresponding relation edges within its two-hop neighborhood to form a local association subgraph. .

[0312] Step S3-3: Based on local subgraph The set of nodes contained therein is used to extract the attribute features of the corresponding entities and generate the initial feature matrix of the subgraph nodes. .

[0313] Step S3-4: Start iterative execution Layer graph convolution operation, in each convolutional layer In the context of subgraphs Each node within According to different relationship types Aggregate its neighbor nodes Based on the feature information, perform the following aggregation and update calculations:

[0314] ,

[0315] in, It is a non-linear activation function. and The first Layer-specific learnable weight matrices and self-loop weight matrices. This is the normalization constant.

[0316] Step S3-5: Completed After multiple convolutional iterations, the target agent nodes are extracted from the final generated node feature matrix. corresponding feature vector And assign it to the local knowledge embedding feature vector. .

[0317] Step S3-6: Output the target agent In the current time slot Local knowledge embedding feature vectors It is used for subsequent knowledge enhancement in the multi-agent state representation and decision-making process.

[0318] Step S4: Remodel the joint optimization mathematical model into a knowledge graph-enhanced multi-agent restricted Markov decision process (KG-MA-CMDP), defining the knowledge-enhanced state space, the knowledge-constrained action space, the knowledge-enhanced reward function, and the constraint function, respectively.

[0319] This paper proposes a knowledge graph-driven multi-agent restricted Markov decision process (KG-MA-CMDP) by integrating knowledge embedding features into multi-agent restricted Markov decision processes (MA-CMDP). This enables deep integration and collaborative optimization of structured knowledge and multi-agent dynamic decision-making processes. KG-MA-CMDP consists of octet... Formal representation, in which The constructed edge computing multi-agent knowledge graph is EC-MAKG. All other elements are redefined by incorporating knowledge embedding features on the basis of the original MA-CMDP. The core design of this model is to construct a knowledge-enhanced state space and a knowledge-constrained action space. At the same time, the reward function and constraint function are refined under knowledge guidance, so that the multi-agent decision-making process can make full use of the structured prior and experience knowledge of the knowledge graph.

[0320] Knowledge-enhanced state space:

[0321] To enable multi-agent decision-making to simultaneously rely on the system's dynamic state and the structured semantic support of knowledge graphs, this paper fuses the agents' local knowledge embedding features with their original local state features to construct a knowledge-enhanced state representation. (In time slots) Below, intelligent agents Knowledge Enhancement Local State Defined as: ,in, For the agent in the original MA-CMDP The local state vectors characterize system dynamics such as server load, remaining resources, and request arrival intensity. For agents extracted from EC-MAKG Local knowledge is embedded in the feature vector, which is obtained by analyzing the node... The local subgraph centered on the entity is obtained through R-GCN encoding and is used to represent the semantics of the multi-relation structure between entities; This represents element-wise multiplication; These are learnable gating coefficients used to adaptively adjust the fusion ratio of original state features and knowledge-embedded features during training. Based on this, the system's knowledge-enhanced global state... It is composed of the knowledge-enhanced local states of all edge server agents, and incorporates the core embedding features of the global knowledge graph to represent global knowledge associations at the system level. Its definition is... ,in It is the core embedded feature of the global knowledge graph.

[0322] Knowledge-constrained action space:

[0323] To reduce ineffective action exploration, suppress policy oscillations, and improve the effectiveness and rationality of decision-making during multi-agent reinforcement learning, this paper introduces prior relation constraints from EC-MAKG into the original MA-CMDP action space to construct a knowledge-constrained action space. Explicit, structured prior knowledge from the knowledge graph imposes hard constraints on the agent's action choices, achieving knowledge-guided action space optimization. Knowledge-constrained local action space For the original local action space A subset of , formally defined as:

[0324] ,

[0325] in, Indicates a satisfying relationship. This represents the set of prior relations in EC-MAKG.

[0326] Knowledge-enhanced reward function:

[0327] This paper directly uses the negative value of the system's global optimization objective as the reward function, in the following form:

[0328] ,

[0329] Building upon this, experiential knowledge from knowledge graphs is introduced to enhance the reward. By measuring the matching degree between joint actions and EC-MAKG, the knowledge-enhanced global reward function is obtained. :

[0330] ,

[0331] in, The weighting coefficient for the experience-knowledge matching reward is used to adaptively balance the contribution of basic decision-making reward and knowledge matching reward. For multi-agent joint action The degree of matching with EC-MAKG experiential knowledge, with a value range of [value range missing]. The higher the value, the better the current collaborative decision-making aligns with the accumulated effective experience, and the more it reflects the experience-guided value of knowledge graphs.

[0332] Constraint function design:

[0333] This paper designs constraint functions for storage resource constraints in edge computing systems, and uses the exponential reconstruction method to transform the storage constraints of multiple servers into a single global constraint function. Storage constraints Defined as:

[0334] ,

[0335] Sum the reconstructed storage constraints of all edge servers to obtain the time slot. global constraint functions under :

[0336] ,

[0337] The threshold of the global constraint function is the total number of edge servers. (Since the threshold for the constraint term is 1 for each server,) The total constraint threshold for each server is ).when At that time, all servers met the storage resource constraints; when At that time, at least one server violated the storage constraint, and The larger the value, the higher the degree of constraint violation in the system.

[0338] Furthermore, to reduce the degree of constraint violation and improve the feasibility and rationality of decision-making, this paper introduces prior information from knowledge graphs to enhance the global constraint function. Specifically, EC-MAKG prior knowledge constraints are transformed into penalty terms and integrated with the global constraint function to construct a knowledge-enhanced global constraint function. :

[0339] ,

[0340] in, This is a weighting coefficient for prior knowledge conflict penalties, used to adjust the degree of influence of knowledge conflict penalties on overall constraints; For multi-agent joint action The degree of conflict with EC-MAKG prior knowledge, with a value range of [value missing]. The higher this value, the greater the degree to which the current collaborative decision violates the inherent constraints of the system, and the greater the penalty that needs to be imposed.

[0341] Optimization goal:

[0342] This paper aims to find the optimal multi-agent joint policy to maximize the system's long-term discount accumulation global reward, while satisfying the global constraint of knowledge enhancement. Let be the joint policy of KG-MA-CMDP. For its optimal strategy, we have:

[0343] ,

[0344] in, It is a set of feasible strategies for multiple agents. Accumulated global rewards for long-term discounts are defined as follows:

[0345] ,

[0346] in, As a discount factor, This represents the total number of time slots. Representation Strategy The expected value of the following mathematical expression.

[0347] Accordingly, To accumulate global constraints for long-term discounts, namely:

[0348] ,

[0349] in, This is the global constraint threshold, whose value is an inherent threshold setting based on the system's storage resource constraints, i.e. .when When, it indicates the strategy Meeting the storage resource constraints of all edge servers during the training period is a feasible strategy.

[0350] Step S5: Based on the centralized training-distributed execution framework, a knowledge graph-driven multi-agent security enhancement near-end policy optimization algorithm KG-S-MAPPO is designed for solving the problem. In the centralized training phase, the global state and knowledge features are integrated through a knowledge-enhanced global Critic network. In the distributed execution phase, each edge server agent autonomously generates service deployment and request distribution decisions based on its local state and knowledge embedding features. Through a knowledge-guided action masking mechanism and a Lagrange augmented objective constraint optimization mechanism, a joint optimization strategy for service deployment and request distribution is obtained under the premise of satisfying system resource constraints.

[0351] Based on a centralized training-distributed execution framework, this paper designs a knowledge-enhanced multi-agent network. The system's operation flow in each time slot is as follows: Each edge server agent interacts with the simulation environment to generate a global state that includes knowledge enhancement. Joint actions Knowledge Enhancement Rewards Knowledge Enhancement Constraints and the enhanced global state in the next time slot The edge computing multi-agent knowledge graph EC-MAKG is updated online based on the latest system state, and the R-GCN encoder is used to extract the local knowledge embedding vectors of each agent. and the mask used for motion masking Subsequently, the local dynamic state of each agent... With corresponding knowledge embedding Perform weighted fusion to generate knowledge-enhanced local states. ; further all intelligent agents Combined with global knowledge embedding, it forms a knowledge-enhanced global state. The aforementioned experience samples are uniformly stored in the global experience replay buffer.

[0352] During the training phase, mini-batch experience data is sampled from the buffer. The centralized evaluation module comprises two knowledge-enhanced Critic networks to... Given input, output a reward value estimate. With constrained value estimation And calculate the reward advantage of knowledge enhancement accordingly. With constraint advantage On the distributed execution side, each agent independently maintains a dual-Actor network: a service deployment Actor network. and request distribution Actor network Finally, all network parameters are updated with gradients based on the advantage estimation of knowledge enhancement, and the empirical relation weights and entity attributes in EC-MAKG are also updated synchronously, thereby achieving collaborative optimization of knowledge representation and multi-agent policies.

[0353] Knowledge-enhanced local actor network design:

[0354] Each intelligent agent Configure two independent Actor networks to decouple service deployment and request distribution decisions. Service Deployment Actor Network Enhancing local state with knowledge As input, output the service deployment probability distribution that satisfies the prior constraints; for services not present in EC-MAKG... The probability of the service related to the relationship is directly set to zero, and the discrete deployment action is obtained through Bernoulli sampling. Request to distribute Actor network Simultaneously receive With action mask The system outputs the distribution parameters of continuously distributed actions, forces the dimensions of actions that violate constraints to zero, and finally obtains the normalized distribution ratio. Joint actions are defined as follows: Global coordinated actions Therefore, intelligent agents The local joint strategy can be expressed as:

[0355] ,

[0356] Among them, joint actions .

[0357] The global joint policy of multiple agents is composed of the product of their local policies, reflecting the independence and global coordination of distributed decision-making.

[0358] ,

[0359] Knowledge-enhanced action shielding mechanism:

[0360] To suppress ineffective exploration and policy oscillation, this paper introduces EC-MAKG prior constraints on top of the existing action masking to construct a dual filtering mechanism. Specifically, the agent... First, a priori mask is generated based on prior relations. At the same time, based on the deployment decision A deployment state mask is generated, and a distribution consistency mask is obtained through global aggregation. Fusion Mask Accurately identify and effectively distribute to target audiences, and apply them to The output forces an invalid action with a probability of zero, thus ensuring that the decision simultaneously satisfies the constraints of service deployment status, distribution feasibility, and resource feasibility.

[0361] Knowledge-enhanced global Critic network design:

[0362] The centralized evaluation module contains two independent knowledge-enhancing Critic networks: a reward value network. With constrained value network Both enhance the global state through knowledge. Given the input, output the expected long-term reward respectively. Violation of expectations with long-term constraints To accurately evaluate the merits of joint actions, generalized advantage estimation (GAE) is used to calculate the reward advantage and constraint advantage of knowledge enhancement:

[0363] ,

[0364] ,

[0365] in This is the GAE coefficient.

[0366] KG-S-MAPPO algorithm:

[0367] The KG-S-MAPPO algorithm is based on the MAPPO (Multi-Agent Proximal Policy Optimization) algorithm, integrating the structured knowledge of EC-MAKG with the constraint optimization ideas of secure deep reinforcement learning. It extends the target policy for policy update by both knowledge enhancement and security constraints, effectively suppressing oscillations and instabilities during policy training. At the same time, it incorporates the constraint optimization capabilities of secure deep reinforcement learning, ensuring that the resource constraints of the edge computing system are met. This solves the problems of low exploration efficiency and poor policy stability of the traditional MAPPO algorithm in complex constraint scenarios.

[0368] Lagrange augmentation target:

[0369] To transform the constrained optimization problem of KG-MA-CMDP into a solvable unconstrained optimization problem, this paper introduces global Lagrange multipliers. With global penalty factor Construct the multi-agent Lagrange augmented objective function, which is formally defined as follows:

[0370] ,

[0371] The first term is the negative long-term global reward for knowledge enhancement; the second term is the augmented Lagrange penalty term, which mainly penalizes policies that violate global constraints; and the third term is the regularization term, which is used to ensure the stability of the Lagrange multiplier update process, avoid policy oscillations caused by multiplier fluctuations, and improve the convergence of algorithm training.

[0372] Local Actor Network Update Target:

[0373] Combining the optimization ideas of the PPO algorithm, importance sampling, and entropy regularization, this paper proposes a method for each edge server agent. Service deployment Actor network With request distribution Actor network Design a local policy to update the target. During the algorithm training... Wheel, defining the old multi-agent joint strategy as The new alliance strategy is intelligent agent The local importance sampling ratio is defined as the ratio of the probability of the old policy to that of the new policy in the current local state, i.e. ,in For intelligent agents The set of local Actor network parameters includes parameters from two sub-networks: service deployment and request distribution. The global importance sampling ratio is the product of the local importance sampling ratios of all agents, i.e. ,in It is the set of parameters for the joint strategy of multiple agents.

[0374] Based on the aforementioned importance sampling ratio, the service deploys an Actor network. The local update target, combined with the PPO pruning operation and entropy regularization term, is defined as:

[0375]

[0376] ,

[0377] in To determine the local importance sampling ratio for deploying the Actor network, The coefficient of the entropy regularization term, This is the entropy function, used to enhance the exploration capabilities of the strategy.

[0378] Correspondingly, a request is made to distribute the Actor network. The local update target is:

[0379]

[0380] ,

[0381] in To request the distribution of the local importance sampling ratio of the Actor network, For its corresponding entropy regularization coefficient, incorporate a knowledge-guided action mask. This ensures that strategy updates comply with knowledge constraints.

[0382] The global constraint optimization objective is constructed based on the global importance sampling ratio and the global constraint advantage function. It is used to strengthen the guiding role of constraints on policy updates and is defined as follows:

[0383] ,

[0384] In summary, the intelligent agent The overall goal of local updates can be formally expressed as:

[0385]

[0386] ,

[0387] Global Lagrange multipliers and penalty factor updates:

[0388] The global Lagrange multipliers are obtained using the projection gradient ascent method. Iterative updates are conducted to strictly ensure... The constraints and their update rules are as follows:

[0389] ,

[0390] in For the first The long-term discount of the wheel accumulates global constraints. This update rule can dynamically adjust the size of the multiplier according to the degree of constraint violation. The more serious the constraint violation, the larger the multiplier, and the stronger the penalty for the policy that violates the constraint.

[0391] In addition, the global constraint violation degree is defined as:

[0392] ,

[0393] Set constraint violation threshold The penalty factor is adjusted based on the comparison between the degree of violation and the threshold. This indicates that the degree of constraint violation exceeds the acceptable range, and the penalty factor is updated to... ,in The penalty growth coefficient is used; otherwise, the penalty factor remains unchanged, i.e. .

[0394] Global Critic network update target:

[0395] Global Reward Value Assessment Network With constrained value assessment network A centralized update approach is adopted, using the mean squared error (MSE) loss function as the update objective. The core objective is to minimize the error between the network value estimate and the global target value. The loss functions are defined as follows:

[0396] ,

[0397] ,

[0398] in, and These are the target reward value evaluation network and the target constraint value evaluation network, respectively. Their parameters are a lag copy of the current corresponding Critic network parameters. This lag update mechanism can effectively reduce the fluctuation of target value during training, avoid the instability of Critic network training caused by sudden changes in target value, and thus improve the convergence stability of the entire algorithm.

[0399] The specific algorithm flow is as follows:

[0400] Step S5-1: Initialization Phase. Initialize the knowledge enhancement service deployment Actor network. With request distribution Actor network Parameters; Initialize reward value Critic network With constrained value Critic network Parameters; initialize global Lagrange multipliers Punishment factor Penalty growth coefficient and cutting factor Initialize the global experience replay buffer. Initialize the parameters of the relational graph convolutional network R-GCN.

[0401] Step S5-2: Begin outer layer training iterations. For each training iteration... Based on current knowledge, enhance multi-agent joint strategies Interact with the edge computing environment to generate an interaction trajectory that includes a knowledge-enhanced global state, joint actions, knowledge-enhanced rewards, and constraint values. .

[0402] Step S5-3: Perform online updates to the edge computing multi-agent knowledge graph EC-MAKG, extract the multi-agent behaviors of the current time slot into experience triples and integrate them into the knowledge graph, and update the attribute features and experience relationship weights of each entity.

[0403] Step S5-4: Encode the updated knowledge graph using R-GCN and extract the local knowledge embedding feature vectors of each edge server agent. And generate knowledge-guided action masking based on prior relationships and deployment status. .

[0404] Step S5-5: Enhance the knowledge experience samples generated in each time slot Store in the global experience replay buffer .

[0405] Steps S5-6: Begin the update loop. From the experience buffer. Medium-sampled small batches of data were used to calculate the knowledge-enhancing reward advantage based on generalized advantage estimation. With knowledge-enhancing constraint advantages .

[0406] Step S5-7: Update the global reward value Critic network using the mean squared error loss function. and global constraint value Critic network Minimize the error between the network value estimate and the target value.

[0407] Step S5-8: Deploy the Actor network for the computing service Local update objective function ;Calculation request distribution Actor network Local update objective function ; Calculate the global constraint optimization objective function .

[0408] Step S5-9: Integrate the above sub-objectives with the Lagrange augmentation penalty term to calculate the local update overall objective function for each agent. .

[0409] Step S5-10: Update the service deployment Actor network parameters using gradient descent. and request distribution of Actor network parameters Synchronously update R-GCN network parameters.

[0410] Step S5-11: Update the global Lagrange multipliers using the projected gradient ascent method. Calculate the constraint violation degree of the current policy. With violation threshold .

[0411] Step S5-12: Determine the degree of constraint violation. If the degree of constraint violation exceeds the acceptable range, i.e. Then the penalty factor will be updated to Otherwise, the penalty factor remains unchanged, i.e. .

[0412] Step S5-13: After the update loop ends, the weights and entity attributes of each experience relationship in the knowledge graph are updated synchronously to complete the training iteration of the current round.

[0413] Step S5-14: After reaching the preset maximum number of training rounds, save the parameters of the trained service deployment Actor network, request distribution Actor network, and each Critic network; when applied, each edge server agent autonomously generates joint decisions for service deployment and request distribution based on local observation status and knowledge embedding features through the trained network.

[0414] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.

Claims

1. A method for deploying and distributing knowledge graph-enhanced services in a cloud-edge environment, characterized in that, The method includes the following steps: Step S1: Considering the characteristics of heterogeneous and limited server resources, dynamic arrival of service requests, and differences in service cold / hot start costs in edge computing networks, construct an edge service deployment model, a service request distribution model, and a service processing latency model; with the goal of minimizing the weighted combination of system service latency and deployment costs, construct a quantitative representation model of overall system performance. Step S2: For edge server storage capacity constraints, computing power constraints, and request distribution consistency constraints, construct a joint optimization mathematical model for service deployment and request distribution that is jointly optimized for service latency and deployment cost. This model is a mixed integer nonlinear programming problem. Step S3: Construct the edge computing multi-agent knowledge graph EC-MAKG, solidify the inherent attributes and prior constraints of servers, services, and requests in the form of triples, and dynamically accumulate effective deployment and distribution experience in the multi-agent collaboration process; use the relational graph convolutional network R-GCN to perform local knowledge embedding on the knowledge graph and extract the knowledge feature vectors of each edge server agent; Step S4: Remodel the joint optimization mathematical model into a knowledge graph-enhanced multi-agent restricted Markov decision process KG-MA-CMDP, defining the knowledge-enhanced state space, the knowledge-constrained action space, the knowledge-enhanced reward function, and the constraint function, respectively. Step S5: Based on the centralized training-distributed execution framework, design the knowledge graph-driven multi-agent security enhancement proximal policy optimization algorithm KG-S-MAPPO for solving; During the centralized training phase, the global state and knowledge features are integrated through a knowledge-enhanced global Critic network. During the distributed execution phase, each edge server agent autonomously generates service deployment and request distribution decisions based on its local state and knowledge embedding features. Through a knowledge-guided action shielding mechanism and a Lagrange augmented objective constraint optimization mechanism, a joint optimization strategy for service deployment and request distribution is obtained under the premise of satisfying system resource constraints.

2. The method according to claim 1, characterized in that: In step S1, the edge computing system model is defined: Consider a typical edge computing system that includes a set of edge servers, denoted as . ,in This indicates the total number of edge servers; Assume the system has Class of services, denoted as Each service They have different startup overheads. Storage requirements and calculation intensity These respectively reflect the computational load of the service during the initialization phase, the memory resources required for operation, and the computational complexity required for processing a unit of data; Based on this, each service request is modeled as a corresponding type. The computational task has an input data size of Therefore, the total computing resources required to process this request are CPU cycles, this metric characterizes the computational load intensity of a single task; make Denotes the set of discrete time slots, where This represents the total number of time slots. This indicates the duration of each time slot, in each time slot In this context, users initiate different types of service requests to their nearest edge servers, which are then recorded. In time slot Inner edge server Received service type The number of requests is used to characterize the distribution of the system's business load in the spatiotemporal dimensions.

3. The method according to claim 1, characterized in that: In step S1, the service deployment model is defined: set up For the binary decision variables of service deployment, where Indicates time slot Service Deployed on edge servers If it is above, then it is 0; otherwise, it is 0. The total storage footprint of services deployed on each edge server must meet capacity constraints, namely: ， ， in, Represents edge server Storage capacity, Indicates service The above constraints ensure the feasibility of service deployment decisions under physical resource limitations, given the storage requirements. Define the following two states: Cold start: If and Then the service In the time gap The transition from a non-deployed state to a deployed state requires initial loading, which corresponds to the cold start process. Hot start: If and If the service remains in the deployment state, its startup overhead is negligible, which corresponds to a hot start. To uniformly characterize service startup behavior, the following indicator function is introduced: ， in ,when A cold start occurs in the current time slot; otherwise, it indicates a warm start or an undeployed state. Based on the above definition, the additional latency introduced by a cold start can be expressed as: ， in, Represents edge server For service The allocated computing power This expression represents the computational overhead required for service initialization, and it quantitatively describes the relationship between cold start latency and resource allocation. The service deployment cost is modeled as the sum of the following two parts: 1) Cold start configuration cost This is used to characterize the initialization overhead when the service is deployed for the first time. 2) Hot start maintenance costs This describes the system resource overhead consumed by a service during its continuous operation. Therefore, service On the edge server During the time gap The total deployment cost is expressed as: 。 4. The method according to claim 1, characterized in that: In step S1, the service request distribution model is defined: definition For edge servers In the time gap Regarding services The request distribution strategy, in which Indicates from server Forwarded to server The request ratio must meet the following constraints: ， ， ， The third constraint ensures that requests can only be forwarded to edge servers where the corresponding services have been deployed. Given a request distribution strategy, the edge server In the time gap Received services The total computational load is expressed as: ， This expression describes the redistribution process of requests from all edge servers in the system, while also satisfying the following computational capacity constraints: ， That is, the computing load received by each edge server per unit of time must not exceed its maximum processing capacity. server Regarding services If the incoming load exceeds its processing limit, the excess will be forwarded to the cloud server. To handle this, define the overload limit as: ， set up For time slots Inner edge server To cloud server If the effective transmission rate is 0, then the forwarding delay for transmissions exceeding the load is: ， Cloud computing latency is: ， in For cloud servers as services The allocated computing power.

5. The method according to claim 1, characterized in that: In step S1, a service processing latency model is defined: Service processing latency is modeled as two components: forwarding latency, computation latency, and potential cold start latency; (1) Forwarding delay: When service requests need to be forwarded between edge servers, additional transmission latency will occur, defining the time required from the server. to server Forwarding service The forwarding delay of the request is: ， Where the numerator represents the amount of data that needs to be transmitted per unit time, and the denominator represents the amount of data that needs to be transmitted per unit time. Indicates the effective transmission rate between servers; In multi-hop network environments, the transmission rate is typically determined by the bottleneck link in the path; that is, the throughput of the entire path is limited by the minimum link capacity on that path. , indicating at time from arrive The set of path links. (Note: The original text contains some formatting errors and inconsistencies. A more accurate translation would require For time slots Inner Link Given the available bandwidth, the transmission rate satisfies: ， When data transmission path When selecting the shortest path based on hop count or the path based on maximum available bandwidth, the transmission rate... All are limited by the minimum link bandwidth on the selected path; In a simpler scenario, if and If the connection between them is a single-hop direct connection, then This can be directly taken as the time slot of the direct link. Instantaneous capacity; (2) Calculate the delay: For those assigned to edge servers The computation latency of a service request can be characterized using the classic M / M / 1 queuing model, defined as: ， in, Indicates service On the server The arrival rate will decrease significantly when the system load approaches its processing capacity, and the computation latency will increase significantly. Taking into account cold start latency, forwarding latency, and computation latency, the total processing latency for a service request can be expressed as: ， Finally, regarding the time interval Arrival at the server services The overall service latency of a request is determined by the slowest processing path, i.e.: 。 6. The method according to claim 1, characterized in that: In step S2, Based on the above model, construct a system based on service latency. Deployment costs A joint optimization problem with weighted combination as the objective, introducing weight coefficients. Then the time interval The optimization objective is: ， ， ， ， ， in, This represents the service deployment decision variable. This indicates the request distribution strategy.

7. The method according to claim 1, characterized in that: In step S3, a knowledge graph is introduced to perform structured modeling of entity relationships and collaborative experience in the edge computing system, and based on this, a knowledge graph-driven multi-agent security enhancement near-end strategy optimization algorithm is proposed. EC-MAKG Construction and Updates: EC-MAKG is a structured semantic network that represents domain knowledge of multi-agent collaboration in edge computing, presented in the form of triples. As the basic representation unit, where Indicates the head entity. Indicates a relationship. Indicates the tail entity; EC-MAKG’s entity system covers the core participants in multi-agent collaborative scenarios of edge computing, while its relation system covers the inherent relationships and dynamic behaviors between entities. Together, they constitute the core semantic foundation of the knowledge graph. Entity set It includes three core entities, each with a unique identifier and corresponding attribute features. These attribute features correspond one-to-one with the parameters of the edge computing system model constructed in this paper. The server intelligent agent entity... Corresponding edge server intelligent agent Its attribute set is Each parameter corresponds to a core characteristic of the server agent, such as its unique identifier, storage resources, and computing resources; service entity Corresponding to various services in the edge computing system Its attribute set is These respectively represent the service's unique identifier, startup overhead, storage requirements, computational intensity, and data scale; Request entity The attribute set for each service request generated within a time slot is as follows: These correspond to the unique identifier of the request, the type of service requested, the server agent that initiated the request, and the request arrival rate within that time slot, respectively. Relationship set Based on knowledge type, they are divided into two main categories: prior relationships and empirical relationships. Prior relationships refer to inherent, static connections between entities, determined by the inherent attributes of the edge computing system. Empirical relationships, on the other hand, refer to dynamic and effective behavioral connections generated during multi-agent collaboration, extracted from the training experience of multi-agent reinforcement learning. These two categories contain six core types, among which... The relationship indicates that the storage resources of the edge server agent can meet the deployment requirements of a certain service. The relationship defines the topology of the physical communication links between edge server agents; The relationship clearly defines the matching relationship between service requests and corresponding service types; The relationship reflects the dynamic deployment behavior of edge server agents towards services; The relationship ensures that the request distribution behavior meets the system feasibility constraints; The relationship describes the request distribution strategy between edge server agents; EC-MAKG adopts a hierarchical construction strategy. First, it builds a prior knowledge layer offline based on the inherent attributes and static associations of the edge computing system. Then, it builds an experience knowledge layer online based on the training experience of multi-agent reinforcement learning. The core of the prior knowledge layer is to provide basic constraints for multi-agent collaboration. It is constructed offline, and three types of prior relations are built based on system constraints and static associations. when At that time, establish ; when At that time, establish ; Based on the request type, establish ; After construction, the prior knowledge is stored in a structured manner using an RDF graph model; The experiential knowledge layer is constructed online based on multi-agent reinforcement learning training experience. It filters effective samples that meet system constraints and offer high rewards from the experience replay buffer, extracts dynamic relationships from these samples, and generates experiential triples. Deploy services, establish ; Satisfying constraints ,Establish ; Forward the request and establish ; When merging experiential triples with prior knowledge layers, weights are assigned to experiential relations based on reward values ​​and constraint satisfaction. At the same time, a knowledge pruning mechanism is introduced to periodically remove low-weight and invalid triples to ensure the effectiveness of the knowledge graph. After each time slot ends, an update operation will be performed. First, effective experience will be extracted from the multi-agent behavior of the current time slot to generate new experience triples, which will then be integrated into the knowledge graph. Second, the weights of existing experience relationships in the knowledge graph will be cumulatively updated based on the reward value and constraint satisfaction of the current time slot. Finally, the attribute features of each entity will be updated. A local knowledge embedding method based on R-GCN: knowledge embedding is performed using R-GCN; An embedding strategy based on local subgraph extraction is employed for each edge server agent. Using the corresponding node as the center, extract its one-hop or two-hop neighborhood from EC-MAKG to form a local subgraph. The subgraph contains service nodes, request nodes, and neighbor server nodes related to the agent's decision-making, and retains multiple types of relationship connections; In the time slot First, the nodes in the local subgraph are initialized based on entity attributes; then, the R-GCN with shared parameters is used in the subgraph. The information propagation and aggregation of multi-layered relationship perception are carried out, enabling the node representation to gradually integrate the structural semantic information of its multi-hop neighborhood. After layer propagation, the embedded representation of the central node is extracted as the agent. Knowledge feature vector .

8. The method according to claim 1, characterized in that: The specific algorithm flow in step S3 is as follows: Step S3-1: Set the algorithm input, including the current time slot. Global knowledge graph under Initial feature vectors of each entity node Target edge server intelligent agent node and the maximum number of convolutional layers in a graph convolutional network. ; Step S3-2: Using the target agent node Centered on the global knowledge graph Extract all nodes and corresponding relation edges within its two-hop neighborhood to form a local association subgraph. ; Step S3-3: Based on local subgraph The set of nodes contained therein is used to extract the attribute features of the corresponding entities and generate the initial feature matrix of the subgraph nodes. ; Step S3-4: Start iterative execution Layer graph convolution operation, in each convolutional layer In the context of subgraphs Each node within According to different relationship types Aggregate its neighbor nodes Based on the feature information, perform the following aggregation and update calculations: ， in, It is a non-linear activation function. and The first Layer-specific learnable weight matrices and self-loop weight matrices. This is the normalization constant; Step S3-5: Completed After multiple convolutional iterations, the target agent nodes are extracted from the final generated node feature matrix. corresponding feature vector And assign it to the local knowledge embedding feature vector. ; Step S3-6: Output the target agent In the current time slot Local knowledge embedding feature vectors It is used for subsequent knowledge enhancement in the multi-agent state representation and decision-making process.

9. The method according to claim 1, characterized in that: In step S4, Knowledge embedding features are incorporated into the Multi-Agent Restricted Markov Decision Process (MA-CMDP) to construct a knowledge graph-driven KG-MA-CMDP. The KG-MA-CMDP consists of octet... Formal representation, in which The constructed edge computing multi-agent knowledge graph is EC-MAKG, and all other elements are redefined by incorporating knowledge embedding features on the basis of the original MA-CMDP. Knowledge-enhanced state space: By fusing the agent's local knowledge embedding features with the original local state features, a knowledge-enhanced state representation is constructed, which is then applied in time slots. Below, intelligent agents Knowledge Enhancement Local State Defined as: ,in, For the agent in the original MA-CMDP The local state vector characterizes the dynamic information of the system; For agents extracted from EC-MAKG Local knowledge is embedded in the feature vector, which is obtained by analyzing the node... The local subgraph centered on the entity is obtained through R-GCN encoding and is used to represent the semantics of the multi-relation structure between entities; This represents element-wise multiplication; These are learnable gating coefficients used to adaptively adjust the fusion ratio of original state features and knowledge-embedded features during training. Based on this, the system's knowledge-enhanced global state is... It is composed of the knowledge-enhanced local states of all edge server agents, and incorporates the core embedding features of the global knowledge graph to represent global knowledge associations at the system level. Its definition is... ,in The core embedded features of the global knowledge graph; Knowledge-constrained action space: Based on the original MA-CMDP action space, prior relation constraints from EC-MAKG are introduced to construct a knowledge-constrained action space for the agent. Knowledge-constrained local action space For the original local action space A subset of , formally defined as: ， in, Indicates a satisfying relationship. Represents the set of prior relations in EC-MAKG; Knowledge-enhanced reward function: The negative value of the system's global optimization objective is directly used as the reward function, in the following form: ， Building upon this, experiential knowledge from knowledge graphs is introduced to enhance rewards. By measuring the degree of matching between joint actions and EC-MAKG, the knowledge-enhanced global reward function is obtained. : ， in, The weighting coefficient for the experience-knowledge matching reward is used to adaptively balance the contribution of basic decision-making reward and knowledge matching reward. For multi-agent joint action The degree of matching with EC-MAKG experiential knowledge, with a value range of [value range missing]. ; Constraint function design: To address the storage resource constraints of edge computing systems, constraint functions are designed. An exponential reconstruction method is used to transform the storage constraints of multiple servers into a single global constraint function. Storage constraints Defined as: ， Sum the reconstructed storage constraints of all edge servers to obtain the time slot. global constraint functions under : ， The threshold of the global constraint function is the total number of edge servers. ,when At that time, all servers met the storage resource constraints; when At that time, at least one server violated the storage constraint, and The larger the value, the higher the degree of constraint violation in the system; The EC-MAKG prior knowledge constraints are transformed into penalty terms and integrated with the global constraint function to construct a knowledge-enhanced global constraint function. : ， in, This is a weighting coefficient for prior knowledge conflict penalties, used to adjust the degree of influence of knowledge conflict penalties on overall constraints; For multi-agent joint action The degree of conflict with EC-MAKG prior knowledge, with a value range of [value missing]. The higher this value, the greater the degree to which the current collaborative decision violates the inherent constraints of the system, and the greater the penalty that needs to be imposed accordingly; Optimization goal: Let's consider a joint strategy of KG-MA-CMDP. For its optimal strategy, we have: ， in, For a set of feasible strategies for multiple agents, Accumulated global rewards for long-term discounts are defined as follows: ， in, As a discount factor, This represents the total number of time slots. Representation Strategy The expected value of the following mathematical expression Accordingly, To accumulate global constraints for long-term discounts, namely: ， in, This is the global constraint threshold, whose value is an inherent threshold setting based on the system's storage resource constraints, i.e. ,when When, it indicates the strategy Meeting the storage resource constraints of all edge servers during the training period is a feasible strategy.

10. The method according to claim 1, characterized in that: In step S5, Based on a centralized training-distributed execution framework, a knowledge-enhanced multi-agent network was designed. The system's operation flow in each time slot is as follows: Each edge server agent interacts with the simulation environment to generate a global state that includes knowledge enhancement. Joint actions Knowledge Enhancement Rewards Knowledge Enhancement Constraints and the enhanced global state in the next time slot Based on empirical samples, the edge computing multi-agent knowledge graph EC-MAKG is updated online according to the latest system state, and the R-GCN encoder is used to extract the local knowledge embedding vectors of each agent. and the mask used for motion masking Subsequently, the local dynamic state of each agent. With corresponding knowledge embedding Perform weighted fusion to generate knowledge-enhanced local states. ; further all intelligent agents Combined with global knowledge embedding, it forms a knowledge-enhanced global state. The aforementioned experience samples are uniformly stored in the global experience replay buffer; During the training phase, small batches of experience data are sampled from the buffer. The centralized evaluation module contains two knowledge-enhanced Critic networks to... Given input, output a reward value estimate. With constrained value estimation And calculate the reward advantage of knowledge enhancement accordingly. With constraint advantage On the distributed execution side, each agent independently maintains a dual-Actor network: a service deployment Actor network. and request distribution Actor network Finally, all network parameters are updated with gradients based on the advantage estimation of knowledge enhancement, and the empirical relation weights and entity attributes in EC-MAKG are also updated synchronously. Knowledge-enhanced local actor network design: Each intelligent agent Configure two independent Actor networks to decouple service deployment and request distribution decisions; service deployment Actor network. Enhancing local state with knowledge As input, output the service deployment probability distribution that satisfies the prior constraints; for services not present in EC-MAKG... The probability of the service related to the relationship is directly set to zero, and the discrete deployment action is obtained through Bernoulli sampling. Request to distribute Actor network Simultaneously receive With action mask The system outputs the distribution parameters of continuously distributed actions, forces the dimensions of actions that violate constraints to zero, and finally obtains the normalized distribution ratio. Joint actions are defined as Global coordinated actions Therefore, intelligent agents The local joint strategy can be expressed as: ， Among them, joint actions ; The global joint policy of multiple agents is composed of the product of their local policies, reflecting the independence and global coordination of distributed decision-making. ， Knowledge-enhanced action shielding mechanism: Based on the existing action masking, EC-MAKG prior constraints are introduced to construct a dual filtering mechanism. Specifically, the agent... First, a priori mask is generated based on prior relations. At the same time, based on the deployment decision A deployment state mask is generated, and a distribution consistency mask is obtained through global aggregation. fusion mask Accurately identify and effectively distribute to target audiences, and apply them to The output of this function has a zero probability of forcing an invalid action. Knowledge-enhanced global Critic network design: The centralized evaluation module contains two independent knowledge-enhancing Critic networks: a reward value network. With constrained value network Both enhance the global state through knowledge. Given the input, output the expected long-term reward respectively. Violation of expectations with long-term constraints To accurately evaluate the merits of joint actions, generalized advantage estimation (GAE) is used to calculate the reward advantage and constraint advantage of knowledge enhancement: ， ， in For GAE coefficients, KG-S-MAPPO algorithm: The KG-S-MAPPO algorithm is based on the MAPPO algorithm and integrates the structured knowledge of EC-MAKG with the constraint optimization idea of ​​secure deep reinforcement learning. It extends the target policy of policy update by both knowledge enhancement and security constraints, while incorporating the constraint optimization capability of secure deep reinforcement learning. Lagrange augmentation target: Introducing global Lagrange multipliers With global penalty factor Construct the multi-agent Lagrange augmented objective function, which is formally defined as follows: ， The first term is the negative long-term global reward for knowledge enhancement; the second term is the augmenting Lagrange penalty term, which mainly penalizes policies that violate global constraints; and the third term is the regularization term, used to ensure the stability of the Lagrange multiplier update process. Local Actor Network Update Target: Combining the PPO algorithm optimization concept, importance sampling, and entropy regularization, for each edge server agent... Service deployment Actor network With request distribution Actor network Design a local policy to update the target, in the first training step of the algorithm. Wheel, defining the old multi-agent joint strategy as The new joint strategy is intelligent agent The local importance sampling ratio is defined as the ratio of the probability of the old policy to that of the new policy in the current local state, i.e. ,in For intelligent agents The set of local Actor network parameters includes parameters from two sub-networks: service deployment and request distribution. The global importance sampling ratio is the product of the local importance sampling ratios of all agents, i.e. ,in It is the set of parameters for the joint strategy of multiple agents; Based on the aforementioned importance sampling ratio, the service deploys an Actor network. The local update target, combined with the PPO pruning operation and entropy regularization term, is defined as: ， in To determine the local importance sampling ratio for deploying the Actor network, The coefficient of the entropy regularization term, It is the entropy function. Correspondingly, a request is made to distribute the Actor network. The local update target is: ， in To request the distribution of the local importance sampling ratio of the Actor network, For its corresponding entropy regularization coefficient, incorporate a knowledge-guided action mask. , The global constraint optimization objective is constructed based on the global importance sampling ratio and the global constraint advantage function. It is used to strengthen the guiding role of constraints on policy updates and is defined as follows: ， In summary, the intelligent agent The overall goal of local updates can be formally expressed as: ， Global Lagrange multipliers and penalty factor updates: The global Lagrange multipliers are obtained using the projection gradient ascent method. Iterative updates are conducted to strictly ensure... The constraints and their update rules are as follows: ， in For the first The long-term discount of the cycle accumulates global constraints. This update rule can dynamically adjust the multiplier size according to the degree of constraint violation. The more severe the constraint violation, the larger the multiplier, and the stronger the penalty for the policy that violates the constraint. In addition, the global constraint violation degree is defined as: ， Set constraint violation threshold The penalty factor is adjusted based on the comparison between the degree of violation and the threshold. This indicates that the degree of constraint violation exceeds the acceptable range, and the penalty factor is updated to... ,in The penalty growth coefficient is used; otherwise, the penalty factor remains unchanged, i.e. , Global Critic network update target: Global Reward Value Assessment Network With constrained value assessment network A centralized update approach is adopted, using the mean squared error (MSE) loss function as the update objective. The core objective is to minimize the error between the network value estimate and the global target value. The loss functions are defined as follows: ， ， in, and These are the target reward value evaluation network and the target constraint value evaluation network, respectively, and their parameters are lagged copies of the parameters of the corresponding Critic network. The specific algorithm flow in step S5 is as follows: Step S5-1: Initialization phase, initialize the Actor network for deploying the knowledge enhancement service. With request distribution Actor network Parameters; Initialize reward value Critic network With constrained value Critic network Parameters; initialize global Lagrange multipliers Punishment factor Penalty growth coefficient and cutting factor Initialize the global experience replay buffer. Initialize the parameters of the relational graph convolutional network R-GCN; Step S5-2: Begin outer layer training iterations. For each training iteration... Based on current knowledge, enhance multi-agent joint strategies Interact with the edge computing environment to generate an interaction trajectory that includes a knowledge-enhanced global state, joint actions, knowledge-enhanced rewards, and constraint values. ; Step S5-3: Perform online updates of the edge computing multi-agent knowledge graph EC-MAKG, extract the multi-agent behaviors of the current time slot into experience triples and integrate them into the knowledge graph, and update the attribute features and experience relationship weights of each entity. Step S5-4: Encode the updated knowledge graph using R-GCN and extract the local knowledge embedding feature vectors of each edge server agent. And generate knowledge-guided action masking based on prior relationships and deployment status. ; Step S5-5: Enhance the knowledge experience samples generated in each time slot Store in the global experience replay buffer ; Steps S5-6: Start the update loop from the experience buffer. Medium-sampled small batches of data were used to calculate the knowledge-enhancing reward advantage based on generalized advantage estimation. With knowledge-enhancing constraint advantages ; Step S5-7: Update the global reward value Critic network using the mean squared error loss function. and global constraint value Critic network Minimize the error between the network value estimate and the target value; Step S5-8: Deploy the Actor network for the computing service Local update objective function ;Calculation request distribution Actor network Local update objective function ; Calculate the global constraint optimization objective function ; Step S5-9: Integrate the above sub-objectives with the Lagrange augmentation penalty term to calculate the local update overall objective function for each agent. ; Step S5-10: Update the service deployment Actor network parameters using gradient descent. and request distribution of Actor network parameters Synchronously update R-GCN network parameters; Step S5-11: Update the global Lagrange multipliers using the projected gradient ascent method. Calculate the constraint violation degree of the current policy. With violation threshold ; Step S5-12: Determine the degree of constraint violation. If the degree of constraint violation exceeds the acceptable range, i.e. Then the penalty factor will be updated to Otherwise, the penalty factor remains unchanged, i.e. ; Step S5-13: After the update loop ends, the weights and entity attributes of each experience relationship in the knowledge graph are updated synchronously to complete the training iteration of the current round. Step S5-14: After reaching the preset maximum number of training rounds, save the parameters of the trained service deployment Actor network, request distribution Actor network, and each Critic network; when applied, each edge server agent autonomously generates joint decisions for service deployment and request distribution based on local observation status and knowledge embedding features through the trained network.

Images