Complex cloud edge collaborative service deployment method based on multi-agent reinforcement learning

Abstract

The invention belongs to the technical field of cloud edge collaborative service deployment, and relates to a complex cloud edge collaborative service deployment method based on multi-agent reinforcement learning. The method comprises the following steps: S1, constructing a dual-agent collaborative deployment framework; s2, an environment module collects resource state data of a cloud side node and topological features of a physical link, and generates preprocessing features; s3, inputting the preprocessing features into a node agent network, training and outputting a target node index, and setting an output result as an action of a node agent; s4, the link agent obtains real-time topological characteristics from a physical link between the source node and the target node, inputs the topological characteristics to the link agent network, and outputs optimal path selection and frequency slot block combination; s5, the evaluation module calculates a collaborative score; and S6, the node agent and the link agent update network parameters according to the instant reward signal. According to the invention, collaboration of cloud and edge computing resources and link resources is realized, and high-quality service provision with low delay and high resource availability is completed.

Description

Technical Field

[0001] This invention belongs to the field of cloud-edge collaborative service deployment technology, specifically relating to a complex cloud-edge collaborative service deployment method based on multi-agent reinforcement learning. Background Technology

[0002] Traditional cloud-edge deployment typically employs integer linear programming to solve static programming problems and heuristic algorithms to solve dynamic optimization problems. In recent years, with the gradual maturation of deep learning methods, reinforcement learning methods, through real-time interaction with environmental information and immediate feedback, are suitable for solving dynamic decision-making problems in zero-shot scenarios. Recent research has proposed single-agent reinforcement learning solutions for problems such as optical network routing planning, transmission quality assurance, and resource allocation. While single-agent reinforcement learning is suitable for solving single-objective optimization problems, cloud-edge collaborative deployment is a two-step optimization problem, involving cloud-edge node deployment and network routing spectrum resource allocation. Therefore, single-agent reinforcement learning cannot effectively solve the cloud-edge collaborative deployment problem. The complete cloud-edge collaborative service deployment problem is an NP-hard problem, with its variables being highly coupled. In real-world large-scale networks, it is difficult to solve this problem using an optimal solution. Therefore, this application proposes a complex cloud-edge collaborative service deployment method based on multi-agent reinforcement learning, achieving the coordination of cloud, edge computing resources, and link resources to provide high-quality services with low latency and high resource availability. Summary of the Invention

[0003] In view of this, the present invention proposes a method for deploying complex cloud-edge collaborative services based on multi-agent reinforcement learning.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for deploying complex cloud-edge collaborative services based on multi-agent reinforcement learning includes the following steps: S1. Construct a dual-agent collaborative deployment framework, including an environment module, an agent module, and an evaluation module. The agent module includes node agents and link agents, and the node agents include cloud agents and edge agents. S2. The environment module collects resource status data of cloud-edge nodes and topology features of physical links in real time, and generates preprocessed features of node agents and link agents. S3. Input the preprocessed features into the node agent network, train the output target node index, and set the output result as the action of the node agent. S4. After the node agent completes its action, the link agent obtains the processed real-time topology features from the physical link between the source node and the target node based on the target node index, inputs the topology features into the link agent network, and outputs the optimal path selection and frequency slot block combination. S5. The evaluation module calculates the collaborative score of the node agent and the link agent through the value function and feeds it back to the policy network of the node agent and the link agent. S6. The node agent and link agent update the network parameters based on the real-time reward signal, so that the node agent and link agent can achieve adaptive learning through network iteration.

[0005] Furthermore, the intelligent agent module includes Individual cloud intelligent agent, The method comprises one edge agent and one link agent, whose state space consists of the cloud-edge node resource state and the link topology state; the method further includes: The joint decision-making module is used to define the joint action space. ,in For the action space of cloud intelligent agents, For the action space of the edge agent. This defines the action space of the link agent; and uses a joint state transition function. This indicates the execution of joint actions. The following is a state Transferred to The probability distribution is used to calculate the state transition probability; The link resource allocation module is configured to allocate the action space of the link agent. Mapping to physical path-slot matrix Column space; actions of linked agents This represents the probability of selecting the w-th slot of the k-th path. With matrix The column vectors are positively correlated; The dynamic adjustment module dynamically adjusts the size of the state space and action space of the link agent based on the number of paths k and the number of frequency slots w, and when the action... probability If the action is higher than other actions, the corresponding frequency slot will be selected first; The reward and strategy optimization module is configured to define reward functions for cloud agents, edge agents, and link agents respectively. , , ; The Nash policy generation module employs a centralized training and distributed execution framework, solving for the joint optimal policy through the Nash equilibrium function. This is to minimize business processing latency and maximize resource availability.

[0006] Furthermore, the workflow of the Nash policy generation module includes: The cloud agent, edge agent, and link agent each update their evaluation functions. , , ; Constructing multi-agent game scenarios … To generate the optimal deployment scheme, the multi-agent equilibrium function is defined as follows: , Representing cloud, edge, and link intelligent agents In state Below, other intelligent agents corresponding to it Adopting a joint strategy The following is obtained using Nash equalization Value; among which , For a game situation to be resolved by a Nash equilibrium, the Nash equilibrium solution must satisfy the following definition: It is a cloud, edge, and link intelligent agent exist In this state, latency is minimized and resource availability is optimized. It is a cloud, edge, and link intelligent agent The set of all possible strategies, The optimal joint strategy for cloud, edge, and link agents under Nash equilibrium.

[0007] Furthermore, the cloud-edge collaborative service deployment training process based on multi-agent reinforcement learning includes the following five steps. The training process, through steps 1 to 5, aims to balance node computing resources and link bandwidth resources among cloud, edge, and link agents in each learning process. Through collaboration among agents, it achieves the optimization goal of maximizing both latency requirements and network resources, enabling flexible and efficient deployment of complex cloud-edge collaborative services. Step 1: The cloud agent, edge agent, and link agent collect features of cloud-edge nodes and links from the deployment environment to construct a cloud-edge node feature matrix. Link feature vector , and serve as input data for each intelligent agent; Step 2: The cloud agent and the edge agent perform linear transformation calculations on the input cloud and edge node features through a policy network. Where W is the weight matrix and B is the bias term, the output is the probability distribution of the cloud and edge nodes to be deployed. And select the nodes to be deployed from the probability distribution according to the random greedy strategy, as the decision action of the agent; Step 3: The link agent performs joint computation on the input node actions and link features through the policy network to generate... Probability distribution of the shortest path intermediate frequency slot block Based on the probability distribution, the frequency slot blocks to be selected on the output link are finally formed, thus forming a complete node-link action sequence. Step 4: After the cloud agent, edge agent, and link agent each execute their selected actions, they obtain the corresponding cloud and edge node rewards from the environment module. , Link rewards ; Step 5: The evaluation module analyzes the state-action relationship through the policy network and constructs evaluation functions based on cloud node rewards, edge node rewards, and link rewards respectively. and The rationality of the actions of cloud agents, edge agents, and link agents is quantitatively evaluated; based on the evaluation results, the agents are driven to perform feedback learning, iteratively optimize the next action selection, and continuously improve the deployment plan.

[0008] Furthermore, the training mechanism for the cloud-edge collaborative service deployment training process includes: Two self-learning training phases are set up, one for training the node agent network and the other for training the link agent network. In the node agent training phase, the loss function of the node agent network is calculated by outputting action probabilities and combining local reward signals and global evaluation network values. The local reward signals reflect the real-time performance of the node agent in cloud-edge collaborative service deployment, and the global evaluation network values ​​represent its comprehensive performance throughout the entire deployment process. In the training phase of the link agent, based on the determined source node and destination node, and combined with the routing and slot selection tasks, the link agent network is used to calculate the loss function of the link agent network through the local reward signal and the output of the link value network. The Floyd shortest path algorithm is used to determine candidate paths; The reward mechanism of the link agent is constructed based on short-term and long-term reward signals, and the discounted cumulative total reward is used as the training objective. An ϵ-greedy strategy is adopted to balance exploration and exploitation, thereby improving the strategy optimization capability in complex coupled environments; By continuously updating the parameters of nodes and link agents through a feedback learning mechanism, the efficient deployment of complex cloud-edge collaborative tasks can be achieved.

[0009] Furthermore, in the link training phase, after successful service deployment, an immediate reward of 1 is set, and link data access is triggered to train routing and frequency slot selection.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: The aforementioned technical solution, through the construction of a dual-agent collaborative deployment framework and a hierarchical training mechanism, achieves superior deployment efficiency and resource adaptability compared to existing technologies in complex cloud-edge collaborative scenarios. Firstly, this application innovatively decouples the node agent and the link agent. The cloud agent, edge agent, and link agent focus on node resource scheduling and link transmission optimization, respectively. This division of labor and collaboration overcomes the optimization bottleneck caused by decision coupling in traditional single-agent deployment methods, enabling parallel decision-making for the allocation of computing resources to cloud and edge nodes and the matching of frequency slots in physical links, significantly improving the real-time performance of large-scale cluster deployments. For example, when the node agent outputs the target node index through the policy network, the link agent can simultaneously calculate the optimal path based on topological features. The collaborative processing of the group of agents accelerates the task processing flow, avoiding the waiting delays in serial decision-making.

[0011] Secondly, this invention employs a closed-loop training mechanism of preprocessing features, policy network output, and evaluation feedback iteration to achieve dynamic adaptive optimization of the deployment strategy. Resource status data and topology features collected in real time by the environment module are transformed into probability distribution decisions by the agent module. The evaluation module then quantifies collaborative scoring through a value function and generates an immediate reward signal. This feedback mechanism enables the agent to continuously learn the patterns of environmental changes. Compared to the limitations of traditional static deployment algorithms that rely on preset rules or offline training models, this application dynamically adjusts network parameters through online learning. For example, the node agent optimizes the selection of nodes to be deployed based on the linear transformation results of the cloud-edge node feature matrix, while the link agent adjusts the frequency slot combination strategy based on the joint calculation of node actions and link features. Continuous model updates address environmental dynamism, thus maintaining high performance even when cloud-edge resources fluctuate or link topology changes.

[0012] Furthermore, this invention incorporates multi-dimensional optimization objectives into feature engineering and decision logic. The policy network of the node agents not only considers the resource status of cloud-edge nodes but also outputs a probability distribution to balance deployment load and resource utilization. The link agents, on the other hand, optimize frequency slot block combinations in the shortest path, taking into account both transmission latency and bandwidth efficiency. This multi-objective collaborative optimization mechanism avoids system performance imbalances caused by single-index optimization. For example, traditional methods may lead to load concentration due to prioritizing high-resource nodes, while this application selects nodes from the probability distribution using a random greedy strategy and dynamically adjusts decision preferences based on the collaborative scoring of the evaluation module, achieving global optimization of resource utilization and service quality. Simultaneously, the layered agent architecture gives the system good scalability, allowing it to adapt to more complex cloud-edge collaborative scenarios by adding agent types or expanding feature dimensions.

[0013] Finally, this invention enhances the interpretability and reproducibility of deployment strategies through a mathematical decision-making process. Steps such as linear transformation operations of node agents and generation of probability distributions of link agents are defined by explicit mathematical formulas, including the product of cloud-edge node feature matrices and weight matrices, and the calculation of probability distributions for frequency slot block selection. This transforms the decision-making logic from traditional experience-driven to data-driven, facilitating the optimization of strategy network performance by adjusting weight matrices and bias terms. This standardized modeling approach not only reduces the cost of manual parameter tuning but also lays the foundation for cross-scenario transfer learning. It allows for adjustments to feature matrix dimensions or value function parameters based on different business needs, enabling rapid adaptation to diverse cloud-edge collaborative services such as video analytics and industrial IoT. Attached Figure Description

[0014] The present invention will now be described in further detail with reference to the accompanying drawings.

[0015] Figure 1 This is a schematic diagram of cloud-edge collaborative service deployment based on multi-agent reinforcement learning according to the present invention.

[0016] Figure 2 This is a schematic diagram of the link state action space of the present invention. Detailed Implementation

[0017] To better understand the present invention, the content of the invention is further clearly illustrated below with reference to embodiments and accompanying drawings. However, the scope of protection of the present invention is not limited to the embodiments described below. Numerous specific details are set forth in the following description to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without one or more of these details.

[0018] Example 1: See Figure 1-2 The purpose of this embodiment is to provide a method for deploying complex cloud-edge collaborative services based on multi-agent reinforcement learning. It includes the following steps: S1. Construct a dual-agent collaborative deployment framework, including an environment module, an agent module, and an evaluation module. The agent module includes node agents and link agents, and the node agents include cloud agents and edge agents. S2. The environment module collects resource status data of cloud-edge nodes and topology features of physical links in real time, and generates preprocessed features of node agents and link agents. S3. Input the preprocessed features into the node agent network, train the output target node index, and set the output result as the action of the node agent. S4. After the node agent completes its action, the link agent obtains the processed real-time topology features from the physical link between the source node and the target node based on the target node index, inputs the topology features into the link agent network, and outputs the optimal path selection and frequency slot block combination. S5. The evaluation module calculates the collaborative score of the node agent and the link agent through the value function and feeds it back to the policy network of the node agent and the link agent. S6. The node agent and the link agent update the network parameters according to the real-time reward signal, so that the node agent and the link agent can achieve adaptive learning through network iteration. In this embodiment, the policy network includes a node agent network and a link agent network.

[0019] Intelligent agent module, including Individual cloud intelligent agent, Each entity comprises one edge agent and one link agent, and its state space consists of the cloud-edge node resource state and the link topology state. The agent module consists of a tuple. It means that, among them For environmental state space, For the action space of cloud intelligent agents, For the first The action space of a cloud-based intelligent agent For the number of cloud intelligent agents, similarly, define... For the action space of the edge agent, For the first The action space of each edge agent This represents the action space of the link-based intelligent agent.

[0020] The method also includes a joint decision-making module, a link resource allocation module, a dynamic adjustment module, a reward and strategy optimization module, and a Nash strategy generation module.

[0021] The joint decision-making module is used to define the joint action space. ,in For the action space of cloud intelligent agents, For the action space of the edge agent. This defines the action space of the link agent; and uses a joint state transition function. It determines the execution of joint operations. In the case of state Transition to the next state The probability distribution is used to calculate the state transition probability.

[0022] The link resource allocation module is configured to allocate the action space of the link agent. Mapping to physical path-slot matrix Column space; actions of linked agents This represents the probability of selecting the w-th slot of the k-th path. With matrix The column vectors are positively correlated. In this application, the decision-making of the link agent network is based on a probability distribution. The output is the action space for the link agent, where s is the current state and a is the possible action. The corresponding probability distribution is: The link agent selects a defined physical path and frequency slot as its action. Action Space Large size and matrix The columns are consistent. The actions of the link agent. Is the first choice The first in the path Each frequency slot. For a specific action. probability A higher probability value than other actions increases the likelihood that an action will be selected.

[0023] Therefore, the decision-making process of the link agent in this application is as follows: State input: Based on the current network state s, calculate the probability distribution of all possible path + spectrum slot combinations; Action space: All (k, w) combinations constitute the action space. Size equal to matrix The number of columns; select actions, sample according to probability distribution, select the (k,w) combination with the highest probability as the action of the link agent, the higher the probability of the combination being selected.

[0024] The dynamic adjustment module, i.e., the size of the state space and action space of the link agent, dynamically adjusts the size of the state space and action space based on the number of paths k and the number of frequency slots w, and when the action... probability Actions with higher priority than others are prioritized for selection of the corresponding frequency slot. The state space and action space dimensions of the link agent are based on parameters and... The settings, depending on the parameter values, result in corresponding changes in the state space and action space. When , At that time, the size of the state action space is... ,like Figure 2 The left part of the network is shown. When... , At that time, the size of the state action space is... ,like Figure 2 The network shown on the right.

[0025] In the above state transition process, the reward function for each agent can be expressed as: .

[0026] The reward and strategy optimization module is configured to define reward functions for cloud agents, edge agents, and link agents respectively. , , .

[0027] No. Reward function for a cloud agent: ; No. Reward function for each side agent: ; Link agent reward function: ; in, The total number of subtasks in the business, and the number of tasks deployed in the business. The number of subtasks in each cloud domain is Deployed in The number of subtasks in each region is When cloud-edge services are successfully allocated, the cloud and edge agents aim to reduce service processing latency and maximize node resource availability, while the link agent aims to reduce service transmission latency and maximize available link resources. , For the first The resource consumption of nodes within the domain after each subtask is deployed in the cloud or edge domain. Indicates the first The cloud intelligent agent and the first Each agent performs an action. and Time-based subtasks are deployed on cloud nodes edge nodes The processing delay. Indicates the link agent's relationship with the first... Sub-task execution actions Network resources available after deployment. Deployment incentives. The reward value is [value] if the deployment is successful. Failure .in These represent the deployment rewards for subtasks in the cloud data center, edge data center, and network link, respectively. State transitions are the result of the combined actions of all agents.

[0028] The Nash policy generation module employs a centralized training and distributed execution framework, solving for the joint optimal policy through the Nash equilibrium function. This is to minimize business processing latency and maximize resource availability.

[0029] The workflow of the Nash policy generation module includes: The cloud agent, edge agent, and link agent each update their evaluation functions. , , ; Constructing multi-agent game scenarios … To generate the optimal deployment scheme, the multi-agent equilibrium function is defined as follows: , Representing cloud, edge, and link intelligent agents In state Below, other intelligent agents corresponding to it Adopting a joint strategy The following is obtained using Nash equalization Value; among which , For a game situation to be resolved by a Nash equilibrium, the Nash equilibrium solution must satisfy the following definition: It is a cloud, edge, and link intelligent agent exist In this state, latency is minimized and resource availability is optimized. It is a cloud, edge, and link intelligent agent The set of all possible strategies, The optimal joint strategy for cloud, edge, and link agents under Nash equilibrium.

[0030] Specifically, each cloud, edge, and link intelligent agent The function depends on the joint action The calculation method is as follows, the first Evaluation function for a cloud agent: ; No. Evaluation function for each agent: ; Evaluation function of the link agent: ; in , Definition and similar.

[0031] The strategy for each agent is: Together they constitute a joint strategy. Multi-agent cloud-edge collaborative deployment is a hybrid task, involving cloud, edge, and link-based intelligent agents. A centralized training and distributed execution approach is adopted. A joint strategy is employed. Update itself in the case of The function observes other intelligent agents simultaneously. of Values, in the same state This creates a game-theoretic situation. … The goal is to coordinate the states and actions of cloud, edge, and link intelligent agents to generate the optimal deployment plan.

[0032] The cloud-edge collaborative service deployment training process based on multi-agent reinforcement learning includes the following five steps. Through steps 1 to 5, the training process aims to balance node computing resources and link bandwidth resources among the cloud, edge, and link agents in each learning cycle. Through collaboration among the agents, the optimization goal of maximizing network resources while meeting latency requirements is achieved, enabling flexible and efficient deployment of complex cloud-edge collaborative services. Step 1: The cloud agent, edge agent, and link agent collect features of cloud-edge nodes and links from the deployment environment to construct a cloud-edge node feature matrix. Link feature vector , and serve as input data for each intelligent agent; Step 2: The cloud agent and the edge agent perform linear transformation calculations on the input cloud and edge node features through a policy network. Where W is the weight matrix and B is the bias term, the output is the probability distribution of the cloud and edge nodes to be deployed. And select the nodes to be deployed from the probability distribution according to the random greedy strategy, as the decision action of the agent; Step 3: The link agent performs joint computation on the input node actions and link features through the policy network to generate... Probability distribution of the shortest path intermediate frequency slot block Based on the probability distribution, the frequency slot blocks to be selected on the output link are finally formed, thus forming a complete node-link action sequence. Step 4: After the cloud agent, edge agent, and link agent each execute their selected actions, they obtain the corresponding cloud and edge node rewards from the environment module. , Link rewards ; Step 5: The evaluation module analyzes the state-action relationship through the policy network and constructs evaluation functions based on cloud node rewards, edge node rewards, and link rewards respectively. and The rationality of the actions of cloud agents, edge agents, and link agents is quantitatively evaluated; based on the evaluation results, the agents are driven to perform feedback learning, iteratively optimize the next action selection, and continuously improve the deployment plan.

[0033] The training mechanism for the cloud-edge collaborative service deployment training process includes: Two self-learning training phases are set up, one for training the node agent network and the other for training the link agent network. In the node agent training phase, the loss function of the node agent network is calculated by outputting action probabilities and combining local reward signals and global evaluation network values. The local reward signals reflect the real-time performance of the node agent in cloud-edge collaborative service deployment, and the global evaluation network values ​​represent its comprehensive performance throughout the entire deployment process. In the training phase of the link agent, based on the determined source node and destination node, and combined with the routing and slot selection tasks, the link agent network is used to calculate the loss function of the link agent network through the local reward signal and the output of the link value network. The Floyd shortest path algorithm is used to determine candidate paths; The reward mechanism of the link agent is constructed based on short-term and long-term reward signals, and the discounted cumulative total reward is used as the training objective. An ϵ-greedy strategy is adopted to balance exploration and exploitation, thereby improving the strategy optimization capability in complex coupled environments; By continuously updating the parameters of nodes and link agents through a feedback learning mechanism, the efficient deployment of complex cloud-edge collaborative tasks can be achieved.

[0034] During the link training phase, after successful service deployment, an immediate reward of 1 is set, and link data access is triggered to train routing and frequency slot selection.

[0035] Specifically, this application sets up two self-learning stages to enable three agents to coordinate deployment. The first stage involves training the node agents in the node agent network to learn, and the second stage involves training the link agents in the link agent network to learn. For the first stage, this application considers improving the learning ability of the node agents from two important aspects. One aspect is focusing on the entire process of cloud-edge collaborative service deployment. Therefore, after the node agent network outputs probabilities, it will... and Take this into account and calculate the loss of the nodal agent network. : , It is a local reward signal for the actions of node intelligent agents in cloud-edge collaborative business deployment. It is a global value that evaluates the network, meaning the performance of node agents in the complete cloud-edge collaborative business deployment process.

[0036] Another aspect is assessing the long-term deployment of cloud-edge collaboration services. This involves focusing on both short-term and long-term deployment incentives for cloud-edge collaboration services. For immediate reward signals, The total discounted cumulative reward for a batch processing round: , After successful service deployment, the immediate reward for the node agent is set to 1, and link data is accessed for a second training iteration. This training performs route selection and frequency slot selection on the identified source and destination nodes. The Floyd shortest path algorithm is then used to determine the preceding... Candidate paths.

[0037] ; ; Then, according to the above formula, through and To calculate the loss of the link agent network . It is a local reward signal that the link agent estimates its actions during the business deployment process. It is the output of the value network of the link agent, used to evaluate its performance during the deployment of a complete virtual network function service chain.

[0038] In the training phase, short-term and long-term business deployment reward signals are also used to determine whether the actions of the link agent meet expectations, which is similar to the training mechanism of the node agent. , Setting the loss of the evaluation network In each training iteration after a state transition, the parameters can be updated. .

[0039] Due to the complexity of the environment and the coupling with complex power cloud-edge collaborative business data, node agents and link agents cannot simply choose the action with the highest probability in the decision-making process. This means that the output may be biased, and better solutions may exist. Therefore, this application adopts an ϵ-greedy strategy to balance exploring the correct solution and utilizing the current environment. Then, node agents and link agents can take actual actions based on ϵ-greedy to transition from the current state to the next state. Through feedback learning during the training process, the optimal collaborative strategy can be discovered, enabling efficient deployment of complex cloud-edge collaborative tasks.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Any other modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention, as long as they do not depart from the spirit and scope of the technical solutions of the present invention, should be covered within the scope of the claims of the present invention.

Claims

1. A method for deploying complex cloud-edge collaborative services based on multi-agent reinforcement learning, characterized in that, Includes the following steps: S1. Construct a dual-agent collaborative deployment framework, including an environment module, an agent module, and an evaluation module. The agent module includes node agents and link agents, and the node agents include cloud agents and edge agents. S2. The environment module collects resource status data of cloud-edge nodes and topology features of physical links in real time, and generates preprocessed features of node agents and link agents. S3. Input the preprocessed features into the node agent network, train it, output the target node index, and set the output result as the action of the node agent. S4. After the node agent completes its action, the link agent obtains the processed real-time topology features from the physical link between the source node and the target node based on the target node index, inputs the topology features into the link agent network, and outputs the optimal path selection and frequency slot block combination. S5. The evaluation module calculates the collaborative score of the node agent and the link agent through the value function and feeds it back to the policy network of the node agent and the link agent. S6. The node agent and link agent update the network parameters based on the real-time reward signal, so that the node agent and link agent can achieve adaptive learning through network iteration.

2. The method for deploying complex cloud-edge collaborative services based on multi-agent reinforcement learning according to claim 1, characterized in that, The intelligent agent module includes Individual cloud intelligent agent, The method comprises one edge agent and one link agent, whose state space consists of the cloud-edge node resource state and the link topology state; the method further includes: The joint decision-making module is used to define the joint action space. ,in For the action space of cloud intelligent agents, For the action space of the edge agent. This defines the action space of the link agent; and uses a joint state transition function. This indicates the execution of joint actions. The following is a state Transferred to The probability distribution is used to calculate the state transition probability; The link resource allocation module is configured to allocate the action space of the link agent. Mapping to physical path-slot matrix Column space; actions of linked agents This represents the probability of selecting the w-th slot of the k-th path. With matrix The column vectors are positively correlated; The dynamic adjustment module dynamically adjusts the size of the state space and action space of the link agent based on the number of paths k and the number of frequency slots w, and when the action... probability If the action is higher than other actions, the corresponding frequency slot will be selected first; The reward and strategy optimization module is configured to define reward functions for cloud agents, edge agents, and link agents respectively. , , ; The Nash policy generation module employs a centralized training and distributed execution framework, solving for the joint optimal policy through the Nash equilibrium function. This is to minimize business processing latency and maximize resource availability.

3. The method for deploying complex cloud-edge collaborative services based on multi-agent reinforcement learning according to claim 2, characterized in that, The workflow of the Nash policy generation module includes: The cloud agent, edge agent, and link agent each update their evaluation functions. , , ; Construct a multi-agent game situation based on the evaluation function. … To generate the optimal deployment scheme, the multi-agent equilibrium function is defined as follows: ， It represents cloud, edge, and link intelligent agents. In state Below, other intelligent agents corresponding to it Adopting a joint strategy The following is obtained using Nash equalization Value; among which , For a game situation to be resolved by a Nash equilibrium, the Nash equilibrium solution must satisfy the following definition: It is a cloud, edge, and link intelligent agent exist In this state, latency is minimized and resource availability is optimized. It is a cloud, edge, and link intelligent agent The set of all possible strategies, The optimal joint strategy for cloud, edge, and link agents under Nash equilibrium.

4. The method for deploying complex cloud-edge collaborative services based on multi-agent reinforcement learning according to claim 3, characterized in that, The cloud-edge collaborative service deployment training process based on multi-agent reinforcement learning includes the following five steps. Through steps 1 to 5, the training process aims to balance node computing resources and link bandwidth resources among the cloud, edge, and link agents in each learning cycle. Through collaboration among the agents, the optimization goal of maximizing network resources while meeting latency requirements is achieved, enabling flexible and efficient deployment of complex cloud-edge collaborative services. Step 1: The cloud agent, edge agent, and link agent collect features of cloud-edge nodes and links from the deployment environment to construct a cloud-edge node feature matrix. Link feature vector , and serve as input data for each intelligent agent; Step 2: The cloud agent and the edge agent perform linear transformation calculations on the input cloud and edge node features through a policy network. Where W is the weight matrix and B is the bias term, the output is the probability distribution of the cloud and edge nodes to be deployed. And select the nodes to be deployed from the probability distribution according to the random greedy strategy, as the decision action of the agent; Step 3: The link agent performs joint computation on the input node actions and link features through the policy network to generate... Probability distribution of the shortest path intermediate frequency slot block Based on the probability distribution, the frequency slot blocks to be selected on the output link are finally formed, thus forming a complete node-link action sequence. Step 4: After the cloud agent, edge agent, and link agent each execute their selected actions, they obtain the corresponding cloud and edge node rewards from the environment module. , Link rewards ; Step 5: The evaluation module analyzes the state-action relationship through the policy network and constructs evaluation functions based on cloud node rewards, edge node rewards, and link rewards respectively. and To quantitatively evaluate the rationality of the actions of cloud agents, edge agents, and link agents; Based on the evaluation results, the agent performs feedback learning, iteratively optimizes the next action selection, and continuously improves the deployment plan.

5. The method for deploying complex cloud-edge collaborative services based on multi-agent reinforcement learning according to claim 4, characterized in that, The training mechanism for the cloud-edge collaborative service deployment training process includes: Two self-learning training phases are set up, one for training the node agent network and the other for training the link agent network. In the node agent training phase, the loss function of the node agent network is calculated by outputting action probabilities and combining local reward signals and global evaluation network values. The local reward signals reflect the real-time performance of the node agent in cloud-edge collaborative service deployment, and the global evaluation network values ​​represent its comprehensive performance throughout the entire deployment process. In the training phase of the link agent, based on the determined source node and destination node, and combined with the routing and slot selection tasks, the link agent network is used to calculate the loss function of the link agent network through the local reward signal and the output of the link value network. The Floyd shortest path algorithm is used to determine candidate paths; The reward mechanism of the link agent is constructed based on short-term and long-term reward signals, and the discounted cumulative total reward is used as the training objective. An ϵ-greedy strategy is adopted to balance exploration and exploitation, thereby improving the strategy optimization capability in complex coupled environments; By continuously updating the parameters of nodes and link agents through a feedback learning mechanism, the efficient deployment of complex cloud-edge collaborative tasks can be achieved.

6. The method for deploying complex cloud-edge collaborative services based on multi-agent reinforcement learning according to claim 5, characterized in that, During the link training phase, after successful service deployment, an immediate reward of 1 is set, and link data access is triggered to train routing and frequency slot selection.

Images