Many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN

Abstract

The invention discloses a many-to-many communication routing method based on multi-agent graph reinforcement learning in an SDWN, and the method comprises the steps: firstly designing a reinforcement learning algorithm based on Q-learning, optimizing the number of network nodes for deploying agents, and reducing the calculation and deployment cost of the conventional method for deploying the agents for all network nodes; then, a centralized training and distributed execution multi-agent deep reinforcement learning algorithm is designed under an AC framework, and in order to reduce the storage space overhead of model training, a graph convolutional neural network and a graph neural network are respectively used to redesign Actor and Critic network structures; the problem that in an existing method, a convolutional neural network is weak in adaptive capacity to topological structure data is solved. And finally, an action space local observation method is proposed to avoid generation of invalid actions, so that the time overhead of model training is reduced, the convergence speed is increased, and the time overhead of model training is reduced, and the convergence speed is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of software defined wireless networks, and in particular to a many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN. Background Art

[0002] The routing decision problem of many-to-many communication is essentially an NP-hard combinatorial optimization problem in mathematical optimization theory. Many classic routing decision methods for many-to-many communication, such as the MUSTER protocol, MMForests, and heuristic algorithms based on the merging of subtree VNF instances of support vector machines, are implemented by independently generating trees for each source and then merging them into a forest. Although this algorithm based on local information consideration is simple and easy to implement, it ignores the cooperation and competition between trees, and they can often only obtain local optimal solutions.

[0003] By obtaining the global state information of the network, it is possible to solve the many-to-many communication routing decision problem in a centralized manner. However, it is difficult for traditional wireless network management methods to obtain global state information in a timely manner, which poses a challenge to solving the many-to-many communication routing plan that meets the network service quality. The software-defined network (SDN) architecture separates the control plane from the data plane. It can obtain global network state information in real time and improve the ability to perceive network state changes. The software-defined wireless network (SDWN) architecture applies the software-defined network architecture concept to wireless networks. It can design reasonable routing strategies at the control layer, so it can effectively adapt to the needs of many-to-many wireless communication application scenarios.

[0004] Compared with the classic routing decision method of many-to-many communication, reinforcement learning is a data-driven artificial intelligence method that can adapt well to the highly dynamic characteristics of network status information. As an advanced reinforcement learning method, multi-agent reinforcement learning uses the mutual cooperation between distributed agents to better adapt to the dynamic changes of network status. At present, there are gradually some literatures that apply multi-agent reinforcement learning to the research of many-to-many communication problems, and have achieved good results.

[0005] The paper (HE W, SUN Y, YANG M, et al. Multi-goal multi-agent learning for task-oriented dialogue with bidirectional teacher–student learning [J]. Knowledge-Based Systems, 2021, 213 (15): 1-11.) proposed a reinforcement learning method for model-free multi-agent proximal policy optimization to solve the many-to-many communication routing decision problem in a highly dynamic environment. However, this method requires the deployment of agents on all network nodes, which increases the spatial and temporal complexity of solving the routing strategy and the deployment cost as the scale of network nodes increases.

[0006] The paper (ALAM MD Z, KHAN KS, JAMALIPOUR A. Multiagent Best Routing in High-Mobility Digital-Twin-Driven Internet of Vehicles (IoV) [J]. IEEE Internet of Things Journal, 2024, 11 (8): 13708-13721.) proposed a multi-agent reinforcement learning method based on a policy value framework, in which multi-agents use deterministic policy gradient (MADDPG) to establish a dynamic coordination graph (DCG) of the ever-changing communication network topology, and the agents cooperate with each other to solve the optimal routing solution. However, since this method uses the CNN method when constructing the Actor and Critic neural network structure, it is difficult to adapt to the communication network topology data with non-Euclidean structure characteristics, which leads to an increase in storage space overhead during model training; in addition, since the output of the Actor network in multi-agent reinforcement learning is often set to a specified length, it is necessary to design some screening or penalty mechanisms to reduce the invalid actions generated by the multi-agents during the learning process, which will increase the time overhead of model training and reduce the convergence speed of the algorithm. Summary of the invention

[0007] The present invention aims to solve the problems that the existing data-driven multi-agent deep reinforcement learning methods have high computational and deployment costs and are difficult to adapt to network topologies with non-Euclidean structural characteristics. In addition, too many invalid actions in the training process will increase storage space and time overhead and slow convergence speed. A many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN is provided.

[0008] To solve the above problems, the present invention is achieved through the following technical solutions:

[0009] The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN includes the following steps:

[0010] Step 1: Obtain network topology information and many-to-many communication tasks through the SDN architecture;

[0011] Step 2: According to the change rules of the two attributes of the network node packet queue length and the cumulative energy consumption of the network node, the change process of the two attributes during the data transmission process is deduced, and the optimization objectives and related constraints are summarized to obtain the optimization problem;

[0012] Step 3: Initialize the set of intelligent network nodes and the set of common network nodes, use the classic random simple path transmission algorithm and the random shortest path transmission algorithm to imitate the two types of nodes, simulate the data transmission process of many-to-many communication tasks in the topology information, determine the best set of intelligent network nodes through the Q-Learning algorithm, and deploy agents on these best intelligent network nodes;

[0013] Step 4: Initialize the agents on the intelligent network nodes, simulate the data transmission process of many-to-many communication tasks in the topology information, train the best agents through the multi-agent graph reinforcement learning algorithm, and deploy these best agents back to the corresponding intelligent network nodes for real-time decision-making on the routing of data packets to achieve data transmission in many-to-many communication.

[0014] Compared with the prior art, the present invention designs a multi-agent graph reinforcement learning-based many-to-many communication routing method (MAGDS-M2M) for many-to-many communication problems under the SDWN architecture, which has the following characteristics:

[0015] 1. In order to determine the network node locations where the agents are deployed, the present invention designs a Q-learning-based reinforcement learning algorithm to optimize the number of network nodes where the agents are deployed, thereby reducing the computational and deployment costs of deploying agents on all network nodes in the existing method.

[0016] 2. After determining the deployment strategy of the multi-agent, the present invention designs a multi-agent deep reinforcement learning algorithm with centralized training and distributed execution under the AC (Actor-Critic) framework. In order to reduce the storage space overhead of model training, the network structures of Actor and Critic are redesigned using graph convolutional neural network (GCN) and graph neural network (GNN), respectively, which solves the problem that the convolutional neural network (CNN) in the existing method has weak adaptability to topological structure data;

[0017] 3. In order to solve the problem that a large number of invalid actions are generated in the fixed-length action space of the Actor network during the training process, the present invention proposes a local observation method of the action space to avoid the generation of invalid actions, thereby reducing the time cost of model training and accelerating the convergence speed;

[0018] 4. The experimental results show that the present invention reduces the task completion delay by 23.11% compared with the benchmark experiment, and verifies that the task completion delay and the standard deviation of the accumulated energy consumption of each network node can be balanced by adjusting the parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is the architecture diagram of the SDWN many-to-many communication routing system of the present invention.

[0020] Figure 2 It is a data flow diagram among multiple agents.

[0021] Figure 3 Design graphs for Actor and Critic networks.

[0022] Figure 4 For the network topology.

[0023] Figure 5 Select a training reward curve for the SmartNet node.

[0024] Figure 6 Training reward curves for multi-agent reinforcement learning.

[0025] Figure 7 The change in the time it takes to complete a task for each model iteration.

[0026] Figure 8 : This is a comparison of the data packet transfer between the existing method SPR-M2M and the method of the present invention MAGDS-M2M, where (a) and (b) are the transmission conditions of the data packets during the application of the two methods, and (c)-(e) are the changes in the length of the data packet queues in the network nodes 4, 5, and 7 of the two methods over time. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific examples and with reference to the accompanying drawings.

[0028] The many-to-many communication routing decision problem in SDWN is expressed as follows: Assume that the SDWN network is modeled as an undirected graph in represents a finite set of network nodes, and ε represents a finite set of links. represents the network node numbered i in the network, represents the network node numbered j in the network, e ij ∈ε represents the network node s i With network nodes j There are links between,i≠j,Fig. The adjacency matrix A = [a ij ]∈{0,1} n×n Indicates the connection mode of network nodes in the network. Represents the number of network nodes, that is, a ij satisfy

[0029]

[0030] In SDWN, the source network node set sending data is represented as use represents the number of source network nodes, and the set of target network nodes receiving data is represented as use Indicates the number of target network nodes, and uses the traffic matrix to represent the many-to-many communication task To indicate that represents a set of natural numbers, element m ij Represents the network node s i Sent to network nodes j The number of packets can be expressed as m p =1 T M1 represents the number of data packets contained in task M, where 1 is a column vector of all 1s of the corresponding dimension.

[0031] In each time period T s Generate a many-to-many communication task flow matrix M within a period of time. The source network node s Data transmission is performed within a period of T s Otherwise, the untransmitted data packets will be discarded. s The number of data packets in the many-to-many communication task is limited, and there is an upper bound κ, that is, satisfying m p <κ. When the data packet passes through the network node buffer area, the first-in-first-out (FIFO) mechanism is adopted. The network node buffer area has the same size and is κ. p The assumption of <κ shows that data packets will not be lost due to network node buffer overflow. Assume that a network node sends a data packet once per unit time t s Completed within, t represents the time period T s The time of the tth data packet sent in the sequence is t+1, and the interval between them is one unit time t. s In addition, assuming that the network node s iIn unit time t s The maximum number of packets sent within a period is c i .

[0032] After the data packet arrives at the target network node, it will be considered to have completed the transmission and will no longer be counted in the packet queue length. When the many-to-many communication task M is completed, the queue length of each network node in the network can be considered to be 0. If ζ is used to represent each cycle T s In the time from the start of data transmission to the completion of the task, in order to ensure that the data packet is not discarded due to timeout, there will be ζ<T s It is established. It can be seen that the length of the packet queue of the network node is a very important network status indicator. Assuming that the vector Indicates that in period T s The length of the packet queue of each network node at the tth time, where p i (t) represents the network node s i The length of the packet queue at time t. Represents network nodes i In period T s The tth time received from another network node s k Number of packets, using Represents network nodes i In period T s The tth time t is sent to the network node s j The number of packets, Representation diagram Network nodes i The set of all neighboring network nodes, then the period T s The t-th time network node s i Number of packets received and period T s The t-th time network node s i The number of packets sent at time t They are:

[0033]

[0034] Network Nodes i The number of packets processed at time t Should meet:

[0035]

[0036] Among them, the first formula represents the network node s i There are packets at time t, and the number of packets sent does not exceed c i ; The second formula represents the network node s iNo packets are forwarded. Therefore, there is a i The number of packets sent meets the following constraints:

[0037]

[0038] Obviously, for non-target network nodes s i The packet queue length is as follows:

[0039]

[0040] For the target network node s i For example, the data arriving at time t consists of two parts, one of which is not in s. i The number of packets to the target network node The other part is s i The number of packets to the target network node Then we have:

[0041]

[0042] Since the packets arriving at the destination will no longer be counted in the packet queue length, the destination network node s i The packet queue length in satisfies:

[0043]

[0044] In particular, for the initial time t = 0:

[0045]

[0046] The first part indicates that the number of packets at the initial time of the source network node is equal to the sum of the number of packets that need to be transmitted from it to all target network nodes, and the second part indicates that there are no packets that need to be transmitted at the initial time on the non-source network node. Indicates the length of the packet queue at the initial time on the network node.

[0047] The time ζ taken to complete the many-to-many communication task M can be expressed as:

[0048] ζ=inf{t>0,p(t)=0,p(0)=p0} (10)

[0049] Wherein, 0 is a column vector of all zeros, and inf{t>0,p(t)=0,p(0)=p0} represents the time taken for a data packet to be transmitted from the beginning to the length of the data packet queue in the network node being 0.

[0050] In addition, network node energy consumption is another important indicator to describe network performance, which is usually expressed by the balance of the accumulated energy consumption of network nodes. Indicates that in period T s The cumulative energy consumption of each network node at the tth time in Indicates that network node i is in period T s The accumulated energy consumption at the tth time is is the set of real numbers, represents a set of non-negative real numbers. The network node energy consumption discussed in this invention is mainly used for sending and receiving data packets. Assume that the network node s i The energy consumption of sending and receiving a data packet is expressed as and Then the network node s i The accumulated energy consumption at time t can be expressed as

[0051]

[0052] From the above formulas (2)-(11), it can be seen that for a given many-to-many task M, the length of the packet queue on each network node in the network is only related to the variable and However, due to the fact that from network node s in time t i Send to j The number of packets is equal to the number of network nodes s j Received from network node s i The number of packets sent, i.e. Therefore, the routing decision problem of many-to-many communication can be regarded as just a variable For the convenience of expression, we combine formula (7) and formula (8) and let

[0053]

[0054] Among them, std(e c (ζ)) represents the vector e c The standard deviation of (ζ).

[0055] Based on the above description, the following optimization problem model can be established for the many-to-many communication routing problem discussed in the present invention: while ensuring that the tasks corresponding to the traffic matrix M are completed as quickly as possible, the accumulated energy on each network node is exhausted as evenly as possible.

[0056] The goal of completing the task corresponding to the traffic matrix M as quickly as possible can be expressed as follows:

[0057]

[0058] The goal of balancing the possible cumulative energy exhaustion at each network node can be described as follows:

[0059]

[0060] Therefore, the present invention obtains the following optimization problem for the many-to-many communication routing decision problem:

[0061]

[0062] The optimization problem is a bi-objective discrete optimization problem. The decision variables are Through linear weighting, it is transformed into the following single-objective optimization problem model:

[0063]

[0064] Among them, β is the objective function and The weighting coefficient of β can be adjusted to meet the needs of different objective functions.

[0065] For the many-to-many communication routing decision optimization problem established above, in order to make the solved many-to-many communication routing plan well adapt to the highly dynamic changes of network status, the present invention first designs a system architecture for obtaining global state information based on software-defined network technology, and then designs an intelligent solution method based on multi-agent graph reinforcement learning.

[0066] The SDN control plane and data plane designed by the present invention perform collaborative perception and intelligent decision-making in a new framework, such as Figure 1 As shown, it consists of a data plane, a control plane, and a knowledge plane.

[0067] The data plane forwards data packets in the network according to the communication tasks defined by the traffic matrix, and is mainly composed of wireless access nodes (APs) and stations (STAs). The APs in the entire network form a wireless multi-hop network. AP nodes are network nodes. It is necessary to consider whether to deploy intelligent agents on these nodes to facilitate the planning of routing decisions for many-to-many communications. Those without intelligent agents are called ordinary AP nodes, and those without intelligent agents are called intelligent AP nodes. Ordinary AP nodes obtain the best path for data packet transmission based on the existing shortest path algorithm, and intelligent AP nodes obtain the best path for data packet transmission through intelligent agents. Each AP node periodically interacts with the control plane and transmits information such as the length of the local data packet queue to the control plane.

[0068] The control plane consists of four parts: global network information perception module, local network information perception module, model deployment module and parameter update module. Figure 1In the figure, ①② indicates that the global and local network information perception modules are responsible for collecting and organizing network status information. The difference is that the global network information perception module needs to collect the length of the data packet queue of each AP node during the first and second stage training, and incrementally calculate the cumulative energy consumption of each AP node, and then convert the collected data into graph structure data for intelligent agent recognition; while the local information perception module is responsible for collecting local network status information during the second stage training and after the training, and also needs to convert the data into graph structure data for intelligent agent recognition. Figure 1 The model deployment modules in ③④ convert the model deployment strategy obtained from the first stage of training into deployment instructions that can be recognized by AP nodes. Figure 1 The parameter update module in ⑤⑥ organizes the model parameters generated by the second stage training and converts them into information sent to the smart AP node.

[0069] The knowledge plane is an important part of the system architecture. The routing decision algorithm of the two-stage intelligent method for many-to-many communication in this invention runs on this plane. This plane is responsible for the main work of the two-stage model training: in the first stage, there is no need for the experience pool to cache data. Figure 1 The agent in the seventh stage directly trains and updates the agent based on the global network status information transmitted by the control plane through the northbound interface, and then transmits the agent deployment strategy generated by the training to the data plane through the control plane. The second stage includes an experience pool. The control plane organizes the network status information into the experience pool in the form required by the agent. Figure 1 In the process, multiple intelligent agents sample small batches from the experience pool for training and updating, and then transmit the model parameters generated by the training to the data plane through the parameter update module of the control plane. The intelligent agents in the first stage need to interact with the data plane multiple times before determining the deployment of the intelligent AP nodes. Each time the data plane interacts, it needs to simulate and complete a communication task corresponding to the traffic matrix. The multiple intelligent agents in the second stage need to start training after the intelligent agents in the first stage complete the training and deploy the intelligent agents, and they also need to interact with the data plane multiple times. The difference is that with the experience pool, the intelligent agents can be updated while collecting data, and the model of the intelligent AP node on the data plane can be updated after completing a communication task corresponding to the traffic matrix. In addition, Figure 1 After the second stage model training is completed, the data plane intelligent AP model can directly make decisions by obtaining local variables without using global information.

[0070] Considering the decision variables of the many-to-many routing decision problem in the optimization problem It has the characteristics of high dimension, discreteness and time-varying, and is an NP-hard optimization problem. The traditional solution algorithm is difficult to complete the solution of this problem in polynomial time, and it is difficult to adapt to its highly dynamically changing network state. This invention proposes a two-stage multi-agent routing method based on Q-learning reinforcement learning and multi-agent graph reinforcement learning for many-to-many communication routing problems.

[0071] Phase 1: Intelligent network node deployment algorithm.

[0072] If intelligent agents are deployed on all network nodes, the spatial and temporal complexity of solving routing strategies and the deployment cost will increase as the scale of network nodes increases. Therefore, the present invention intends to select some network nodes for intelligent deployment, which requires the design of a network node selection scheme for intelligent agent deployment. In view of the fact that the network node selection method of the traditional graph theory method is difficult to adapt to the highly dynamic characteristics of the network state, the present invention designs an intelligent network node deployment method based on Q-learning reinforcement learning. The design process is described in detail as follows:

[0073] First, consider that all network nodes are divided into two types: intelligent network nodes and ordinary network nodes.

[0074]

[0075] in, Represents a collection of intelligent network nodes, using Indicates the number of intelligent network nodes, Represents a set of common network nodes, using Indicates the number of common network nodes. It will use agents to determine the next network node to transmit the packet to.

[0076] In order to find a suitable intelligent network node, the present invention sets the intelligent network node As a state, all possible sets of intelligent network nodes constitute the state space in Represents a collection of intelligent network nodes, using Indicates the number of all intelligent network nodes. Because network nodes are either intelligent network nodes or ordinary network nodes, n sno =2 n .

[0077] For a set of intelligent network nodes, a step action can be to delete a network node or to add a network node, wherein deleting a network node is to classify the selected network node into a set of ordinary network nodes; adding a network node is to add the selected network node to the set of intelligent network nodes. The present invention uses a 2n-dimensional one-hot vector space l∈{1,2,…,2n}, and satisfy

[0078]

[0079] but The size is

[0080] It should be noted that in the determined state Next, if l≤n and Indicates that s l Join the intelligent network node set; when l>n, record l n =mod(l,n), where mod(l,n) represents the operation of l modulo n. If Indicates that Delete from the smart network node set. This representation is not the most concise, but it can avoid the problem that the physical meaning of the action changes in different states.

[0081] In the many-to-many communication task M, Represents a set of data packets, the number of which satisfies Assume that Indicates that in the network node s i The set of packets processed on Represents network node s at time t i The jth data packet processed, according to the network node's ability to process data packets at a time, we know that j≤c i .

[0082] Define the mapping of data packets to their destination network nodes des: There are two types of data packet transmission algorithms here. One is the random simple path transmission algorithm, which randomly transmits the data packet to the next network node according to all simple paths from the data packet to the destination network node; the other is the random shortest path transmission algorithm, which randomly transmits the data packet to the next network node according to all shortest paths from the data packet to the destination network node. Use fph: represents the mapping of data packets transmitted to the next set of network nodes according to the random simple path transmission algorithm, where is the network node s iA subset of neighbor network nodes, representing the set of next network nodes to which the data packet can be delivered; similarly, using sph: Represents the mapping of the data packet transmission to the next set of network nodes according to the random shortest path transmission algorithm. In addition, in order to discuss the generation of loops, the mapping pre is used: It represents the mapping of the data packet to the previous network node it passed through. Obviously, the previous network node that the data packet passed through is also the neighboring network node of the current network node.

[0083] The purpose of selecting intelligent network nodes is to help these network nodes make better decisions. It is necessary to define the time t intelligent network node s i The jth packet processed The reward for processing a data packet includes the following two parts:

[0084]

[0085] in, and It means removing the predecessor network node from the possible next network node of the data packet to avoid the generation of loops. They represent the intelligent network node s at time t through the random simple path transmission algorithm and the random shortest path transmission algorithm. i Processed packets The reward is further weighted to obtain the intelligent network node s at time t i Processed packets Rewards

[0086]

[0087] Among them, β no Represents the weight coefficients of the two transmission algorithms.

[0088] The reward function R obtained after determining the set of intelligent network nodes no It can be expressed as

[0089]

[0090] That is, the sum of the rewards generated by all important network nodes before the task is completed. It can be seen that the reward function Reward Space

[0091] use and R no (k) represents the state, action and reward value of the kth step respectively. It is easy to define the state of k+1 steps That is, in the collection Add or delete a network node. Based on this, the present invention uses the Q-learning algorithm to solve the optimal action, using Indicates that the i-th iteration is in the tuple The action value function on it, its iterative update process is

[0092]

[0093] in, Indicates in status Next, for all possible Take the maximum value. 0≤α no <1 is the learning rate, 0<γ no <1 is a discount factor used to reduce the impact of future estimates.

[0094] Based on the above analysis, an intelligent network node deployment algorithm is designed as shown in Algorithm 1.

[0095]

[0096]

[0097] According to the division in formula (17), the present invention will be in the intelligent network node set Deploy intelligent agents on the network nodes in the network and use them to make decisions. The classic random shortest path transmission algorithm is used on the network nodes. The agents deployed on all intelligent network nodes constitute a multi-agent structure.

[0098] Phase 2: Multi-agent Graph Reinforcement Learning Algorithm

[0099] After the first stage of agent training, the deployment of the agent is completed on the corresponding network nodes according to the obtained intelligent network node deployment strategy. However, to determine the routing decision plan for many-to-many communication, it is also necessary to train these deployed agents. Taking into account the highly dynamic characteristics of network status in many-to-many communication, the present invention designs a multi-agent graph reinforcement learning algorithm based on the AC framework for routing decisions for many-to-many communication. In order to reduce the cost of model training, the algorithm uses GCN and GNN to design the network structure of Actor and Critic respectively; in addition, in order to reduce the time cost of model training, the algorithm designs a local observation method in the process of Actor network generating actions to avoid the generation of invalid actions, thereby improving the convergence speed, which ensures the feasibility of the algorithm. The design process is described in detail as follows:

[0100] In the optimization problem, network node i must be in period T s Time t to network node sj The number of packets transmitted depends on the network nodes s i How to process each data packet during this period of time. In order to help network nodes optimize processing solutions, intelligent agents are deployed on intelligent network nodes. represents the set of agents, where I i Indicates the agent numbered i. Use cur: Indicates the mapping of the agent to the network node number, with ι i =cur(I i ) represents agent I i The network node number. The following introduces the multi-agent model from the aspects of state space, action space and reward function.

[0101] (1) State space

[0102] Assume that the packet queue length p(t) and the accumulated energy used on the network node are c (t) in period T s They do not change from time t to t+1 and are used by all agents. They are called global variables. represents the global state space, where

[0103] S(t)=[p(t),e c (t)](25)

[0104] Since the order in which packets arrive at the agent is random, only the first packet needs to be considered during training. The same model can be used to make decisions for subsequent data. For the convenience of derivation, it will be referred to as data packet in the following text.

[0105] Use and Indicates data packet The location code of the previous network node, the network node where the current data packet is located, and the target network node corresponding to the data. and Respectively meet:

[0106]

[0107] because and The three variables are data packets They are unique, so they are called data packets Local variables.

[0108] Use o i : Represents Agent I iLocal observation mapping of the global variable S(t). Through local mapping, the state space of all agents can be expressed as in Represents Agent I i The state space, S i (t) represents agent I i Processing data packets The state when it is

[0109]

[0110] Among them, i (p(t)) and o i (e c (t)) respectively represent agent I i The local observation of two global variables, the number of packets and the accumulated energy consumption, is actually completed by the network nodes through the graph convolutional neural network.

[0111] (2) Action Space

[0112] The action space of the agent is represented as in Represents Agent I i The action space, and in Represents Agent I i Network node l i In addition to the neighboring network nodes, the agent may decide to cache the data packet in the local network node, so it is necessary to add 1 to the number of neighboring network nodes. In other words, is a one-hot encoded vector representing agent I i Packet Pass it to the corresponding neighbor network node or cache it in the current network node to obtain the state space The size is

[0113] (3) Reward Function

[0114] use It represents the global state change of the environment under the action of each agent. On this basis, given the initial state S(0) of the environment, the Actor network can be used to sample the environment and obtain the sample sequence

[0115]

[0116] In the above sequence, the state S(t) only represents the global state. iCorresponding local observations are required. In addition, there are two possibilities for the end time T. One is that all packets in the network have reached the corresponding target network node at time T. The other is that the maximum sampling time T is reached. m , that is, T = T m .

[0117] For sample sequence Fragments, in

[0118]

[0119] Obtained through the environment, similar to formula (9)

[0120] S(t+1)=[p(t+1),e c (t+1)].(32) For the objective function Design Agent I i The single-step reward function is

[0121]

[0122] Among them, sum(·) represents the sum of the corresponding vectors. From this reward, we can see that as the number of data packets in the network gradually decreases, the cost of single-step data processing also gradually decreases.

[0123] For the objective function Design the single-step reward as

[0124]

[0125] Among them, max(·), min(·), std(·) represent the maximum value, minimum value and standard deviation of the corresponding vector respectively. From this reward, we can see that the greater the difference in the accumulated energy consumption of each network node, the greater the cost of single-step data processing.

[0126] Reward function R i (t) In addition to including and In addition, in order to avoid loops, the reward function R i (t) also includes the loop penalty C h , its specific expression is as follows:

[0127]

[0128] Among them, β is the weight, C h is the set loop penalty, which is a constant less than 0, and I(·) is the indicative function, that is,

[0129]

[0130] During the MAGDS-M2M model training process, the sequence Add the reward function value and store it in the experience pool In the pool where experience is stored The sample is

[0131] (4) Design of the Actor and Critic network structure and the local observation method in the action space

[0132] Since the reinforcement learning algorithm is designed to be implemented under the AC framework, the reasonable design of the Actor and Critic network structure is indispensable to effectively solve the routing decision problem of many-to-many communication through cooperation and competition among agents. In addition, in order to avoid invalid actions, a local observation method is designed for the action space.

[0133] First, in order to adapt to the network topology data, the present invention adopts graph convolutional neural network as Actor network. i , which can be expressed as

[0134]

[0135] Among them, θ i Represents Agent I i The Actor network can learn parameters. In particular, for the data packet Can be Get Agent I i The processing action for it is specifically expressed as

[0136]

[0137] in, is the output of the graph neural network. is a local observation mapping of the action space, using Represents Agent I i The local observation matrix, where k represents the network node ι i The neighbor network nodes are The order number that appears in the corresponding The position in Indicates that satisfy

[0138]

[0139] The local mapping of the action space here can be expressed as

[0140]

[0141] Among them, HiT Denotes the matrix H i The transpose of . For Gumbel-Softmax sampling mapping, the mapping method designed by the present invention is oa i (·) and Gumbel-Softmax sampling map gbs i (·) all satisfy the property that gradients can be back-propagated, and their chain gradients are all 1. In addition, the local observation method in the action space here avoids the generation of invalid actions and solves the convergence problem of fixed-length actions.

[0142] Compared with the Actor network, the Critic network needs to fit a centralized action value function. The graph convolution has limited ability to fit this function, so in addition to the graph convolution layer, there is also a fully connected layer. i ,use represents its local state space, where

[0143]

[0144] in, represents the inverse mapping of the local observation in the action space, which is calculated as follows:

[0145]

[0146] According to the mapping oa i The definition of (·), It also satisfies the property that the gradient can be returned, and the chain gradient is also 1. Represents Agent I i Data Packet The local observation state of Agent I i The Critic network can be expressed as

[0147]

[0148] Among them, φ i It is Agent I i Critic network parameters, The purpose of is to learn the value of the centralized action-value function, so its output is a real number. i Processing packets hour

[0149]

[0150] For Agent I i have

[0151]

[0152] in, It is a softmax mapping, which is different from the Gumbel-Softmax sampling mapping. It does not consider the gradient to be returned. It is Agent I i The centralized action-value function.

[0153] In order to solve the problem of overestimation, the design of the target network is introduced. i The Actor target network is represented as:

[0154]

[0155] in, It is Agent I i The parameters of the Actor target network, which are different from the parameters of the Actor network only in value but consistent in structure.

[0156] Similarly, Agent I i The critic target network is expressed as

[0157]

[0158] in, It is Agent I i The parameters of the Critic target network are different from the Critic network only in parameters.

[0159] For data packets Similar to The calculation is

[0160]

[0161] Agent I i The centralized action-value function represented by the target network is

[0162]

[0163] Based on the above design, the multi-agent training paradigm of CTDE is used between agents. The specific data flow of collaboration and competition between multiple agents is as follows: Figure 2 The specific design of the Actor and Critic network structure is as follows Figure 3 As shown (only one agent is shown). ①~④ are the Actor network parts, which is a graph convolutional neural network as a whole. ① contains the global state and local state. The dotted border represents the global state, which is the packet queue length p(t) on the network node and the accumulated energy consumption e on the network node. c(t). ② is the first convolution layer of the Actor network. ③ is the second convolution layer of the Actor network. It should be noted that these convolution layers do not need to directly obtain the global state in the intelligent data plane, because each network node only uses limited information about local network nodes, which can be completed by obtaining information about neighboring network nodes. ④ shows that the result after convolution is a graph. ⑤~⑨ realize the local observation of the action space. First, ⑤ expand the graph in the order of network nodes. ⑥ By mapping oa i (·) local observation of neighboring network nodes. ⑦ By mapping gbs i (·) Get the corresponding action code. ⑧ Through mapping Map the action codes to all network nodes. Then restore the graph structure by nodes. It should be noted that if the action in ⑧ points to the network node where the data packet was previously located, a penalty C will be given when calculating the reward. h The ellipsis at ⑩ means that the data interaction with other agents is omitted. In addition to the global state, other agents will provide their local observations and map the sfm k (·) The action obtained, so the number of input channels of Critic is 4n im +2, which contains 2 global state channels, and each agent provides 3 local state channels and 1 action channel. Represents the implementation of the Critic network, which is a graph neural network as a whole. is a convolutional layer. The features generated by the convolutional layer are summed with the original features in a 1:1 ratio to obtain new features. Sum the features on the channel to facilitate the design of the linear layer. It is a linear layer, and finally obtains a value to fit the centralized action value function value.

[0164] (5) Design of parameter update process for actor and critic networks

[0165] Under the network structure design in the previous section, the parameter update process of the Actor and Critic networks will change accordingly, so the parameter update process needs to be explained here. Small batch b s Sampling, using Indicates that the batch size is b s Samples of Represents the distribution of sample state S, which is used in the original network and target network respectively. and Represents the representation of S to action function in the sample, and the action calculation process in formula (44) and formula (48) is represented as a i =μi (S) and a′ i =μ′ i (S). In this representation, the batch size is b s The centralized action value function of the sample is expressed as The centralized action value function of the batch sample under the target network representation is expressed as Agent I i The reward is denoted as R i (S).

[0166] use Represents Agent I i The loss function of the Critic network under batch samples can be expressed as follows according to the method of temporal difference:

[0167]

[0168] Among them, 0<γ<1 is the attenuation factor, which reflects the impact of the subsequent centralized action value function on the current one.

[0169] About φ i The gradient update is

[0170]

[0171] in, yes Representation under batch samples.

[0172] use Represents Agent I i The loss function of the Actor network under batch samples uses a similar regularization method. The design is as follows:

[0173]

[0174] Among them, λ||a i || 2 Item is action a i Regularization, λ>0 is the regularization coefficient.

[0175] According to the mapping oa i , gbs i Keep the gradient transferable and the chain gradient to 1, and you can calculate it according to the chain rule Its about θ i The gradient of

[0176]

[0177] in, yes In the representation of batch samples,

[0178]

[0179] To this end, the Critic network parameter φ i and Actor network parameters θ i The update expression is

[0180]

[0181] in, and They are φ i and θ i The learning rate is a hyperparameter, and λ is also a hyperparameter. Under their influence, the update process can ignore the constant.

[0182] Use φ i (k),θ i (k) and Respectively represent the parameters of the original network and the target network Critic and Actor network iteration k steps, the present invention uses a soft update method to iterate them

[0183]

[0184] Among them, 0<τ<1 is a positive number, k u represents the time of soft update, and mod represents the remainder operation. Their network initialization is the same when k = 0, that is,

[0185]

[0186] Through the above derivation and analysis process, it can be seen that the Actor and Critic parameter optimization process of the algorithm designed by the present invention meets the requirements of the measure gradient theorem and can guarantee period convergence like the Actor-Critic algorithm. The present invention will also verify the convergence of the designed algorithm through experiments and their results. The implementation process of the multi-agent graph reinforcement learning algorithm is shown in Algorithm 2.

[0187]

[0188] Aiming at the specific characteristics of many-to-many communication routing problems, the multi-agent deep reinforcement learning method with centralized training and distributed execution designed in the AC framework adopts GCN and GNN to design the Actor and Critic network structures respectively, which can effectively reduce the storage space overhead of model training, and designs a local observation method in the process of generating actions in the Actor network to avoid the generation of invalid actions, reduce the time overhead of model training and accelerate the convergence speed.

[0189] According to the above analysis, the present invention proposes a many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN, which includes the following steps:

[0190] Step 1: Obtain network topology information G through the SDN architecture, and obtain many-to-many communication tasks M based on historical data.

[0191] Step 2: According to the change rules of the two attributes of the network node packet queue length and the cumulative energy consumption of the network node, the change process of the two attributes during data transmission is deduced, and the optimization objectives and related constraints are summarized to obtain the optimization problem:

[0192]

[0193] Step 3: Initialize the intelligent network node set and the common network node set, and use the classic random simple path transmission algorithm and random shortest path transmission algorithm to simulate the many-to-many communication task M in the topology information. In the data transmission process, the best set of intelligent network nodes is determined through the Q-Learning algorithm, and the intelligent agents are deployed on these best intelligent network nodes.

[0194] Initialize the intelligent network node set to an empty set, simulate the many-to-many communication task M in the topology information In the transmission process, the network nodes are evaluated and decided to add or reduce the network nodes in the intelligent network node set, and then the many-to-many communication task M is simulated in the topology information. The transmission process is repeated and evaluated again, until the required number of intelligent network nodes is reached, which is the optimal network node set. In the specific implementation process, it is also necessary to design a reasonable reward function that is conducive to the generation of the best network nodes and adapts to the performance of the deployed intelligent agent in step 4. The specific implementation steps are as follows:

[0195] Step 3.1, define all intelligent network node sets as state space, divide intelligent network nodes into ordinary network node sets or select ordinary network nodes as all possible definitions of intelligent network node sets as action space; and initialize the Q-table according to the definitions of state space and action space, and initialize the state at the same time.

[0196] Step 3.2: Use the classic random simple path transmission algorithm and random shortest path transmission algorithm to simulate the many-to-many communication task M in the topology information During the data transmission process, the reward is calculated and the action is obtained through the Q-table.

[0197] Step 3.3: Feed the action back to the environment to obtain the next state and update the Q-table.

[0198] Step 3.4: Repeat steps 3.2 and 3.3 until the number of network nodes in the intelligent network node set is equal to the required number of intelligent network nodes, and return the optimal Q-table.

[0199] Step 3.5: After the training is completed, the set of intelligent network nodes is obtained according to the optimal Q-table, and the intelligent agents are deployed on the intelligent network nodes.

[0200] Step 4: Initialize the agents on the intelligent network nodes and simulate the many-to-many communication task M in the topology information. In the data transmission process, the best agents are trained through the multi-agent graph reinforcement learning algorithm, and these best agents are deployed back to the corresponding intelligent network nodes to determine the routing of data packets in real time, so as to realize data transmission of many-to-many communication.

[0201] Initialize the agents on the intelligent network nodes and simulate the many-to-many communication task M in the topology information In the data transmission process, the data in the transmission process is stored in the data cache. When the data in the cache reaches a certain amount, data is sampled from it to train the intelligent agent. After training each intelligent agent, the new intelligent agent intelligent network node is used to simulate the many-to-many communication task M in the topology information. During the data transmission process, the data in the transmission process is stored in the data buffer area, and then sampled and trained again, and this cycle continues until the training process converges. After the training is completed, the optimal agent parameters generated during the training process are deployed, and data transmission for many-to-many communication can be achieved. The specific implementation steps are as follows:

[0202] Step 4.1, experience pool sampling: Each agent samples from the experience pool;

[0203] Step 4.2, Actor network: Each agent extracts the queue length of all network node data packets, the accumulated energy consumption of all network nodes, the position code of the network node where the current agent is located, and the previous network node position code and the target network node position code of the data packet to be decided on the network node where the current agent is located from the sampled samples, and converts them into graph structure data and inputs them into the Actor network of the current agent based on the graph convolutional neural network (GCN), and extracts the features of the next network node through the Actor network;

[0204] Step 4.3, local observation of action space: Each agent first limits the feature selection of the next-hop network node to the range of neighboring network nodes through local mapping, then maps it to the best next-hop network node through the Gumbel-Softmax method, and finally obtains the position encoding of the best next-hop network node through local inverse mapping of the action space;

[0205] Step 4.4, Critic network: Each agent extracts the length of the data packet queue of all network nodes, the accumulated energy consumption of all network nodes, the position codes of the network nodes where all agents are located, the position codes of the previous network nodes where the decision-making data packets on the network nodes where all agents are located, and the position codes of the target network nodes from the sampled samples, and obtains the position codes of all the best next-hop network nodes through interaction with other agents, and inputs them into the Critic network based on the graph neural network (GNN) of the current agent, and obtains rewards through the Critic network;

[0206] Step 4.5, network parameter update: The current agent updates the network parameters of its Critic network through gradient backpropagation based on the reward, and then updates the network parameters of its Actor network through feedback;

[0207] Step 4.6: Loop through steps 4.1 to 4.5 until convergence (network parameters of the Actor network and Critic network do not change within a predetermined number of times, or a preset number of iterations is reached), thereby obtaining the best Actor network and Critic network for all agents, and deploying the Actor parameters of the agents based on the best Actor network and Critic network back to the corresponding intelligent network nodes for real-time decision-making on data packet routing, so as to achieve data transmission for many-to-many communication.

[0208] The following experimental analysis will be conducted from three parts: experimental environment, parameter settings and experimental results.

[0209] (1) Experimental environment

[0210] The network topology used in the experiment is as follows: Figure 4 As shown, its network nodes are divided into three types: data entry network nodes are source network nodes, relay network nodes are responsible for data packet transmission, and data center network nodes are target network nodes. It should be noted that the shortest path algorithm here only considers the number of path hops and ignores factors such as distance, but it can be seen from the process of deriving the above design scheme that it is not limited to this.

[0211] (2) Parameter settings

[0212] For ease of description, we use Represents the sequence representation of a set, that is, the order in which elements are added based on the set. In this embodiment, the network node sequence is set to The source network node sequence is The target network node sequence is The corresponding task matrix is ​​set as according to Definition of m ij Represents the source sequence The sequence of the i-th network node sent to the destination network node The number of packets of the jth network node, such as m 11 =40 means that the number of packets sent from the source network node 14 to the destination network node 10 is 40. Through fine-tuning (i.e., meta-learning) to adapt to different M. In order to avoid the triviality of the problem, different unit time packet processing capabilities are given to network nodes. That is, the data processing capacity of the four network nodes numbered 1, 2, 12, and 13 per unit time is three times that of other network nodes.

[0213] (3) Experimental results

[0214] Intelligent network node selection experiment, this part is responsible for selecting the network nodes that need intelligent agent-assisted decision-making. The many-to-many communication task matrix of the following experimental results is given.

[0215] Figure 5 is the experimental training reward curve for selecting intelligent network nodes using Q-learning. The result is obtained using the parameter α no =0.01,E1=600,n r =5. Figure 5 The middle curve is the reward value of the experiment during the training process, and the star points are the points where exploration occurs. Except for the star points, it can be clearly seen that the curve gradually increases, which illustrates the effectiveness of the algorithm.

[0216] Table 2 Intelligent network node selection results

[0217]

[0218] Table 2 shows the results of intelligent network node selection, where the parameter is set to α no =0.01,E1=600,n r The values ​​are 1 to 10, which are the settings of the number of intelligent network nodes in the table. The selection result and reward value are the set of intelligent network nodes selected after the number of intelligent network nodes is specified and the corresponding reward value.

[0219] As can be seen in Table 2, the selection results when the number of intelligent network nodes is 5 The selection results in the previous four cases are covered, and the reward value from the number of intelligent network nodes from 1 to 5 has been monotonically increasing. It starts to fluctuate after the number of intelligent network nodes reaches 5, and the corresponding reward value when a better result is reached again is 238.5, which is only 0.06 more than 237.9 when the number of intelligent network nodes is 5, but the number of network nodes has increased by two, and the cost of the number of network nodes is relatively large. Therefore, the present invention is Figure 4 In the task matrix M, the number of intelligent network nodes is set to 5, and the selection result is This result will be used to determine the deployment of the agent.

[0220] The routing optimization experiment is realized through multi-agent graph reinforcement learning. By deploying agents on intelligent network nodes, the agents assist in deciding which next hop the network node sends the data packet to. For network nodes on ordinary network nodes, the random shortest path transmission algorithm is directly used to pass the data packet to the next hop.

[0221] In order to illustrate the effectiveness of the design method of the present invention, a benchmark experiment was designed, namely the SPR-M2M algorithm adapted from MMForests. The algorithm uses the Dijkstra algorithm between each source and destination pair to obtain the shortest hop many-to-many communication path algorithm. The experimental results show that the delay of the algorithm to complete task M is 225 slots, and the standard deviation of the accumulated energy consumption on each network node when completing the task is 740.757.

[0222] Figure 6 is the reward result during the multi-agent graph reinforcement learning training process. The learning rate of the agent Actor network is The learning rate of the agent Critic network is Batch update parameter b s =8, number of cycles E2 = 1500, loop penalty C h =1. The rewards in the figure are all less than 0, and their absolute values ​​represent the total cost of the agent processing data under task M, where one generation represents the cost of completing task M once. As can be seen from the figure, as the number of iterations increases, the reward trend of each agent gradually increases and tends to be stable, which shows the effectiveness and convergence of the algorithm. In addition, it is noted that the reward functions of agents 1, 2, 4, and 5 have decreased after about 600 generations, while the reward of agent 3 has increased, indicating that the agents have coordinated with each other.

[0223] Figure 7 It is the performance curve of the time it takes for the model generated during the training process to complete the task under case 1. The curve represents the time delay for the model to complete the task every 5 generations during the training process, and the straight line is the time delay 225slot for SPR-M2M to complete the task. It can be seen from the figure that as the training continues, the time it takes for MAGDS-M2M to complete the task becomes smaller and smaller, and gradually exceeds SPR-M2M, which illustrates the effectiveness of the method proposed in the present invention.

[0224] exist Figure 7It can be seen that the model achieves the best effect at the 605th generation, and the time to complete the task is only 173 slots, which is 23.11% lower than SPR-M2M. The 605th generation model is recorded as MAGDS-M2M-v605. The following further compares the differences between MAGDS-M2M-v605 and SPR-M2M algorithms from the changes in the packet paths of the completed tasks and the number of packets at key network nodes.

[0225] from Figure 8 As can be seen from the figure, compared with SPR-M2M, MAGDS-M2M-v605 makes better use of the network and takes into account all paths in the network. Figure 8 As can be seen from (a) and (b), the network nodes with more frequent data packet interactions are network nodes 4, 5, and 7. Figure 8 (c)-(e) show that in SPR-M2M, network node 4 is in a congested state for a long time, that is, the packet queue length is high for a long time, network node 5 is idle for part of the time, and network node 7 is always idle, which leads to a great waste of resources. In contrast, MAGDS-M2M-v605 effectively solves this problem. Figure 8 (c) shows that the maximum length of the packet queue in network node 4 has changed from about 100 to about 60, which significantly reduces the congestion. Figure 8 (d) shows that the maximum length of the packet queue in network node 5 changes from about 70 to about 40. Figure 8 (e) shows that network node 7 changes from a completely idle state to a maximum packet queue length of about 80, but there is still idle time, which shows that network node 7 is not yet a key node that hinders the speed of task completion. In addition, MAGDS-M2M-v605 significantly shortens the time to complete the task, which shows that MAGDS-M2M-v605 makes full use of node 7 to effectively share the burden of network nodes 4 and 5, thereby improving the speed of task completion.

[0226] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention, and therefore the present invention is not limited to the above specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN, characterized by: The steps include: Step 1: Obtain network topology information and many-to-many communication tasks through the SDN architecture; Step 2: According to the change rules of the two attributes of the network node packet queue length and the cumulative energy consumption of the network node, the change process of the two attributes during the data transmission process is deduced, and the optimization objectives and related constraints are summarized to obtain the optimization problem; Step 3: Initialize the set of intelligent network nodes and the set of common network nodes, use the classic random simple path transmission algorithm and random shortest path transmission algorithm to simulate the data transmission process of many-to-many communication tasks in topology information, determine the best set of intelligent network nodes through the Q-Learning algorithm, and deploy agents on these best intelligent network nodes; Step 4: Initialize the agents on the intelligent network nodes, simulate the data transmission process of many-to-many communication tasks in the topology information, train the best agents through the multi-agent graph reinforcement learning algorithm, and deploy these best agents back to the corresponding intelligent network nodes for real-time decision-making on the routing of data packets to achieve data transmission in many-to-many communication.

2. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 1, characterized in that: In step 1, the traffic matrix M = [m i′j′ ] to represent; where m i′j′ Represents the source network node s i′ Sent to destination network node s j′ The number of packets, i′=1,2,…,n s , n s Indicates the number of source network nodes, j′=1,2,…,n d , n d Indicates the number of target network nodes.

3. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 1, characterized in that: In step 2, the optimization problem is: in, is the optimization objective, β represents the weighted coefficient of the two objectives, Indicates the goal of completing the task as quickly as possible. Indicates the goal of exhausting possible equilibriums at each network node; Represents network nodes i Sent to network node s at time t j The number of packets; c i Represents network nodes i The threshold value of the number of packets sent in one time step; e ij Represents network nodes i With network nodes j The links between i represents the network node numbered i, s j Represents the network node numbered j, i≠j; represents a set of natural numbers, ε represents a set of links, Represents network nodes i The set of neighbor network nodes, Represents a collection of network nodes.

4. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 1, characterized in that: The specific process of step 3 is as follows: Step 3.1, define all intelligent network node sets as state space, and define all possible ways of dividing intelligent network nodes into common network node sets or adding common network nodes to intelligent network node sets as action space; and initialize the Q-table according to the definitions of state space and action space, and initialize the state; Step 3.2, use the classic random simple path transmission algorithm and random shortest path transmission algorithm to simulate the data transmission process of many-to-many communication tasks in topology information, calculate the reward and obtain the action through the Q-table; Step 3.3: Feed the action back to the environment to obtain the next state and update the Q-table; Step 3.4, repeating steps 3.2 and 3.3 until the number of network nodes in the intelligent network node set is equal to the required number of intelligent network nodes, and returning the optimal Q-table; Step 3.5: After the training is completed, the best set of intelligent network nodes is obtained according to the best Q-table, and the intelligent agents are deployed on these best intelligent network nodes.

5. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 4, characterized in that: In step 3.2, the reward R no The calculation formula is: In the formula, Represents the network node s at time t through the classic random simple path transmission algorithm i Processing packets Rewards, Represents the network node s at time t through the random shortest path transmission algorithm i Processing packets Rewards, Represents network node s at time t i The kth packet processed, β no represents the weight coefficient of the two transmission algorithms, ζ represents the number of cycles T s The time from the start of data transmission to the completion of the task, Indicates the currently selected intelligent network node set, c i Represents network nodes i The threshold for the number of packets sent in one time step.

6. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 5, characterized in that: Through the classic random simple path transmission algorithm, the intelligent network node s is i Processing packets Rewards for: Through the random shortest path transmission algorithm, intelligent network node s at time t i Processing packets Rewards for: In the formula, In the classic random simple path transmission algorithm, Remove the predecessor network node from the possible next network node; Indicates that in the random shortest path transmission algorithm, in the data packet Remove the predecessor network node from the possible next network node; Represents network node s at time t i The kth packet processed.

7. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 1, characterized in that: The specific process of step 4 is as follows: Step 4.1, each agent samples from the experience pool; Step 4.2, each agent extracts the length of all network node data packet queues, the accumulated energy consumption of all network nodes, the position code of the network node where the current agent is located, and the previous network node position code and the target network node position code of the data packet to be decided on the network node where the current agent is located from the sampled samples, and converts them into graph structure data and inputs them into the Actor network of the current agent based on the graph convolutional neural network, and extracts the features of the next network node through the Actor network; Step 4.3, each agent first limits the feature selection of the next-hop network node to the range of neighboring network nodes through local mapping, then maps it to the best next-hop network node through the Gumbel-Softmax method, and finally obtains the position encoding of the best next-hop network node through local inverse mapping of the action space; Step 4.4, each agent extracts the length of all network node data packet queues, the accumulated energy consumption of all network nodes, the position codes of all network nodes where the agents are located, the position codes of the previous network nodes where the decision-making data packets on the network nodes where all agents are located, and the position codes of the target network nodes from the sampled samples, and obtains the position codes of all the best next-hop network nodes through interaction with other agents, and inputs them into the current agent's Critic network based on the graph neural network, and obtains rewards through the Critic network; Step 4.5: The current agent updates the network parameters of its Critic network through gradient backpropagation based on the reward, and then updates the network parameters of its Actor network through feedback; Step 4.6: Loop through steps 4.1 to 4.5 until convergence, thereby obtaining the best Actor network and Critic network for all agents, and deploy the Actor parameters of the agents based on the best Actor network and Critic network back to the corresponding intelligent network nodes for real-time routing of data packets to achieve data transmission for many-to-many communication.

8. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 7, characterized in that: In step 4.3, the action space is to pass the data packet to the corresponding neighbor network node or cache the data packet in the current network node.

9. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 7, characterized in that the steps In 4.4, intelligent network nodes s i The reward R of the agent on i (t) is: In the formula, represents the single-step reward for completing the task as quickly as possible, represents the single-step reward for the target of exhausting the possible equilibrium at each network node, C h represents the loop penalty, C h is a constant less than 0; represents the set of intelligent network nodes; β represents the weighted coefficient of the two objectives, is the indicator function, represents the action taken by the current agent, Represents the position code corresponding to the action made by the current agent, Indicates the previous network node position code of the data packet to be decided on the network node where the current intelligent agent is located.

10. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 9, characterized in that: A single-step reward for completing the task as quickly as possible for: The single-step reward for the target of exhausting the possible equilibrium at each network node for: Indicative Function for: Where p(t) represents the length vector of the packet queues on all network nodes at time t, p(t+1) represents the length vector of the packet queues on all network nodes at time t+1, and κ represents the upper bound of the number of packets in the network; c (t) represents the cumulative energy consumption vector of all network nodes at time t, e c (t+1) represents the cumulative energy consumption vector of all network nodes at time t+1; represents the action taken by the current agent, Represents the position code corresponding to the action made by the current agent, Indicates the previous network node position code of the pending decision data packet on the network node where the current agent is located; sum(·) represents the sum of the vector, max(·) represents the maximum value of the vector, min(·) represents the minimum value of the vector, and std(·) represents the standard deviation of the vector.

Images