Many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN

Abstract

The present invention discloses a many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN. First, a Q-learning-based reinforcement learning algorithm is designed to optimize the number of network nodes for deploying agents, thereby reducing the computational and deployment costs of deploying agents on all network nodes in the existing method. Then, a multi-agent deep reinforcement learning algorithm with centralized training and distributed execution is designed under the AC framework. In order to reduce the storage space overhead of model training, the network structures of actors and critics are redesigned using graph convolutional neural networks and graph neural networks, respectively, solving the problem that convolutional neural networks in existing methods have weak adaptability to topological structure data. Finally, a local observation method in the action space is proposed to avoid the generation of invalid actions, thereby reducing the time overhead of model training and accelerating the convergence speed.

Description

Technical Field

[0001] The present invention relates to the technical field of software-defined wireless networks, and in particular to a many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN. Background Art

[0002] The routing decision problem for many-to-many communication is essentially an NP-hard combinatorial optimization problem in mathematical optimization theory. Many classic routing decision methods for many-to-many communication, such as the MUSTER protocol, MMForests, and heuristic algorithms based on support vector machine subtree VNF instance merging, implement this problem by independently generating trees for each source and then merging them into a forest. While this algorithm, which considers local information, is simple and easy to implement, it ignores the collaboration and competition between trees, often resulting in only local optimal solutions.

[0003] Obtaining global network state information enables centralized solution of many-to-many communication routing decisions. However, traditional wireless network management approaches struggle to obtain global state information in a timely manner, posing a challenge to solving many-to-many communication routing plans that meet network quality of service. Software-defined networking (SDN) architectures separate the control and data planes, enabling real-time access to global network state information and improving awareness of network state changes. Software-defined wireless networking (SDWN) architectures apply SDN architecture principles to wireless networks, enabling the design of reasonable routing strategies at the control layer. Therefore, they are effectively adaptable to many-to-many wireless communication scenarios.

[0004] Compared to classic routing decision-making methods for many-to-many communication, reinforcement learning, a data-driven artificial intelligence approach, is well-suited to the highly dynamic nature of network state information. Multi-agent reinforcement learning, an advanced reinforcement learning approach, leverages the collaboration between distributed agents to better adapt to dynamic changes in network state. Currently, research is increasingly applying multi-agent reinforcement learning to many-to-many communication problems, with promising results.

[0005] The paper (HE W, SUN Y, YANG M, et al. Multi-goal multi-agent learning for task-oriented dialogue with bidirectional teacher–student learning [J]. Knowledge-Based Systems, 2021, 213(15): 1-11.) proposes a reinforcement learning method for model-free multi-agent proximal policy optimization to solve the many-to-many communication routing decision problem in a highly dynamic environment. However, this method requires the deployment of agents on all network nodes, which increases the spatial and temporal complexity of solving the routing strategy and the deployment cost as the network node scale increases.

[0006] The paper (ALAM MD Z, KHAN KS, JAMALIPOUR A. Multiagent Best Routing in High-Mobility Digital-Twin-Driven Internet of Vehicles (IoV) [J]. IEEE Internet of Things Journal, 2024, 11(8): 13708-13721.) proposes a multi-agent reinforcement learning method based on a policy-value framework, in which multi-agents use deterministic policy gradients (MADDPG) to establish a dynamic coordination graph (DCG) of a constantly changing communication network topology. The agents coordinate and cooperate with each other to solve the optimal routing solution. However, since this method uses the CNN method to construct the Actor and Critic neural network structure, it is difficult to adapt to the communication network topology data with non-Euclidean structure characteristics, which leads to an increase in storage space overhead during model training. In addition, since the output of the Actor network in multi-agent reinforcement learning is often set to a specified length, it is necessary to design some screening or penalty mechanisms to reduce the invalid actions generated by the multi-agents during the learning process, which will increase the time overhead of model training and reduce the convergence speed of the algorithm. Summary of the Invention

[0007] The present invention aims to solve the problems that existing data-driven multi-agent deep reinforcement learning methods have high computational and deployment costs, are difficult to adapt to network topologies with non-Euclidean structural characteristics, and have excessive invalid actions during training, which increases storage space and time overhead and slows convergence speed. A many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN is provided.

[0008] To solve the above problems, the present invention is achieved through the following technical solutions:

[0009] The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN includes the following steps:

[0010] Step 1: Obtain network topology information and many-to-many communication tasks through the SDN architecture;

[0011] Step 2: Based on the changing patterns of the two attributes, namely, the length of the network node packet queue and the cumulative energy consumption of the network node, the changing process of the two attributes during data transmission is deduced, and the optimization objectives and related constraints are summarized to obtain the optimization problem.

[0012] Step 3: Initialize the set of intelligent network nodes and the set of ordinary network nodes. Use the classic random simple path transmission algorithm and random shortest path transmission algorithm to simulate the two types of nodes, simulate the data transmission process of many-to-many communication tasks in the topology information, determine the optimal set of intelligent network nodes through the Q-Learning algorithm, and deploy intelligent agents on these optimal intelligent network nodes.

[0013] Step 4: Initialize the agents on the intelligent network nodes, simulate the data transmission process of the many-to-many communication task in the topology information, train the best agents through the multi-agent graph reinforcement learning algorithm, and deploy these best agents back to the corresponding intelligent network nodes to determine the routing of data packets in real time to achieve data transmission in many-to-many communication.

[0014] Compared with the existing technology, this paper designs a multi-agent graph reinforcement learning-based many-to-many communication routing method (MAGDS-M2M) for many-to-many communication problems under the SDWN architecture, which has the following characteristics:

[0015] 1. To determine the network node locations for deploying agents, this paper designs a Q-learning-based reinforcement learning algorithm to optimize the number of network nodes where agents are deployed, thereby reducing the computational and deployment costs of deploying agents on all network nodes compared to existing methods.

[0016] 2. After determining the multi-agent deployment strategy, this paper designs a multi-agent deep reinforcement learning algorithm with centralized training and distributed execution within the AC (Actor-Critic) framework. To reduce the storage space overhead of model training, the network structures of the Actor and Critic are redesigned using a graph convolutional neural network (GCN) and a graph neural network (GNN), respectively. This solves the problem that the convolutional neural network (CNN) in existing methods has a relatively weak adaptability to topological structure data.

[0017] 3. To solve the problem that the fixed-length action space of the Actor network generates a large number of invalid actions during training, this paper proposes a local observation method in the action space to avoid the generation of invalid actions, thereby reducing the time cost of model training and accelerating the convergence speed;

[0018] 4. Experimental results show that the present invention reduces the task completion delay by 23.11% compared with the benchmark experiment, and verifies that the task completion delay and the standard deviation of the cumulative energy consumption of each network node can be balanced by adjusting parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is the architecture diagram of the SDWN many-to-many communication routing system of the present invention.

[0020] Figure 2 It is a data flow diagram among multiple agents.

[0021] Figure 3 Design graphs for Actor and Critic networks.

[0022] Figure 4 For the network topology.

[0023] Figure 5 Select a training reward curve for the Smart Network node.

[0024] Figure 6 Training reward curves for multi-agent reinforcement learning.

[0025] Figure 7 The change in the time it takes to complete a task for model iteration.

[0026] Figure 8 This is a comparison of the data packet transfer between the existing SPR-M2M method and the MAGDS-M2M method of the present invention, where (a) and (b) are the data packet transmission conditions during the application of the two methods, and (c)-(e) are the changes in the data packet queue lengths in network nodes 4, 5, and 7 over time for the two methods, respectively. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific examples and the accompanying drawings.

[0028] The many-to-many communication routing decision problem in SDWN is expressed as follows: Assume that the SDWN network is modeled as an undirected graph in represents a finite set of network nodes, and ε represents a finite set of links. Represents the network node numbered i in the network, represents the network node numbered j in the network, e ij ∈ε represents the network node s i With network nodes j There are links between,i≠j,Fig. The adjacency matrix A=[a ij ]∈{0,1} n×n Indicates the connection mode of network nodes in the network, Indicates the number of network nodes, where a ij satisfy

[0029]

[0030] In SDWN, the source network node set sending data is represented as use Represents the number of source network nodes, and the target network node set receiving data is represented as use Indicates the number of target network nodes, and uses the traffic matrix to represent the many-to-many communication task To indicate that Represents a set of natural numbers, element m ij Represents the network node s i Sent to network node s j The number of packets, you can use m p =1 T M1 represents the number of data packets contained in task M, where 1 is a column vector of all 1s of the corresponding dimension.

[0031] In each time period T s Generate a many-to-many communication task flow matrix M within this period, and the source network node s Data transmission is performed within the period T s Otherwise, the untransmitted data packets will be discarded. Assume that in period T s The number of data packets in the many-to-many communication task is limited, and there is an upper bound κ, that is, satisfying m p <κ. When data packets pass through the network node buffer area, a first-in-first-out (FIFO) mechanism is adopted. The network node buffer area has the same size and is κ. p Assuming that the data packet will not be lost due to network node buffer overflow, it is assumed that the network node sends a data packet once per unit time t s Completed within, t represents the time period T s The time of the tth data packet sent in the sequence is t+1, and the time of the next data packet sent is t+1. The time interval between them is one unit time t. s In addition, assume that the network node s iIn unit time t s The maximum number of data packets sent within is c i .

[0032] After the data packet arrives at the target network node, it will be considered that the transmission is completed and will no longer be counted in the packet queue length. When the many-to-many communication task M is completed, the queue length of each network node in the network can be considered to be 0. If ζ is used to represent each cycle T s The time from the start of data transmission to the completion of the task, in order to ensure that the data packet will not be discarded due to timeout, there will be ζ<T s It is established. This shows that the packet queue length of the network node is a very important network status indicator. Assuming that the vector Indicates that in period T s The length of the packet queue of each network node at time t, where p i (t) represents the network node s i The length of the packet queue at time t. Represents network node s i In period T s The tth time is received from another network node s k Number of packets, using Represents network node s i In period T s The tth time t is sent to the network node s j The number of packets, Representation diagram Network nodes i The set of all neighboring network nodes, then the period T s The t-th time network node s i Number of packets received and period T s The t-th time network node s i The number of packets sent at time t They are:

[0033]

[0034] Network nodes i Number of packets processed at time t Should meet the following requirements:

[0035]

[0036] Among them, the first formula represents the network node s i There are data packets at time t, and the number of data packets sent does not exceed c i ; The second formula represents the network node s iNo packets are forwarded. Therefore, there is a i The number of packets sent must meet the following constraints:

[0037]

[0038] Obviously, for non-target network node s i The packet queue length is as follows:

[0039]

[0040] For the target network node s i For example, the data arriving at time t includes two parts, one of which is not in s i The number of packets to the target network node The other part is s i The number of packets to the target network node Then we have:

[0041]

[0042] Since the data packets arriving at the destination will no longer be counted in the packet queue length, the destination network node s i The packet queue length in satisfies:

[0043]

[0044] In particular, for the initial time t = 0:

[0045]

[0046] The first part indicates that the number of packets at the initial time of the source network node is equal to the sum of the number of packets that need to be transmitted from it to all target network nodes, and the second part indicates that there are no packets that need to be transmitted at the initial time on non-source network nodes. Indicates the length of the packet queue at the initial time on the network node.

[0047] The time ζ required to complete the many-to-many communication task M can be expressed as:

[0048] ζ=inf{t>0,p(t)=0,p(0)=p0} (10)

[0049] Wherein, 0 is a column vector of all zeros, and inf{t>0,p(t)=0,p(0)=p0} represents the time taken for the data packet to be transmitted from the beginning to the length of the data packet queue in the network node being 0.

[0050] In addition, network node energy consumption is another important indicator to describe network performance, which is usually expressed by the balance of network node cumulative energy consumption. Indicates that in period T s The cumulative energy consumption of each network node at the tth time, where Indicates that network node i is in period T s The cumulative energy consumption at time t is: is the set of real numbers, Represents a set of non-negative real numbers. The network node energy consumption discussed in this invention is mainly used for sending and receiving data packets. Assume that the network node s i The energy consumption of sending and receiving a data packet is expressed as and Then the network node s i The accumulated energy consumption at time t can be expressed as

[0051]

[0052] From the above formulas (2)-(11), it can be seen that given a many-to-many task M, the length of the packet queue on each network node in the network is only related to the variable and However, due to the fact that from the network node s in time t i Send to s j The number of packets is equal to the number of network nodes s j Received from network node s i The number of packets sent, i.e. Therefore, the routing decision problem of many-to-many communication can be regarded as just a variable For the convenience of expression, combining formula (7) and formula (8), let

[0053]

[0054] Among them, std(e c (ζ)) represents the vector e c The standard deviation of (ζ).

[0055] Based on the above description, the following optimization problem model can be established for the many-to-many communication routing problem discussed in the present invention: while ensuring that the tasks corresponding to the traffic matrix M are completed as quickly as possible, the accumulated energy on each network node is exhausted as evenly as possible.

[0056] The goal of completing the task corresponding to the traffic matrix M as quickly as possible can be expressed as follows:

[0057]

[0058] The goal of balancing the cumulative energy consumption at each network node can be described as follows:

[0059]

[0060] This leads to the following optimization problem for the many-to-many communication routing decision problem established by the present invention:

[0061]

[0062] The optimization problem is a dual-objective discrete optimization problem, and the decision variables are Through linear weighting, it is converted into the following single-objective optimization problem model:

[0063]

[0064] Where β is the objective function and The weighted coefficient of β is adjusted to meet the needs of different objective functions.

[0065] For the many-to-many communication routing decision optimization problem established above, in order to make the solved many-to-many communication routing plan well adaptable to the highly dynamic characteristics of network status, the present invention first designs a system architecture based on software-defined network technology to obtain global status information, and then designs an intelligent solution method based on multi-agent graph reinforcement learning.

[0066] The new framework designed by the present invention for SDN control plane and data plane to carry out collaborative perception and intelligent decision-making, such as Figure 1 As shown, it consists of the data plane, control plane and knowledge plane.

[0067] The data plane forwards data packets within the network according to the communication tasks defined by the traffic matrix. It is primarily composed of wireless access nodes (APs) and stations (STAs). The APs in the entire network form a wireless multi-hop network. AP nodes are network nodes, and it is necessary to consider whether to deploy intelligent agents on these nodes to facilitate routing decisions for many-to-many communications. Those without intelligent agents are called ordinary AP nodes, while those with intelligent agents are called intelligent AP nodes. Ordinary AP nodes use existing shortest path algorithms to determine the optimal path for data packet transmission, while intelligent AP nodes use intelligent agents to determine the optimal path for data packet transmission. Each AP node periodically interacts with the control plane, transmitting information such as the length of the local data packet queue to the control plane.

[0068] The control plane consists of four parts: global network information perception module, local network information perception module, model deployment module and parameter update module. Figure 1In the figure, ① and ② indicate that the global and local network information perception modules are responsible for collecting and organizing network status information. The difference is that the global network information perception module needs to collect the length of the data packet queue of each AP node during the first and second stage training, and incrementally calculate the cumulative energy consumption of each AP node, and then convert the collected data into graph structure data for intelligent agent recognition; while the local information perception module is responsible for collecting local network status information during the second stage training and after the training, and also needs to convert the data into graph structure data for intelligent agent recognition. Figure 1 The model deployment modules ③ and ④ convert the model deployment strategy obtained from the first stage of training into deployment instructions that can be recognized by AP nodes. Figure 1 The parameter update modules ⑤ and ⑥ organize the model parameters generated by the second stage training and convert them into information sent to the smart AP node.

[0069] The knowledge plane is an important part of the system architecture. The two-stage intelligent routing decision algorithm for many-to-many communication in this invention runs on this plane. This plane is responsible for the main work of the two-stage model training: in the first stage, there is no need to cache data in the experience pool. Figure 1 The agent in the middle (7) trains and updates the agent directly based on the global network status information transmitted by the control plane through the northbound interface, and then transmits the agent deployment strategy generated by the training to the data plane through the control plane. The second stage includes an experience pool. The control plane organizes the network status information into the experience pool in the form required by the agent. Figure 1 In the process, multiple intelligent agents sample small batches from the experience pool for training and updating, and then transmit the model parameters generated by the training to the data plane through the parameter update module of the control plane. The intelligent agents in the first stage need to interact with the data plane multiple times before determining the deployment of the intelligent AP nodes. Each time the interaction occurs, the data plane needs to simulate and complete a communication task corresponding to the traffic matrix. The multiple intelligent agents in the second stage need to start training after the intelligent agents in the first stage complete the training and deploy the intelligent agents, and they also need to interact with the data plane multiple times. The difference is that with the experience pool, the intelligent agents can be updated while collecting data, and the model of the intelligent AP node on the data plane can be updated after completing a communication task corresponding to the traffic matrix. In addition, Figure 1 After the second stage of model training is completed, the data plane intelligent AP model can directly make decisions by obtaining local variables without having to use global information.

[0070] Considering the decision variables of the many-to-many routing decision problem in the optimization problem This problem is high-dimensional, discrete, and time-varying, making it an NP-hard optimization problem. Traditional algorithms struggle to solve it in polynomial time and are difficult to adapt to highly dynamic network conditions. This paper addresses the many-to-many communication routing problem by proposing a two-stage multi-agent routing solution based on Q-learning reinforcement learning and multi-agent graph reinforcement learning.

[0071] Phase 1: Intelligent network node deployment algorithm.

[0072] If intelligent agents are deployed on all network nodes, the spatial and temporal complexity of solving routing strategies and the deployment cost will increase as the network node scale increases. Therefore, the present invention intends to select some network nodes for intelligent deployment, which requires the design of a network node selection scheme for intelligent agent deployment. Given that traditional graph theory methods for network node selection are difficult to adapt to the highly dynamic characteristics of network status, the present invention designs an intelligent network node deployment method based on Q-learning reinforcement learning. The design process is described in detail as follows:

[0073] First, consider that all network nodes are divided into two types: intelligent network nodes and ordinary network nodes.

[0074]

[0075] in, Represents a collection of intelligent network nodes, using Indicates the number of intelligent network nodes, Represents a set of common network nodes, using Indicates the number of common network nodes. If the network node It will use an agent to determine the next network node to transmit the data packet to.

[0076] In order to find a suitable intelligent network node, the present invention sets the intelligent network node As a state, the set of all possible intelligent network nodes constitutes the state space in Represents a collection of intelligent network nodes, using Indicates the number of all intelligent network nodes. Because network nodes are either intelligent network nodes or ordinary network nodes, n sno =2 n .

[0077] For a set of intelligent network nodes, one step can be to delete a network node or to add a network node. Deleting a network node means dividing the selected network node into a set of common network nodes; adding a network node means adding the selected network node to the set of intelligent network nodes. l∈{1,2,…,2n}, and satisfy

[0078]

[0079] but The size is

[0080] It should be noted that in the determined state Next, if l≤n and Indicates that s l Join the intelligent network node set; when l>n, record l n = mod(l,n), where mod(l,n) represents the operation of l modulo n. If Indicates that Delete from the smart network node set. This representation is not the most concise, but it can avoid the problem of the physical meaning of the action changing in different states.

[0081] Under the many-to-many communication task M, use Represents a set of data packets, the number of which satisfies Assume that Indicates that in the network node s i The set of packets processed on Represents network node s at time t i The jth data packet processed, according to the network node's ability to process data packets at a time, we know that j≤c i .

[0082] Define the mapping of data packets to their destination network nodes des: There are two types of data packet transmission algorithms: one is the random simple path transmission algorithm, which randomly transmits the data packet to the next network node according to all simple paths from the data packet to the destination network node; the other is the random shortest path transmission algorithm, which randomly transmits the data packet to the next network node according to all shortest paths from the data packet to the destination network node. Use fph: represents the mapping of data packets to the next set of network nodes according to the random simple path transmission algorithm, where is the network node s iA subset of neighboring network nodes represents the set of next network nodes to which the data packet can be delivered; similarly, use sph: Represents the mapping of the data packet to the next set of network nodes according to the random shortest path transmission algorithm. In addition, in order to discuss the generation of loops, the mapping pre is used: It represents the mapping of the data packet to the previous network node it passed through. Obviously, the previous network node that the data packet passed through is also the neighboring network node of the current network node.

[0083] The purpose of selecting intelligent network nodes is to help these network nodes make better decisions. It is necessary to define the time t intelligent network node s i The jth packet processed The reward for processing data packets includes the following two parts:

[0084]

[0085] in, and This means removing the predecessor network node from the possible next network node of the data packet to avoid loops. They represent the intelligent network node s at time t through the random simple path transmission algorithm and the random shortest path transmission algorithm. i Processed packets The reward is further weighted to obtain the time t smart network node s i Processed packets Rewards

[0086]

[0087] Among them, β no Represents the weight coefficients of the two transmission algorithms.

[0088] The reward function R obtained after determining the set of intelligent network nodes no It can be expressed as

[0089]

[0090] That is, the sum of the rewards generated by all important network nodes before the task is completed. It can be seen that the reward function Reward Space

[0091] use and R no (k) represents the state, action and reward value of the kth step respectively. It is easy to get the state of k+1 steps by definition That is, in the collection Add or delete a network node. Based on this, the present invention uses the Q-learning algorithm to solve the optimal action, using Indicates that the i-th iteration is in the tuple The action value function on it, its iterative update process is

[0092]

[0093] in, Indicates that the status Next, for all possible Take the maximum value. 0≤α no <1 is the learning rate, 0<γ no <1 is a discount factor used to reduce the impact of future estimates.

[0094] Based on the above analysis, the intelligent network node deployment algorithm is designed as shown in Algorithm 1.

[0095]

[0096]

[0097] According to the division in formula (17), the present invention will be in the intelligent network node set Deploy the agent on the network nodes in the network and use the agent to make decisions. The classic random shortest path transmission algorithm is used on the network nodes. The agents deployed on all intelligent network nodes form a multi-agent structure.

[0098] Phase 2: Multi-agent Graph Reinforcement Learning Algorithm

[0099] After the first stage of agent training, the agent is deployed on the corresponding network node according to the obtained intelligent network node deployment strategy. However, to determine the routing decision plan for many-to-many communication, it is also necessary to train these deployed agents. Taking into account the highly dynamic characteristics of network status in many-to-many communication, the present invention designs a multi-agent graph reinforcement learning algorithm based on the AC framework for routing decisions in many-to-many communication. In order to reduce the cost of model training, the algorithm uses GCN and GNN to design the network structure of Actor and Critic respectively; in addition, in order to reduce the time cost of model training, the algorithm designs a local observation method in the process of Actor network generating actions to avoid the generation of invalid actions, thereby improving the convergence speed, which ensures the feasibility of the algorithm. The design process is described in detail as follows:

[0100] In the optimization problem, network node i must be in period T s Time t to network node sj The number of packets transmitted depends on the network nodes s i How to process each data packet during this period. In order to help network nodes optimize the processing solution, intelligent agents are deployed on intelligent network nodes. Represents a set of agents, where I i Indicates the agent numbered i. Use cur: Indicates the mapping from the agent to the network node number, with ι i =cur(I i ) represents agent I i The network node number. The following describes the multi-agent model from the aspects of state space, action space, and reward function.

[0101] (1) State space

[0102] Assume that the packet queue length p(t) and the accumulated energy e on the network node are c (t) in period T s The variables do not change from time t to t+1 and are used by all agents. They are called global variables. represents the global state space, where

[0103] S(t)=[p(t),e c (t)](25)

[0104] Since the order in which packets arrive at the agent is random, only the first packet needs to be considered during training. The same model can be used to make decisions for subsequent data.

[0105] Use and Indicates data packet The location code of the previous network node, the network node where the current data packet is located, and the target network node corresponding to the data. and Satisfy respectively:

[0106]

[0107] because and The three variables are data packets They are unique, so they are called data packets local variables.

[0108] Use o i : Represents agent I iLocal observation mapping of global variables S(t). Through local mapping, the state space of all agents can be expressed as in Represents agent I i The state space, S i (t) represents agent I i Processing data packets The state when , it can be expressed as

[0109]

[0110] Among them, i (p(t)) and o i (e c (t)) respectively represent agent I i The local observation of two global variables, the number of packets and the cumulative energy consumption, is actually completed by the network nodes through the graph convolutional neural network.

[0111] (2) Action Space

[0112] The action space of the agent is represented as in Represents agent I i The action space, and in Represents agent I i Network node l i In addition to the neighboring network nodes, the agent may decide to cache the data packet in the local network node, so it is necessary to add 1 to the number of neighboring network nodes. In other words, is a one-hot encoded vector representing agent I i The data packet Pass it to the corresponding neighbor network node or cache it in the current network node to obtain the state space The size is

[0113] (3) Reward Function

[0114] use It represents the global state change of the environment under the action of each agent. On this basis, given the initial state S(0) of the environment, the Actor network can be used to sample the environment and obtain the sample sequence

[0115]

[0116] The state S(t) in the above sequence only represents the global state. iCorresponding local observations are required. In addition, there are two possibilities for the end time T. One is that all data packets in the network have reached the corresponding target network node at time T. The other is that the maximum sampling time T is reached. m , that is, T = T m .

[0117] For sample sequence Fragments, in

[0118]

[0119] Obtained through the environment, similar to formula (9)

[0120] S(t+1)=[p(t+1),e c (t+1)].(32) For the objective function Design Agent I i The single-step reward function is

[0121]

[0122] Here, sum(·) represents the sum of the corresponding vectors. From this reward, we can see that as the number of data packets in the network gradually decreases, the cost of single-step data processing also gradually decreases.

[0123] For the objective function Design the single-step reward as

[0124]

[0125] Here, max(·), min(·), and std(·) represent the maximum, minimum, and standard deviation of the corresponding vector, respectively. This reward indicates that the greater the difference in cumulative energy consumption across network nodes, the greater the cost of processing data in a single step.

[0126] Reward function R i (t) In addition to containing and In addition, in order to avoid loops, the reward function R i (t) also includes the loop penalty C h , its specific expression is as follows:

[0127]

[0128] Among them, β is the weight, C h is the set loop penalty, which is a constant less than 0, and I(·) is the indicator function, that is,

[0129]

[0130] During the training of the MAGDS-M2M model, the sequence Add the reward function value and store it in the experience pool In the pool where experience is stored The sample is

[0131] (4) Design of the Actor and Critic network structure and the local observation method in the action space

[0132] Since the reinforcement learning algorithm is implemented within the AC framework, effectively solving the routing decision problem in many-to-many communication through cooperation and competition among agents requires a well-designed actor and critic network structure. Furthermore, to avoid invalid actions, a local observation method is designed for the action space.

[0133] First, in order to adapt to the network topology data, the present invention adopts graph convolutional neural network as the Actor network. i , which can be expressed as

[0134]

[0135] Among them, θ i Represents agent I i The Actor network can learn parameters. In particular, for the data packet Can be achieved through Get Agent I i The processing action for it is specifically expressed as

[0136]

[0137] in, is the output of the graph neural network, It is a local observation mapping of the action space, using Represents agent I i The local observation matrix, where k represents the network node ι i The neighbor network nodes are The sequence number that appears in the corresponding The position in Indicates that satisfy

[0138]

[0139] The local mapping of the action space here can be expressed as

[0140]

[0141] Among them, HiT Represents the matrix H i The transpose of . For Gumbel-Softmax sampling mapping, the mapping method designed by the present invention is oa i (·) and Gumbel-Softmax sampling map gbs i (·) all satisfy the property that gradients can be backpropagated, and their chain gradients are all 1. In addition, the local observation method in the action space here avoids the generation of invalid actions and solves the convergence problem of fixed-length actions.

[0142] Compared with the Actor network, the Critic network needs to fit the centralized action value function. The graph convolution has limited ability to fit the function, so in addition to the graph convolution layer, there is also a fully connected layer. i ,use represents its local state space, where

[0143]

[0144] in, Represents the inverse mapping of the local observation in the action space, which is calculated as follows:

[0145]

[0146] According to the mapping oa i The definition of (·), It also satisfies the property that the gradient can be returned, and the chain gradient is also 1. Represents agent I i Data packets The local observation state of Agent I i The critic network can be expressed as

[0147]

[0148] Among them, φ i It is Agent I i Critic network parameters, The purpose of is to learn the value of the centralized action value function, so its output is a real number. i Processing packets hour

[0149]

[0150] For Agent I i have

[0151]

[0152] in, It is a softmax mapping, which is different from the Gumbel-Softmax sampling mapping. It does not consider the gradient to be returned. It is Agent I i The centralized action-value function.

[0153] In order to solve the problem of overestimation, the design of the target network is introduced to transform the agent I i The Actor target network is represented as:

[0154]

[0155] in, It is Agent I i The parameters of the Actor target network are different from those of the Actor network in terms of value but are identical in structure.

[0156] Similarly, agent I i The critic target network is expressed as

[0157]

[0158] in, It is Agent I i The parameters of the critic target network are different from the critic network only in parameters.

[0159] For data packets Similar to The calculation of

[0160]

[0161] Agent I i The centralized action-value function represented by the target network is

[0162]

[0163] Based on the above design, the multi-agent training paradigm of CTDE is used between agents. The specific data flow of collaboration and competition between multiple agents is as follows: Figure 2 The specific design of the Actor and Critic network structure is as follows. Figure 3 As shown (only one agent is shown). ① to ④ are the Actor network parts, which is a graph convolutional neural network as a whole. ① contains the global state and local state. The dotted border represents the global state, which is the packet queue length p(t) on the network node and the accumulated energy consumption e on the network node. c(t). ② is the first convolution layer of the Actor network. ③ is the second convolution layer of the Actor network. It should be noted that these convolution layers do not need to directly obtain the global state in the intelligent data plane, because each network node only uses limited information about local network nodes, which can be completed by obtaining information about neighboring network nodes. ④ shows that the result after convolution is a graph. ⑤~⑨ realize the local observation of the action space. First, ⑤ expand the graph according to the order of network nodes. ⑥ By mapping oa i (·) Realize local observation of neighboring network nodes. ⑦ By mapping gbs i (·) Get the corresponding action code. ⑧ Through mapping Map the action codes to all network nodes. Then, restore the graph structure by node. It should be noted that if the action in ⑧ points to the network node where the data packet was previously located, a penalty C will be given when calculating the reward. h The ellipsis at ⑩ means that the data interaction of other agents is omitted. In addition to the global state, other agents will provide their local observations and map the sfm k (·) The action obtained, so the number of input channels of Critic is 4n im +2, which contains 2 global state channels, and each agent provides 3 local state channels and 1 action channel. Represents the implementation of the Critic network, which is a graph neural network as a whole. is a convolutional layer. The features generated by the convolutional layer are summed with the original features in a 1:1 ratio to obtain new features. Sum the features on the channel to facilitate the design of the linear layer. It is a linear layer, and finally obtains a value to fit the centralized action-value function value.

[0164] (5) Parameter update process design of actor and critic networks

[0165] Under the network structure design in the previous section, the parameter update process of the Actor and Critic networks will change accordingly, so the parameter update process needs to be explained accordingly. Small batch b s Sampling, using Indicates a batch size of b s The sample, using Indicates the distribution of sample state S, which is used in the original network and target network respectively. and Represents the representation of S to action function in the sample, and expresses the action calculation process in formula (44) and formula (48) as a i =μi (S) and a′ i =μ′ i (S). In this representation, the batch size is b s The centralized action value function of the sample is expressed as The centralized action value function of the batch sample under the target network representation is expressed as Agent I i The reward is represented by R i (S).

[0166] use Represents agent I i The loss function of the Critic network under batch samples can be expressed as follows based on the temporal difference method:

[0167]

[0168] Among them, 0<γ<1 is the attenuation factor, which reflects the impact of the subsequent centralized action-value function on the current one.

[0169] About φ i The gradient update is

[0170]

[0171] in, yes Representation under batch samples.

[0172] use Represents agent I i The loss function of the Actor network under batch samples uses a similar regularization method. The design is as follows:

[0173]

[0174] Among them, λ||a i || 2 Item is action a i Regularization, λ>0 is the regularization coefficient.

[0175] According to the mapping oa i , gbs i Keeping the gradient transferable and the chain gradient to 1, we can calculate it according to the chain rule. Its about θ i The gradient of

[0176]

[0177] in, yes In the representation of batch samples,

[0178]

[0179] To this end, the critic network parameter φ i and Actor network parameters θ i The update expression is

[0180]

[0181] in, and They are φ i and θ i The learning rate is a hyperparameter, and λ is also a hyperparameter. Under their influence, the update process can ignore the constant.

[0182] Use φ i (k),θ i (k) and Represent the parameters of the original network and target network Critic and Actor network iteration k steps respectively. The present invention uses a soft update method to iterate them

[0183]

[0184] Among them, 0<τ<1 is a positive number, k u represents the time of soft update, mod represents the remainder operation. Their network initialization is the same when k=0, that is,

[0185]

[0186] The above derivation and analysis demonstrate that the actor and critic parameter optimization process of the algorithm designed in this paper satisfies the requirements of the measure gradient theorem and, like the actor-critic algorithm, guarantees convergence. This paper will further verify the convergence of the designed algorithm through experiments and their results. The implementation of the multi-agent graph reinforcement learning algorithm is shown in Algorithm 2.

[0187]

[0188] Aiming at the specific characteristics of many-to-many communication routing problems, the present invention designs a multi-agent deep reinforcement learning method with centralized training and distributed execution under the AC framework, adopts GCN and GNN to design the Actor and Critic network structures respectively, which can effectively reduce the storage space overhead of model training, and designs a local observation method in the process of generating actions in the Actor network to avoid the generation of invalid actions, reduce the time overhead of model training and accelerate the convergence speed.

[0189] Based on the above analysis, the present invention proposes a many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN, which includes the following steps:

[0190] Step 1: Obtain network topology information G through the SDN architecture and obtain many-to-many communication tasks M based on historical data.

[0191] Step 2: Based on the changing patterns of the two attributes, the length of the network node packet queue and the cumulative energy consumption of the network node, the changing process of the two attributes during data transmission is deduced, and the optimization objectives and related constraints are summarized, thus obtaining the optimization problem:

[0192]

[0193] Step 3: Initialize the intelligent network node set and the common network node set, and use the classic random simple path transmission algorithm and random shortest path transmission algorithm to simulate the many-to-many communication task M in the topology information. During the data transmission process, the optimal set of intelligent network nodes is determined through the Q-Learning algorithm, and intelligent agents are deployed on these optimal intelligent network nodes.

[0194] Initialize the intelligent network node set to an empty set, simulate the many-to-many communication task M in the topology information During the transmission process, the system evaluates and decides whether to add network nodes or reduce network nodes in the intelligent network node set, and then simulates the many-to-many communication task M in the topology information. The process of transmission is repeated and evaluated again, and this process is repeated until the required number of intelligent network nodes is reached, which is the optimal network node set. In the specific implementation process, it is also necessary to design a reasonable reward function to facilitate the generation of optimal network nodes and adapt to the performance of the deployed intelligent agent in step 4. The specific implementation steps are as follows:

[0195] Step 3.1, define the set of all intelligent network nodes as the state space, and define all possibilities of dividing intelligent network nodes into the set of ordinary network nodes or selecting ordinary network nodes as the set of intelligent network nodes as the action space; and initialize the Q-table according to the definitions of the state space and action space, and initialize the state at the same time.

[0196] Step 3.2: Use the classic random simple path transmission algorithm and random shortest path transmission algorithm to simulate the many-to-many communication task M in the topology information During the data transmission process, rewards are calculated and actions are obtained through the Q-table.

[0197] Step 3.3: Feed the action back to the environment to obtain the next state and update the Q-table.

[0198] Step 3.4: Repeat steps 3.2 and 3.3 until the number of network nodes in the intelligent network node set is equal to the required number of intelligent network nodes, and return the optimal Q-table.

[0199] Step 3.5: After the training is completed, obtain the set of intelligent network nodes according to the optimal Q-table and deploy the intelligent agents on the intelligent network nodes.

[0200] Step 4: Initialize the intelligent agent on the intelligent network node and simulate the many-to-many communication task M in the topology information. In the data transmission process, the best agents are trained through the multi-agent graph reinforcement learning algorithm, and these best agents are deployed back to the corresponding intelligent network nodes to determine the routing of data packets in real time, so as to realize data transmission of many-to-many communication.

[0201] Initialize the intelligent agent on the intelligent network node and simulate the many-to-many communication task M in the topology information In the data transmission process, the data in the transmission process is stored in the data cache. When the data in the cache reaches a certain amount, data is sampled from it to train the intelligent agent. After training each intelligent agent, the new intelligent agent intelligent network node is used to simulate the many-to-many communication task M in the topology information. During the data transmission process, the data in the transmission process is stored in the data buffer area, and then sampled and trained again, and this cycle continues until the training process converges. After the training is completed, the optimal agent parameters generated during the training process are deployed, and data transmission for many-to-many communication can be achieved. The specific implementation steps are as follows:

[0202] Step 4.1, experience pool sampling: Each agent samples from the experience pool;

[0203] Step 4.2, Actor Network: Each agent extracts the queue length of all network nodes, the accumulated energy consumption of all network nodes, the position code of the network node where the current agent is located, and the previous network node position code and target network node position code of the packet to be decided on the network node where the current agent is located from the sampled samples. These are converted into graph structure data and input into the Actor Network of the current agent based on the Graph Convolutional Neural Network (GCN). The features of the next network node are extracted through the Actor Network.

[0204] Step 4.3: Local observation of the action space: Each agent first restricts the feature selection of the next-hop network node to the range of neighboring network nodes through local mapping, then maps it to the best next-hop network node through the Gumbel-Softmax method, and finally obtains the position encoding of the best next-hop network node through local inverse mapping of the action space to the corresponding position encoding;

[0205] Step 4.4, Critic Network: Each agent extracts the length of all network node packet queues, the cumulative energy consumption of all network nodes, the location codes of all network nodes where the agent is located, the location codes of the previous network node where the decision-making data packets on the network node where the agent is located, and the location codes of the target network node from the sampled samples. It also obtains the location codes of all the best next-hop network nodes through interaction with other agents, and inputs them into the current agent's Critic Network based on the graph neural network (GNN), and obtains rewards through the Critic Network.

[0206] Step 4.5, Network Parameter Update: The current agent updates the network parameters of its Critic network through gradient backpropagation based on the reward, and then updates the network parameters of its Actor network through feedback;

[0207] Step 4.6: Loop through steps 4.1 to 4.5 until convergence (the network parameters of the Actor network and Critic network do not change within a predetermined number of times, or the preset number of iterations is reached). This results in the optimal Actor network and Critic network for all agents, and deploys the Actor parameters of the agents based on the optimal Actor network and Critic network back to the corresponding intelligent network nodes for real-time routing of data packets, thereby achieving data transmission for many-to-many communication.

[0208] The following experimental analysis will be conducted from three parts: experimental environment, parameter settings and experimental results.

[0209] (1) Experimental environment

[0210] The network topology used in the experiment is as follows Figure 4 As shown in the figure, network nodes are divided into three types: data entry nodes are source nodes, relay nodes are responsible for data packet transmission, and data center nodes are destination nodes. It should be noted that the shortest path algorithm here only considers the number of path hops and ignores factors such as distance. However, as the above design derivation process shows, this is not limited to this.

[0211] (2) Parameter settings

[0212] For ease of description, use Represents the sequence representation of a set, that is, the order in which elements are added to the set. In this embodiment, the network node sequence is set to The source network node sequence is The target network node sequence is The corresponding task matrix is ​​set as according to Definition of m ij Represents the source sequence The sequence of the i-th network node sent to the destination network node The number of packets of the jth network node, such as m 11 =40 means that the number of packets sent from the source network node 14 to the destination network node 10 is 40. Through fine-tuning (i.e., meta-learning) to adapt to different M. In order to avoid the triviality of the problem, different unit time packet processing capabilities are given to network nodes. That is, the data processing capacity of the four network nodes numbered 1, 2, 12, and 13 per unit time is three times that of other network nodes.

[0213] (3) Experimental results

[0214] The intelligent network node selection experiment is responsible for selecting network nodes that require agent-assisted decision-making. The following experimental results are given in the many-to-many communication task matrix.

[0215] Figure 5 This is the experimental training reward curve for selecting intelligent network nodes using Q-learning. The result is obtained using the parameter α no =0.01,E1=600,n r =5. Figure 5 The middle curve is the reward value of the experiment during the training process, and the star points are the points where exploration occurs. Except for the star points, it can be clearly seen that the curve gradually increases, which shows the effectiveness of the algorithm.

[0216] Table 2 Intelligent network node selection results

[0217]

[0218] Table 2 shows the results of intelligent network node selection, where the parameter is set to α no =0.01,E1=600,n r The values ​​are 1 to 10, which are the settings for the number of smart network nodes in the table. The selection result and reward value are the set of smart network nodes selected after the number of smart network nodes is specified and the corresponding reward value.

[0219] As can be seen in Table 2, the results of the selection when the number of intelligent network nodes is 5 The selection results in the previous four cases are covered, and the reward value from the number of intelligent network nodes from 1 to 5 has been monotonically increasing. It starts to fluctuate after the number of intelligent network nodes is 5, and the corresponding reward value when a better result is reached again is 238.5, which is only 0.06 more than 237.9 when the number of intelligent network nodes is 5. However, the number of network nodes has increased by two, and the cost of the number of network nodes is relatively large. Therefore, the present invention is Figure 4 Topology, task matrix M task, select the number of intelligent network nodes is set to 5, the selection result is This result will be used to determine the deployment of the agent.

[0220] This experiment uses multi-agent graph reinforcement learning to optimize routing. By deploying agents on intelligent network nodes, the agents help determine the next hop to which the network nodes send data packets. For network nodes on ordinary networks, a random shortest path transmission algorithm is used to directly pass the data packet to the next hop.

[0221] To illustrate the effectiveness of the design method of the present invention, a benchmark experiment was designed, namely the SPR-M2M algorithm adapted from MMForests. The algorithm uses the Dijkstra algorithm to obtain the shortest hop many-to-many communication path algorithm between each source and destination pair. The experimental results show that the time delay for the algorithm to complete task M is 225 slots, and the standard deviation of the cumulative energy consumption on each network node when completing the task is 740.757.

[0222] Figure 6 is the reward result during the multi-agent graph reinforcement learning training process. The learning rate of the agent Actor network is The learning rate of the agent critic network is Batch update parameter b s =8, number of cycles E2 = 1500, loop penalty C h = 1. The rewards in the figure are all less than 0, and their absolute values ​​represent the total cost of data processing by the agents under task M. A single generation represents the cost of completing task M. The figure shows that as the number of iterations increases, the rewards for each agent gradually increase and then stabilize, demonstrating the effectiveness and convergence of the algorithm. Furthermore, it is noted that around 600 generations, the reward functions for agents 1, 2, 4, and 5 decrease, while the reward for agent 3 increases, indicating coordination between the agents.

[0223] Figure 7 The performance curve for the task completion time of the models generated during training under Case 1 is shown. The curved line represents the task completion latency for every five generations of the generated models during training, while the straight line represents the task completion latency for SPR-M2M (225 slots). As can be seen from the figure, as training progresses, the task completion time of MAGDS-M2M decreases and gradually surpasses that of SPR-M2M, demonstrating the effectiveness of the proposed method.

[0224] exist Figure 7As can be seen from the figure, the model achieves optimal results at generation 605, completing the task in just 173 slots, a 23.11% reduction compared to SPR-M2M. The generation 605 model is denoted as MAGDS-M2M-v605. The following further compares the differences between MAGDS-M2M-v605 and SPR-M2M algorithms by looking at the packet paths used to complete the task and the changes in the number of packets at key network nodes.

[0225] from Figure 8 As can be seen from the figure, compared with SPR-M2M, MAGDS-M2M-v605 makes better use of the network and takes into account all paths in the network. Figure 8 As can be seen from (a) and (b), the network nodes with more frequent data packet interactions are network nodes 4, 5, and 7. Figure 8 (c)-(e) show that in SPR-M2M, network node 4 is in a congested state for a long time, that is, the packet queue length remains high for a long time, network node 5 is idle for part of the time, and network node 7 is always idle, which leads to a large waste of resources. In contrast, MAGDS-M2M-v605 effectively solves this problem. Figure 8 (c) shows that the maximum length of the packet queue in network node 4 has changed from about 100 to about 60, which significantly reduces the congestion. Figure 8 (d) shows that the maximum length of the packet queue in network node 5 changes from about 70 to about 40. Figure 8 (e) shows that network node 7 went from being completely idle to having its packet queue length reach a maximum of approximately 80, but there was still some idle time, indicating that network node 7 was not yet a critical node hindering task completion. Furthermore, MAGDS-M2M-v605 significantly shortened task completion time, demonstrating that MAGDS-M2M-v605 fully utilized node 7 to effectively share the burden of network nodes 4 and 5, thereby accelerating task completion.

[0226] It should be noted that although the embodiments of the present invention described above are illustrative, they are not intended to limit the present invention. Therefore, the present invention is not limited to the above-mentioned specific embodiments. Without departing from the principles of the present invention, any other embodiments obtained by those skilled in the art under the guidance of the present invention are deemed to be within the protection of the present invention.

Claims

1. A many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN, characterized by: The steps are as follows: Step 1: Obtain network topology information and many-to-many communication tasks through the SDN architecture; Step 2: Based on the changing patterns of the two attributes, namely, the length of the network node packet queue and the cumulative energy consumption of the network node, the changing process of the two attributes during data transmission is deduced, and the optimization objectives and related constraints are summarized to obtain the optimization problem. Step 3: Initialize the set of intelligent network nodes and the set of ordinary network nodes, use the classic random simple path transmission algorithm and random shortest path transmission algorithm to simulate the data transmission process of many-to-many communication tasks in the topology information, determine the optimal set of intelligent network nodes through the Q-Learning algorithm, and deploy intelligent agents on these optimal intelligent network nodes; Step 4: Initialize the agents on the intelligent network nodes, simulate the data transmission process of the many-to-many communication task in the topology information, train the best agents through the multi-agent graph reinforcement learning algorithm, and deploy these best agents back to the corresponding intelligent network nodes to determine the routing of data packets in real time to achieve many-to-many communication data transmission; Step 4.1: Each agent samples from the experience pool. Step 4.2: Each agent extracts the queue length of all network nodes, the accumulated energy consumption of all network nodes, the position code of the network node where the current agent is located, and the previous network node position code and the target network node position code of the data packet to be decided on the network node where the current agent is located from the sampled samples. These data are converted into graph structure data and input into the Actor network based on the graph convolutional neural network of the current agent. The features of the next network node are extracted through the Actor network. Step 4.3: Each agent first restricts the feature selection of the next-hop network node to the range of neighboring network nodes through local mapping, then maps it to the best next-hop network node through the Gumbel-Softmax method, and finally obtains the position encoding of the best next-hop network node through local inverse mapping of the action space to the corresponding position encoding; Step 4.4: Each agent extracts the queue lengths of all network node packets, the accumulated energy consumption of all network nodes, the location codes of all network nodes where the agent is located, the location codes of the previous network nodes where the decision-making packets on the network nodes where the agent is located, and the location codes of the target network nodes from the sampled samples. It then interacts with other agents to obtain the location codes of all optimal next-hop network nodes, inputs them into the current agent's Critic network based on the graph neural network, and obtains rewards through the Critic network. Step 4.5: Based on the reward, the current agent updates the network parameters of its Critic network through gradient backpropagation, and then updates the network parameters of its Actor network through feedback; Step 4.6: Loop through steps 4.1 to 4.5 until convergence, thereby obtaining the optimal Actor network and Critic network for all agents. The Actor parameters of the agents based on the optimal Actor network and Critic network are deployed back to the corresponding intelligent network nodes for real-time routing of data packets to achieve many-to-many communication data transmission.

2. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 1 is characterized in that: In step 1, the traffic matrix M = [m ij ] to represent; where m ij Indicates that the source network node s i Sent to destination network node s j The number of packets, i′=1,2,…,n s , n s Indicates the number of source network nodes, j′=1,2,…,n d , n d Indicates the number of target network nodes.

3. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 1 is characterized in that: In step 2, the optimization problem is: in, is the optimization objective, β represents the weighted coefficient of the two objectives, represents the objective function to complete the task as quickly as possible, The objective function representing the possible equilibrium of the cumulative energy exhaustion at each network node; Represents network node s i Sent to network node s at time t j The number of packets; c i Represents network node s i The threshold value of the number of packets sent in one time step; e ij Represents network node s i With network nodes j The link between i represents the network node numbered i, s j Represents the network node numbered j, i≠j; represents the set of natural numbers, ε represents the set of links, Represents network node s i The set of neighbor network nodes, Represents a collection of network nodes.

4. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 1 is characterized in that: The specific process of step 3 is as follows: Step 3.

1. Define the set of all intelligent network nodes as a state space, and define all possible ways of dividing intelligent network nodes into a set of common network nodes or adding common network nodes to a set of intelligent network nodes as an action space; and initialize the Q-table according to the definitions of the state space and action space, and initialize the state; Step 3.2: Use the classic random simple path transmission algorithm and random shortest path transmission algorithm to simulate the data transmission process of the many-to-many communication task in the topology information, calculate the reward and obtain the action through the Q-table; Step 3.3: Feed the action back to the environment to obtain the next state and update the Q-table; Step 3.4: Repeat steps 3.2 and 3.3 until the number of network nodes in the intelligent network node set is equal to the required number of intelligent network nodes, and return the optimal Q-table; Step 3.5: After the training is completed, the best set of intelligent network nodes is obtained according to the best Q-table, and the intelligent agents are deployed on these best intelligent network nodes.

5. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 4, characterized in that: In step 3.2, the reward R no The calculation formula is: Where, Represents the network node s at time t through the classic random simple path transmission algorithm i Processing packets Rewards, Represents the network node s at time t through the random shortest path transmission algorithm i Processing packets Rewards, Represents network node s at time t i The kth packet processed, β no represents the weight coefficients of the two transmission algorithms, ζ represents the time per period T s The time from the start of data transmission to the completion of the task, Indicates the currently selected intelligent network node set, c i Represents network node s i The threshold for the number of packets sent in one time step.

6. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 5, characterized in that: Through the classic random simple path transmission algorithm, the intelligent network node s is i Processing packets Rewards for: Through the random shortest path transmission algorithm, the intelligent network node s at time t i Processing packets Rewards for: Where, In the classic random simple path transmission algorithm, Remove the predecessor network node from the possible next network node; Indicates that in the random shortest path transmission algorithm, in the data packet Remove the predecessor network node from the possible next network node; Represents network node s at time t i The kth packet processed.

7. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 1 is characterized in that: In step 4.3, the action space is to deliver the data packet to the corresponding neighbor network node or cache the data packet in the current network node.

8. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 1 is characterized in that the steps In 4.4, intelligent network nodes s i The agent's reward R on i (t) is: Where, represents the single-step reward for completing the task as quickly as possible, represents the single-step reward for the goal of exhausting the possible equilibrium at each network node, C h represents the loop penalty, C h is a constant less than 0; represents the set of intelligent network nodes; β represents the weighted coefficient of the two objectives, is the indicator function, Indicates the action taken by the current agent, Indicates the position code corresponding to the action made by the current agent, Indicates the previous network node position code of the data packet to be decided on the network node where the current intelligent agent is located.

9. The many-to-many communication routing method based on multi-agent graph reinforcement learning in SDWN according to claim 8, characterized in that: Single-step reward for completing the task as quickly as possible for: The single-step reward for the target of exhausting the possible equilibrium at each network node for: Indicator Function for: Where p(t) represents the length vector of the packet queues on all network nodes at time t, p(t+1) represents the length vector of the packet queues on all network nodes at time t+1, and κ represents the upper bound of the number of packets in the network; e c (t) represents the cumulative energy consumption vector of all network nodes at time t, e c (t+1) represents the cumulative energy consumption vector of all network nodes at time t+1; Indicates the action taken by the current agent, Indicates the position code corresponding to the action made by the current agent, Indicates the previous network node position code of the data packet to be decided on the network node where the current intelligent agent is located; sum(·) represents the sum of the vector, max(·) represents the maximum value of the vector, min(·) represents the minimum value of the vector, and std(·) represents the standard deviation of the vector.

Images