Intelligent routing method, device and equipment based on reinforcement learning

Abstract

The invention discloses an intelligent routing method, device and equipment based on reinforcement learning. Comprising the steps that a communication network model graph is constructed, the communication network model graph comprises a plurality of network nodes, and each network node is an intelligent agent; at any moment, each network node obtains network state information obtained after current interaction with the environment; inputting the network state information into a strategy network of the intelligent agent, and outputting an action with the maximum probability; and determining the next network node of the data packet routing according to the output action information, and sending the data packet to the selected next network node. According to the method, the adaptive multi-agent reinforcement learning algorithm is adopted, each routing node makes the routing decision according to the current network state without establishing a routing table, and compared with other algorithms, the method has the advantages that the lower average transmission delay of data packets is realized, and the network routing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of network routing technology, and in particular to an intelligent routing method, device and equipment based on reinforcement learning. Background Art

[0002] Routing is the process of selecting a transmission path for data packets in the network, and is the core link in network communication. The continuous expansion of network scale and the complexity of network status have brought problems such as efficiency and reliability to data packet routing. Therefore, how to improve the efficiency of data packet routing and reduce transmission delay has always attracted the attention of researchers.

[0003] Current dynamic routing algorithms usually calculate the best path to the destination based on certain optimization criteria, such as the minimum number of hops or the minimum cost. In addition, in order to adapt to changes in network status and calculate the best route, it is often necessary to regularly exchange local databases with surrounding or even more distant nodes, which creates certain bandwidth pressure, and the routing strategy is not time-effective. This type of algorithm usually has the following two problems: 1) Limitations of network status perception: When the network scale is large, due to the large number of nodes and their wide distribution, it is difficult for a single node to fully perceive the real-time status of the entire network, including path congestion and other situations; 2) The singleness of path selection: Since data packets select paths based on the same rules, a surge in a certain type of traffic may cause one path to be overloaded while node resources on other paths are idle. Summary of the invention

[0004] The embodiments of the present application provide a reinforcement learning-based intelligent routing method, apparatus, and device to at least solve the technical problems of low routing efficiency and large transmission delay in related technologies.

[0005] According to one aspect of an embodiment of the present application, there is provided an intelligent routing method based on reinforcement learning, comprising:

[0006] Constructing a communication network model graph, wherein the communication network model graph includes a plurality of network nodes, wherein each network node is an intelligent agent;

[0007] At any time, each network node obtains the current network status information obtained after interacting with the environment;

[0008] Inputting the network state information into the agent's policy network and outputting the action with the highest probability;

[0009] The next network node for the data packet routing is determined according to the output action information, and the data packet is sent to the selected next network node.

[0010] In an optional embodiment, the network status information includes the starting node of the data packet, the target node of the data packet, three historical actions of the current network node, and the number of data packets currently queued by adjacent network nodes.

[0011] In an optional embodiment, after each network node obtains the network status information currently obtained after interacting with the environment, the method further includes:

[0012] Get the set of all adjacent network nodes of the network node where the current agent is located;

[0013] An action space of the agent is generated according to the set of all adjacent network nodes.

[0014] In an optional embodiment, before inputting the network status information into the policy network of the agent, the method further includes:

[0015] Construct a network model of the agent;

[0016] The network model of the intelligent agent includes a strategy network and an evaluation network;

[0017] The policy network consists of an input layer, a hidden layer, and an output layer, and is used to decide which action should be taken under the current network state;

[0018] The evaluation network consists of an input layer, a hidden layer, and an output layer, and is used to evaluate the value of actions under the current strategy.

[0019] In an optional embodiment, after determining the next network node for routing the data packet according to the output action information, the method further includes:

[0020] Determining whether the next network node of the data packet routing is a target node;

[0021] If it is not the target node, record the number of hops, transmission cost, queuing delay, and status value of the data packet at the network node;

[0022] If it is the target node, calculate the reward value of each node on the entire transmission path from the starting node to the target node.

[0023] In an optional embodiment, calculating the reward value of the node includes:

[0024] The reward value of the current node is obtained by subtracting the transmission cost from the reward value of the next hop node and the queuing delay of the current node.

[0025] In an optional embodiment, calculating the state value of the network node includes:

[0026] The network state information is input into the evaluation network of the intelligent agent to obtain the state value under the current strategy.

[0027] In an optional embodiment, it also includes:

[0028] Calculate the loss value of the evaluation network and the strategy network of any node according to the state value and the reward value;

[0029] The evaluation network and the policy network are updated based on the loss values ​​of the evaluation network and the policy network.

[0030] According to another aspect of the embodiment of the present application, there is also provided an intelligent routing device based on reinforcement learning, including:

[0031] A network graph construction module, used to construct a communication network model graph, wherein the communication network model graph includes a plurality of network nodes, wherein each network node is an intelligent agent;

[0032] The state space acquisition module is used for each network node to obtain the current network state information obtained after interacting with the environment at any time;

[0033] A decision module, used for inputting the network state information into the strategy network of the agent and outputting the action with the highest probability;

[0034] The execution module is used to determine the next network node of the data packet routing according to the output action information, and send the data packet to the selected next network node.

[0035] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned intelligent routing method based on reinforcement learning through the computer program.

[0036] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0037] The embodiment of the present application provides an intelligent routing method based on reinforcement learning, which regards each network node as an independent intelligent agent, and makes routing decisions through a separate model, giving the optimal action according to the current real-time network status. It enables the intelligent agent to dynamically adjust the routing decision according to the current network status to achieve load balancing, avoid overloading of a single path, and improve the flexibility of routing. It effectively reduces the average delay of data packets transmitted in the network, meets the needs of application scenarios with high real-time requirements, and improves the routing efficiency of data packets. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 is a flow chart of an intelligent routing method based on reinforcement learning according to an embodiment of the present application;

[0040] Figure 2 is a communication network model diagram according to an embodiment of the present application;

[0041] Figure 3 is a schematic diagram of an intelligent routing method based on reinforcement learning according to an embodiment of the present application;

[0042] Figure 4 is a simulated network node diagram according to an embodiment of the present application;

[0043] Figure 5 is a schematic diagram of a simulation result according to an embodiment of the present application;

[0044] Figure 6 is a schematic diagram of a simulation result according to an embodiment of the present application;

[0045] Figure 7 is a schematic diagram of an intelligent routing device based on reinforcement learning according to an embodiment of the present application;

[0046] Figure 8 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] This application proposes a low-latency routing algorithm based on multi-agent reinforcement learning, with the goal of minimizing the transmission delay between nodes and learning the optimal routing strategy. In the scheme of this application, each node is regarded as an independent agent, and routing decisions are made through a separate model. The proposed scheme not only takes into account the characteristics of the data packet during the training process, but also takes into account the current status of the neighboring nodes (data packet queuing status, etc.). Fully considering the information of adjacent nodes can, on the one hand, enhance the agent's perception of the network and realize the estimation of network congestion; on the other hand, it enables the agent to dynamically adjust the routing decision according to the current network status to achieve load balancing, avoid overloading of a single path, and improve the flexibility of routing.

[0050] The following is combined with Figure 1 The intelligent routing method based on reinforcement learning in the embodiment of the present application is introduced in detail. It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this respect. On the contrary, the implementation of the present application can be applied to any applicable scenario. Figure 1 As shown, the method mainly includes the following steps:

[0051] S101 constructs a communication network model diagram, which includes multiple network nodes, each of which is an intelligent agent.

[0052] like Figure 2 As shown in the figure, a communication network model diagram is constructed. A communication network consists of multiple network nodes, which are connected to each other through links. Each node can forward and receive data packets. Let the undirected graph G = (V, E) represent the topology of the communication network, where V and E are the set of router nodes and the set of links between nodes respectively. Let the data transmission rate of node i be C i .

[0053] Define a data packet as a triple (s, d, B), where s represents the starting node of the data packet in the network, d represents the destination node of the data packet route, and B is the data packet size in bytes. The transmission time of a data packet with a data size of B at node i can be expressed as

[0054] Each node in the network is regarded as an intelligent agent.

[0055] Depend on Figure 2It can be seen that the task of packet routing is to transmit each packet from the starting node to its destination node through the relay of multiple routers, which mainly includes three parts: the packet arrives at the starting node, the packet queues, and the packet transmission. Among them, the queue in the node follows the first-in-first-out principle. The packet routing problem is to find the optimal transmission path between the source node and the destination node. Each router continuously passes the packet at the head of its queue to its neighbor node until the packet reaches the destination.

[0056] The goal of this application is to design a routing algorithm that minimizes the average transmission delay of all data packets from the source node to the destination node. Let P be the set of all data packets to be transmitted, t p is the total transmission delay of data packet p from the starting point to the destination node. Based on this, the task of minimizing the average transmission delay of data packets can be expressed as:

[0057]

[0058] Wherein, K is the size of the data packet set P, that is, the total number of all data packets.

[0059] S102 At any time, each network node obtains network status information obtained after currently interacting with the environment.

[0060] The network status information includes the starting node of the data packet, the target node of the data packet, the three historical actions of the current network node, and the number of data packets currently queued by the adjacent network nodes.

[0061] In an environment where the network state is constantly changing, it is crucial to ensure that the agent can effectively observe the network information it needs. The state space of each node is defined as a four-tuple, including the starting point of the data packet, the target node of the data packet, the three historical actions of this node, and the number of packets currently queued by the neighboring nodes.

[0062] If we only focus on the attributes of the data packet, such as the starting node, the destination node, and the size of the data packet, this observation method may ignore the complexity and changes of the network environment and has certain limitations. Therefore, this application considers taking indicators that can reflect the immediate congestion status of the network as part of the input, such as the number of data packets currently queued by adjacent nodes, to improve the flexibility of routing decisions.

[0063] Furthermore, after each network node obtains the network state information obtained after the current interaction with the environment, it also includes: obtaining all adjacent network node sets of the network node where the current intelligent agent is located; and generating the action space of the intelligent agent according to all adjacent network node sets.

[0064] It can be understood that the action space is defined as the set of neighbor nodes of node i. Accordingly, for each agent, the size of the action space is equal to the number of its neighboring nodes.

[0065] S103 inputs the network status information into the agent's strategy network and outputs the action with the highest probability.

[0066] First, construct the agent network model. The agent network model includes a policy network and an evaluation network. The policy network consists of an input layer, a hidden layer, and an output layer, and is used to decide which action should be taken in the current network state; the evaluation network consists of an input layer, a hidden layer, and an output layer, and is used to evaluate the value of the action under the current policy.

[0067] Specifically, each agent uses the Actor-Critic framework, and the agent consists of two networks: an Actor policy network is responsible for generating actions, and a Critic evaluation network is responsible for evaluating the value of the current state. The Actor and Critic use the same neural network structure. The input layer inputs the state obtained after interacting with the environment. The input layer is followed by a hidden layer containing 256 neurons. Its output is obtained through the softmax function to obtain the probability of each selectable action. The agent makes corresponding routing decisions based on the probability.

[0068] Furthermore, action decisions are made based on the agent-based policy network. Each node (agent) in the network learns and routes independently. At any time t, each node observes the current network state, and based on the current state representation, inputs the current state into the agent policy network, outputs the action with the highest probability, that is, the next hop node that determines the routing of the data packet, which can be expressed as:

[0069] a t = argmax a Q(θ q );

[0070] Among them, Q is the policy neural network, θ q is the network parameter, a t is the output of the policy network in the current state.

[0071] S104 determines the next network node for the data packet routing according to the output action information, and sends the data packet to the selected next network node.

[0072] In one embodiment, the next hop node of the data packet's route is determined based on the action with the highest output probability. After the next hop node is determined based on the policy network, the data packet is sent through the selected path.

[0073] In an optional embodiment, after determining the next network node for routing the data packet according to the output action information, the method further includes:

[0074] Determine whether the next network node of the data packet routing is the target node; if it is not the target node, record the number of hops, transmission cost, queuing delay and state value of the data packet at the network node; if it is the target node, calculate the reward value of each node on the entire transmission path from the starting node to the target node.

[0075] The definition of reward is to make the agent choose paths that can reduce the transmission delay and queuing time of data packets, while reducing the number of data packet transmissions, thereby improving the overall performance of the network. For any node i, the reward r is defined in this application i It is the reward of the next hop node minus the transmission cost minus the queuing delay of the node, which can be specifically expressed as:

[0076] r i =γ·r i6+1 -q i+1 -cost

[0077] Where γ is the attenuation factor, which is 0.9 in an exemplary embodiment, cost is the transmission cost, and q i+1 is the queuing delay at the next node. i is the reward of the current node, r i+1 is the reward of the next hop node. The transmission cost is set to a constant, which indicates that there is a cost to transmit data packets. The agent will try to reduce the cost during the training process, so as to reduce the number of data packet transmissions as much as possible.

[0078] The calculation of the state value of the network node includes: inputting the network state information into the evaluation network of the intelligent agent to obtain the state value under the current strategy; and outputting the action value under the current strategy according to the evaluation network.

[0079] Furthermore, according to the state value and the reward value, the loss value of the evaluation network and the policy network of any node is calculated; and the evaluation network and the policy network are updated based on the loss value of the evaluation network and the policy network.

[0080] In one embodiment of the present application, the loss values ​​of the evaluation network and the strategy network for any node i are calculated as follows:

[0081]

[0082] Among them, L critic Indicates the loss value of the evaluation network; L actor represents the loss value of the policy network; J is the set of all data packets passing through node i, Q(θ q) represents the reward value on the node, V(θ π ) represents the state value of the node, π * Represents the optimal strategy. The meaning of the strategy network loss expression is that when the TD error value is greater than 0, the training network increases the probability of selecting the action, otherwise it reduces the probability of selecting the action.

[0083] Furthermore, back-propagation is performed to update the model.

[0084]

[0085] Among them, α1 and α2 are the learning rates of the policy network and the evaluation network respectively.

[0086] Through the above update method, the agent updates the policy network to optimize routing decisions and updates the value network to better evaluate the value of the current state, so as to obtain better performance in the future, and continues to iterate until convergence or the number of iterations is reached.

[0087] The routing scheme of the present application provides a multi-agent joint routing framework based on deep reinforcement learning. Through the combination of multi-agent information, each agent dynamically makes routing decisions based on the real-time network status to ensure high reliability and low latency of data packet transmission. Among them, the signaling content between nodes in multi-agent interaction includes the node unique identifier and the number of data packets currently queued by adjacent nodes. Through less signaling overhead, nodes are supported to perceive the network status to support efficient routing decisions. In the scheme of the present application, the definition of reinforcement learning reward is the reward of the next node minus the transmission cost and the queuing time of the node. This definition aims to reduce the transmission delay of data packets, and also reduce the number of data packet transmissions.

[0088] In order to facilitate understanding of the solution provided in the embodiment of the present application, Figure 3 Further description.

[0089] like Figure 3 As shown, each node in the network is regarded as an agent, and for each agent, it includes a policy network and an evaluation network.

[0090] First, obtain the environment state. Obtain the attributes of the current data packet to be transmitted and request the number of currently queued data packets from the adjacent nodes. Input the obtained network state information into the policy network and output the action. According to the output action, get the next hop node of the data packet. Input the obtained environment state into the evaluation network to get the value of the current action.

[0091] Furthermore, the reward is calculated, the TD error is calculated based on the reward and value output, and the policy network and value network are updated based on the error loss. The network is optimized. Through the above steps, it is continuously iterated until convergence, thus improving the routing performance of the network.

[0092] like Figure 4 As shown in the simulation, a network topology with 9 nodes was randomly generated using the Python simulation environment. Figure 5 The simulation results show that in 300 iterations, the cumulative returns of all intelligent nodes show a decreasing trend, which means that the delay and number of transmissions are effectively reduced. The proposed solution helps to improve the transmission efficiency and performance of the overall network.

[0093] Figure 6 The simulation results compare the cumulative distribution function (CDF) of the proposed scheme with the classic OSPF algorithm. It can be seen that the average delay of data packet transmission in the proposed scheme is smaller, and more than 80% of the data packets can reach the target node within 5ms. This is because the proposed scheme has a more comprehensive perception of the network status, while avoiding the singleness of path selection and achieving load balancing.

[0094] According to another aspect of the embodiments of the present application, there is also provided a reinforcement learning-based intelligent routing device for implementing the above-mentioned reinforcement learning-based intelligent routing method. Figure 7 As shown, the device comprises:

[0095] A network diagram construction module 701 is used to construct a communication network model diagram, wherein the communication network model diagram includes a plurality of network nodes, wherein each network node is an intelligent agent;

[0096] The state space acquisition module 702 is used for each network node to obtain the network state information obtained after the current interaction with the environment at any time;

[0097] Decision module 703, used to input network status information into the agent's strategy network and output the action with the highest probability;

[0098] The execution module 704 is used to determine the next network node for the data packet routing according to the output action information, and send the data packet to the selected next network node.

[0099] It should be noted that the intelligent routing device based on reinforcement learning provided in the above embodiment only uses the division of the above functional modules as an example when executing the intelligent routing method based on reinforcement learning. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the intelligent routing device based on reinforcement learning provided in the above embodiment and the embodiment of the intelligent routing method based on reinforcement learning belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0100] According to another aspect of the embodiments of the present application, an electronic device corresponding to the intelligent routing method based on reinforcement learning provided in the aforementioned embodiments is also provided to execute the intelligent routing method based on reinforcement learning.

[0101] Please refer to Figure 8 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 8 As shown, the electronic device includes: a processor 800, a memory 801, a bus 802 and a communication interface 803, and the processor 800, the communication interface 803 and the memory 801 are connected through the bus 802; the memory 801 stores a computer program that can be run on the processor 800, and when the processor 800 runs the computer program, it executes the intelligent routing method based on reinforcement learning provided in any of the aforementioned embodiments of the present application.

[0102] The memory 801 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 803 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0103] The bus 802 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs, and the processor 800 executes the program after receiving an execution instruction. The intelligent routing method based on reinforcement learning disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 800, or implemented by the processor 800.

[0104] The processor 800 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 800. The above processor 800 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 801, and the processor 800 reads the information in the memory 801 and completes the steps of the above method in combination with its hardware.

[0105] The electronic device provided in the embodiment of the present application and the intelligent routing method based on reinforcement learning provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. An intelligent routing method based on reinforcement learning, characterized in that: include: Constructing a communication network model graph, wherein the communication network model graph includes a plurality of network nodes, wherein each network node is an intelligent agent; At any time, each network node obtains the current network status information obtained after interacting with the environment; Inputting the network state information into the agent's policy network and outputting the action with the highest probability; The next network node for the data packet routing is determined according to the output action information, and the data packet is sent to the selected next network node.

2. The method according to claim 1, characterized in that The network status information includes the starting node of the data packet, the target node of the data packet, three historical actions of the current network node, and the number of data packets currently queued by adjacent network nodes.

3. The method according to claim 1, characterized in that After each network node obtains the current network status information obtained after interacting with the environment, it also includes: Get the set of all adjacent network nodes of the network node where the current agent is located; An action space of the agent is generated according to the set of all adjacent network nodes.

4. The method according to claim 1, characterized in that: Before inputting the network state information into the strategy network of the agent, the method further includes: Construct a network model of the agent; The network model of the intelligent agent includes a strategy network and an evaluation network; The policy network consists of an input layer, a hidden layer, and an output layer, and is used to decide which action should be taken under the current network state; The evaluation network consists of an input layer, a hidden layer, and an output layer, and is used to evaluate the value of actions under the current strategy.

5. The method according to claim 1, characterized in that After determining the next network node for the data packet routing according to the output action information, the method further includes: Determining whether the next network node of the data packet routing is a target node; If it is not the target node, record the number of hops, transmission cost, queuing delay, and status value of the data packet at the network node; If it is the target node, calculate the reward value of each node on the entire transmission path from the starting node to the target node.

6. The method according to claim 5, characterized in that Calculate the node's reward value, including: The reward value of the current node is obtained by subtracting the transmission cost from the reward value of the next hop node and the queuing delay of the current node.

7. The method according to claim 5, characterized in that Calculate the status value of network nodes, including: The network state information is input into the evaluation network of the intelligent agent to obtain the state value under the current strategy.

8. The method according to claim 5, characterized in that Also includes: Calculate the loss value of the evaluation network and the strategy network of any node according to the state value and the reward value; The evaluation network and the policy network are updated based on the loss values ​​of the evaluation network and the policy network.

9. An intelligent routing device based on reinforcement learning, characterized in that: include: A network graph construction module, used to construct a communication network model graph, wherein the communication network model graph includes a plurality of network nodes, wherein each network node is an intelligent agent; The state space acquisition module is used for each network node to obtain the current network state information obtained after interacting with the environment at any time; A decision module, used for inputting the network state information into the strategy network of the agent and outputting the action with the highest probability; The execution module is used to determine the next network node of the data packet routing according to the output action information, and send the data packet to the selected next network node.

10. An electronic device, characterized in that: The invention comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the intelligent routing method based on reinforcement learning as claimed in any one of claims 1 to 8 when executing the program instructions.

Images