A single-agent routing control method, device, equipment and medium

Abstract

This application discloses a single-agent routing control method, apparatus, device, and medium, relating to the field of routing control optimization, including: constructing a single-agent of a network to be controlled based on a node topology matrix of the network to be controlled; iterating the routing optimization path output by the single-agent based on a preset neural network framework until a preset number of iterations is reached, and outputting the routing optimization path of the final iteration; wherein, during each iteration, the routing optimization path is gradually generated based on the neural network framework through deep reinforcement learning; during each generation step, the next-hop routing node is selected based on a sorting mechanism to update the routing optimization path; and each routing node of the network to be controlled is controlled based on the routing optimization path of the final iteration. Implementation of this application can solve the technical problem that existing routing control methods are prone to generating unreachable routes.

Description

Technical Field

[0001] The present application relates to the field of routing control optimization, and in particular to a single-agent routing control method, apparatus, device, and medium. Background Art

[0002] The distribution communication network is a core component of the power system, responsible for information transmission and communication between different power nodes. As the scale of the power grid continues to expand, manual inspection and maintenance methods are no longer sufficient to ensure the stable operation of the distribution communication network, necessitating more efficient maintenance methods. In recent years, the use of AI algorithms to intelligently adjust traffic flow and route node routing within the distribution communication network has become a research focus, enabling efficient and high-quality maintenance of the distribution communication network.

[0003] In existing technologies, the primary approach to maintaining distribution communication networks using AI algorithms is deep reinforcement learning routing optimization based on AI-constructed agents. Among these, several prominent research efforts include: a quality-of-service (QoS)-aware routing algorithm using deep reinforcement learning in knowledge-defined networks. In this algorithm, the agent determines the optimal strategy by interactively learning from domain data networks and maximizing an expected reward function, thereby optimizing routing configuration. A second approach maximizes a reward function common to all routes by jointly learning the network environment and its dynamic changes. A third approach involves a two-stage joint routing selection and network resource scheduling algorithm based on reinforcement learning, addressing transmission reliability and efficiency issues in large-scale mesh networks. Finally, a mobile ad hoc network routing protocol based on a Deep Q-Network (DQN) calculates node costs by calculating their residual energy and mobility speed. Routing control is performed based on the Q-value of each node calculated by the DQN, reducing control overhead and improving packet transmission rates. While these algorithms can optimize routing to some extent, achieving effective grid maintenance, they are prone to unreachable routes in large-scale routing scenarios. Summary of the Invention

[0004] The present application provides a single-agent routing control method, apparatus, device and medium to solve the technical problem that existing routing control methods easily generate unreachable routes.

[0005] According to a first aspect of the embodiments of the present application, a single-agent routing control method is provided, comprising:

[0006] Construct a single intelligent agent of the network to be controlled according to the node topology matrix of the network to be controlled;

[0007] Based on a preset neural network framework, the routing optimization path output by the single agent is iterated until a preset number of iterations is reached, and the routing optimization path of the final iteration is output; wherein, during each iteration, the routing optimization path is gradually generated based on the neural network framework through deep reinforcement learning; during each generation step, the next hop routing node is selected based on a sorting mechanism to update the routing optimization path;

[0008] According to the routing optimization path of the last iteration, each routing node of the control network is controlled.

[0009] This application first constructs a single intelligent agent based on the node topology matrix of the network to be controlled, and then iterates the routing optimization path output by the single intelligent agent based on the neural network framework. In each round of iteration, the routing optimization path is gradually generated through deep reinforcement learning, which can enable the routing optimization path to continuously approach the target routing node from the starting routing node, thereby completing the task. In each step of generation, the next-hop routing node is selected based on the sorting mechanism to update the routing optimization path. The next-hop routing node is selected through the sorting mechanism, which can reasonably select the next-hop routing node and avoid the generation of paths with unreachable routes. Therefore, when controlling the routing nodes of the network to be controlled according to the routing optimization path of the last iteration, routing control can be performed more reasonably.

[0010] In certain embodiments of the present application, constructing a single agent of the network to be controlled according to the node topology matrix of the network to be controlled specifically includes:

[0011] Obtain the node topology, node location and node traffic of each routing node in the network to be controlled;

[0012] Constructing the node topology matrix according to the node topology, the node location, and the node traffic; wherein the node topology matrix includes node weights of each routing node in the network to be controlled; and the node weights are calculated based on the node topology;

[0013] According to the node topology matrix, a single intelligent agent of the network to be controlled is constructed.

[0014] This application first obtains the node data of each routing node in the network to be controlled, then constructs a node topology matrix with node weights, and then constructs a single intelligent agent, which facilitates the subsequent iteration of the routing optimization path output by the single intelligent agent and is more in line with the actual routing optimization.

[0015] In certain embodiments of the present application, the stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes:

[0016] Inputting the initial state of the single agent into the neural network framework to obtain an initial action;

[0017] generating an initial routing optimization path according to the initial state and the initial action;

[0018] Through deep reinforcement learning, the initial route optimization path is iteratively updated step by step based on the neural network framework to generate a route optimization path; wherein, in each step of generation, the updated state of the single agent is obtained according to the current state of the single agent, and the network parameters of the initial route optimization path and the neural network framework are updated according to the updated state; in the first step of generation, the current state of the single agent is the initial state; in the last step of generation, the updated state of the single agent corresponds to the route optimization path.

[0019] This application first inputs the initial state of a single agent into a neural network framework to obtain an initial action, then generates an initial routing optimization path, and gradually iterates and updates the initial routing optimization path through deep reinforcement learning to generate a routing optimization path. In each step of generation, an updated state is obtained according to the current state of the single agent, and the network parameters of the initial routing optimization path and the neural network framework are updated according to the updated state. By gradually iterating and updating the path through deep reinforcement learning, it can be ensured that the path continuously approaches the target routing node from the starting routing node, and the network parameters of the neural network framework are updated at each step, which can timely update the network parameters and accelerate the convergence speed of the framework.

[0020] In certain embodiments of the present application, obtaining an updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state, specifically includes:

[0021] According to the current state of the single agent, obtaining the current action of the single agent based on the neural network framework;

[0022] Selecting a next-hop routing node of a current routing node based on a ranking mechanism; wherein the current routing node corresponds to a current action of the single agent;

[0023] Obtaining the updated state of the single agent according to the next-hop routing node of the current routing node;

[0024] The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

[0025] This application first obtains the current action based on the current state of the single intelligent agent based on the neural network framework, and then selects the next-hop routing node of the current routing node corresponding to the current action according to the sorting mechanism. It can reasonably select the next-hop routing node to avoid the generation of paths with unreachable routes, and then obtain the updated state according to the next-hop routing node, and update the initial routing optimization path and the network parameters of the neural network framework according to the updated state, which can ensure that the path continuously approaches the target routing node from the starting routing node, and can update the network parameters in time, thereby increasing the convergence speed of the framework.

[0026] In certain embodiments of the present application, selecting the next-hop routing node of the current routing node based on the sorting mechanism specifically includes:

[0027] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0028] Obtaining an adjacent node distance sorting table and an adjacent node weight sorting table based on descending sorting according to the adjacent node target distance list and the adjacent node weight list;

[0029] According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next-hop routing node of the current routing node is selected.

[0030] The present application first obtains an adjacent node target distance list and an adjacent node weight list corresponding to the adjacent routing node list of the current routing node, then obtains a corresponding sorting table based on descending sorting, and obtains an adjacent node comprehensive sorting table based on the sorting table, and then selects the next-hop routing node of the current routing node. Through the target distance list and weight list corresponding to the adjacent routing node list of the current routing node, the potential next-hop routing nodes of the current routing node can be reasonably quantified, and then when performing comprehensive sorting to select the next-hop routing node, the selected next-hop routing node can be made more reasonable, avoiding the generation of paths with unreachable routes.

[0031] In certain embodiments of the present application, updating the network parameters of the neural network framework according to the update state specifically includes:

[0032] Calculating a reward value of the current state according to the current state and the updated state;

[0033] According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on the gradient descent algorithm;

[0034] The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically:

[0035] ;

[0036] in, is the sample set, They are the current state, current action, reward value and updated state respectively. For the The expected cumulative reward of an evaluation network under the current state and current action, For the The network parameters of the evaluation network, is the target value, is the discount factor, For the The target evaluation network is the expected cumulative reward under the updated state and the action corresponding to the updated state, For the The network parameters of the target evaluation network are used.

[0037] This application first calculates the reward value based on the current state and the updated state, and then updates the network parameters of the neural network framework based on the gradient descent algorithm, which can update the corresponding network so that it converges to the range required by the routing optimization task.

[0038] In certain embodiments of the present application, the stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate the route optimization path further includes:

[0039] When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state;

[0040] The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically:

[0041] ;

[0042] in, is the sample set, is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, is the output of the action network in the current state, are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0043] When the number of generated steps reaches a preset generation step threshold, the present application updates the network parameters of the neural network framework based on the gradient ascent algorithm according to the current state, and can update the corresponding network so that it converges to the range required by the routing optimization task.

[0044] According to a second aspect of the embodiments of the present application, there is provided a single-agent routing control device, comprising a single-agent construction module, a path optimization module, and a routing control module;

[0045] The single agent construction module is used to construct a single agent of the network to be controlled according to the node topology matrix of the network to be controlled;

[0046] The path optimization module is configured to iterate the route optimization path output by the single agent based on a preset neural network framework until a preset number of iterations is reached, and output the route optimization path of the final iteration; wherein, during each iteration, the route optimization path is gradually generated based on the neural network framework through deep reinforcement learning; during each generation step, the next hop routing node is selected based on a sorting mechanism to update the route optimization path;

[0047] The routing control module is used to control each routing node of the network to be controlled according to the routing optimization path of the last iteration.

[0048] In certain embodiments of the present application, the single-agent construction module includes a node data acquisition unit, a topology matrix construction unit, and a single-agent construction unit;

[0049] The node data acquisition unit is used to obtain the node topology, node location and node traffic of each routing node in the network to be controlled;

[0050] The topology matrix construction unit is configured to construct the node topology matrix according to the node topology, the node position, and the node traffic; wherein the node topology matrix includes node weights of each routing node in the network to be controlled; and the node weights are calculated based on the node topology;

[0051] The single agent construction unit is used to construct a single agent of the network to be controlled according to the node topology matrix.

[0052] In certain embodiments of the present application, the stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes:

[0053] Inputting the initial state of the single agent into the neural network framework to obtain an initial action;

[0054] generating an initial routing optimization path according to the initial state and the initial action;

[0055] Through deep reinforcement learning, the initial route optimization path is iteratively updated step by step based on the neural network framework to generate a route optimization path; wherein, in each step of generation, the updated state of the single agent is obtained according to the current state of the single agent, and the network parameters of the initial route optimization path and the neural network framework are updated according to the updated state; in the first step of generation, the current state of the single agent is the initial state; in the last step of generation, the updated state of the single agent corresponds to the route optimization path.

[0056] In certain embodiments of the present application, obtaining an updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state, specifically includes:

[0057] According to the current state of the single agent, obtaining the current action of the single agent based on the neural network framework;

[0058] Selecting a next-hop routing node of a current routing node based on a ranking mechanism; wherein the current routing node corresponds to a current action of the single agent;

[0059] Obtaining the updated state of the single agent according to the next-hop routing node of the current routing node;

[0060] The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

[0061] In certain embodiments of the present application, selecting the next-hop routing node of the current routing node based on the sorting mechanism specifically includes:

[0062] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0063] Obtaining an adjacent node distance sorting table and an adjacent node weight sorting table based on descending sorting according to the adjacent node target distance list and the adjacent node weight list;

[0064] According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next-hop routing node of the current routing node is selected.

[0065] In certain embodiments of the present application, updating the network parameters of the neural network framework according to the update state specifically includes:

[0066] Calculating a reward value of the current state according to the current state and the updated state;

[0067] According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on the gradient descent algorithm;

[0068] The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically:

[0069] ;

[0070] in, is the sample set, They are the current state, current action, reward value and updated state respectively. For the The expected cumulative reward of an evaluation network under the current state and current action, For the The network parameters of the evaluation network, is the target value, is the discount factor, For the The target evaluation network is the expected cumulative reward under the updated state and the action corresponding to the updated state, For the The network parameters of the target evaluation network are used.

[0071] In certain embodiments of the present application, the stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate the route optimization path further includes:

[0072] When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state;

[0073] The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically:

[0074] ;

[0075] in, is the sample set, is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, is the output of the action network in the current state, are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0076] This application first constructs a single intelligent agent based on the node topology matrix of the network to be controlled, and then iterates the routing optimization path output by the single intelligent agent based on the neural network framework. In each round of iteration, the routing optimization path is gradually generated through deep reinforcement learning, which can enable the routing optimization path to continuously approach the target routing node from the starting routing node, thereby completing the task. In each step of generation, the next-hop routing node is selected based on the sorting mechanism to update the routing optimization path. The next-hop routing node is selected through the sorting mechanism, which can reasonably select the next-hop routing node and avoid the generation of paths with unreachable routes. Therefore, when controlling the routing nodes of the network to be controlled according to the routing optimization path of the last iteration, routing control can be performed more reasonably.

[0077] According to the third aspect of the implementation of the present application, a computer device is provided, comprising: a processor; a memory; a computer program stored in the memory and configured to be executed by the processor; wherein when the processor executes the computer program, a single-agent routing control method as described in the present application is implemented.

[0078] According to the fourth aspect of the embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute a single-agent routing control method described in the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 : A schematic flow chart of a single-agent routing control method shown in certain embodiments of the present application;

[0080] Figure 2 : A module structure diagram of a single-agent routing control device shown in certain embodiments of the present application. DETAILED DESCRIPTION

[0081] The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below in conjunction with the accompanying drawings are exemplary and are only used to explain some embodiments of the present application and should not be understood as limiting the embodiments of the present application. Based on the embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0082] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly indicate the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of this application, unless otherwise clearly specified, "multiple" and "several" mean two or more.

[0083] The existing deep reinforcement learning routing optimization methods based on AI-constructed intelligent agents include: first, a service quality-aware routing algorithm that applies deep reinforcement learning in a knowledge-defined network, in which the intelligent agent determines the best strategy through interactive learning of the domain data network and maximization of the expected reward function, thereby optimizing the routing configuration; second, maximizing the reward function common to all routes by jointly learning the network environment and dynamic changes; third, a two-stage joint routing selection and network resource scheduling algorithm based on reinforcement learning; fourth, a mobile ad hoc network routing protocol based on a deep Q network DQN, which determines the node cost by calculating the node's residual energy and moving speed, and performs routing control based on the Q value of each node calculated by DQN. All of the above methods can optimize routing to a certain extent, but in large-scale routing scenarios, all of the above methods are prone to unreachable routing. Therefore, how to avoid the routing control method from generating unreachable routing is still a technical problem that needs to be solved urgently in the existing technology.

[0084] Based on the above technical background, please refer to Figure 1 The present application embodiment provides a single-agent routing control method, including steps S101 to S103, each of which is as follows:

[0085] Step S101: constructing a single intelligent agent of the network to be controlled according to the node topology matrix of the network to be controlled.

[0086] In certain embodiments of the present application, constructing a single agent of the network to be controlled according to the node topology matrix of the network to be controlled specifically includes:

[0087] Obtain the node topology, node location and node traffic of each routing node in the network to be controlled;

[0088] Constructing the node topology matrix according to the node topology, the node location, and the node traffic; wherein the node topology matrix includes node weights of each routing node in the network to be controlled; and the node weights are calculated based on the node topology;

[0089] According to the node topology matrix, a single intelligent agent of the network to be controlled is constructed.

[0090] This application first obtains the node data of each routing node in the network to be controlled, then constructs a node topology matrix with node weights, and then constructs a single intelligent agent, which facilitates the subsequent iteration of the routing optimization path output by the single intelligent agent and is more in line with the actual routing optimization.

[0091] In certain embodiments of the present application, the node traffic includes node inbound traffic and node inbound bandwidth utilization.

[0092] Specifically, suppose the network to be controlled has routing nodes, then the node topology matrix is ​​constructed according to the node topology, the node position and the node traffic, specifically as follows:

[0093] The node position includes a node abscissa and a node ordinate;

[0094] According to the node inbound bandwidth utilization, node inbound traffic, node abscissa and node ordinate of all routing nodes, a node information matrix is ​​constructed; wherein the dimension of the node information matrix is ;

[0095] Input the node topology, the node information matrix and the path performance index into a preset network model, and obtain the node weight of each routing node through dimensionality reduction coding;

[0096] The node topology matrix is ​​constructed according to the node topology, the node information matrix and the node weight of each routing node.

[0097] Step S102: Based on a preset neural network framework, the routing optimization path output by the single agent is iterated until a preset iteration round is reached, and the routing optimization path of the last iteration is output; wherein, in each round of iteration, the routing optimization path is gradually generated based on the neural network framework through deep reinforcement learning; in each step of generation, the next hop routing node is selected based on a sorting mechanism to update the routing optimization path.

[0098] Specifically, in the current routing optimization path, the path length is expressed as , the sum of the inbound bandwidth utilization of all routing nodes is expressed as , the sum of the inflow of all routing nodes is expressed as , then the state space of the single agent is expressed as ; The node weight of each routing node is expressed as , and serves as the action space of the single agent.

[0099] In certain embodiments of the present application, the neural network framework includes but is not limited to a DQN network, a DDPG network, and a TD3 network, with the TD3 network being a preferred embodiment. Specifically, the neural network framework includes an action network, two evaluation networks, a target action network, and two target evaluation networks.

[0100] Specifically, in each round of training of the neural network framework, the following steps are performed:

[0101] Step 1: Initialize the parameters of the action network Actor And the parameters of the evaluation network Critic , and copy the parameters of the Actor network and the Critic network to the target action network TargetActor and the target evaluation network TargetCritic, that is ; Initialize the replay buffer capacity to ; Initialize discount factor , learning rate , soft update coefficient , maximum step length and the maximum number of rounds .

[0102] Step 2: Based on the single agent and neural network framework, the routing optimization path is updated step by step. In each step of training, specifically:

[0103] Step 2.1: Input the current state of the single agent into the Actor network to get the current action:

[0104]

[0105] in, is the output of the Actor network, is the network parameter of the Actor network, is additive noise, is the standard deviation of the normal distribution of additive noise, are the minimum and maximum values ​​of the output action, Indicates that Crop to Specifically, the additive noise Actions generated by the perturbation policy network.

[0106] Step 2.2: Based on the current action Get the corresponding current routing node, and select the next hop routing node of the current routing node based on the sorting mechanism. According to the next hop routing node of the current routing node, get the next state (update state) , and according to the next state Calculate the reward value of the current action .

[0107] The next hop routing node of the current routing node is selected based on the sorting mechanism, specifically:

[0108] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0109] Obtaining an adjacent node distance sorting table and an adjacent node weight sorting table based on descending sorting according to the adjacent node target distance list and the adjacent node weight list;

[0110] Adding the adjacent node distance ranking table and the adjacent node weight ranking table item by item to obtain an adjacent node comprehensive ranking table;

[0111] According to the comprehensive ranking table of adjacent nodes, the next hop routing node of the current routing node is selected.

[0112] For example, if the adjacent routing node list of the current routing node 0 is , the corresponding adjacent node target distance list is , then the adjacent node distance sorting table is ; The corresponding adjacent node weight list is , then the adjacent node weight sorting table is At this time, the comprehensive ranking table of adjacent nodes is According to the comprehensive ranking table of adjacent nodes, the next hop routing node of the current routing node 0 should be routing 2.

[0113] Step 2.3: Set the reward value , Current Action , current status and the next state Stored in the replay buffer. When updating the parameters of the neural network framework, the total number of samples randomly extracted from the replay buffer is The target action is calculated through the TargetActor network:

[0114]

[0115] in, is the output of the TargetActor network, is the network parameter of the TargetActor network, is additive noise, is the standard deviation of the normal distribution of additive noise. Specifically, the additive noise Actions generated by the target policy network to perturb them.

[0116] Based on the target action, the minimum value of the two target evaluation network outputs is selected to calculate the target value:

[0117]

[0118] in, is the reward value, is the discount factor, For the The expected cumulative reward of a TargetCritic network under the updated state and the action corresponding to the updated state, For the The network parameters of the TargetCritic network.

[0119] Specifically, whether a "dead route" is generated is determined by a detection function. If the detection function detects that there are no duplicate nodes on the current route optimization path, then there is no "dead route"; if the detection function detects that there are duplicate nodes on the current route optimization path, then there is a "dead route".

[0120] Step 2.4: Use the gradient descent method to update the network parameters of the Critic network to minimize the loss function:

[0121]

[0122] in, is the sample set, They are the current state, current action, reward value and updated state respectively. For the The expected cumulative reward of an evaluation network under the current state and current action, For the The network parameters of the evaluation network.

[0123] Since the update frequency of the Target network and the Actor network is lower than that of the Critic network, the gradient ascent method is used to update the network parameters of the Actor network every time the number of training steps reaches the preset generation step threshold:

[0124]

[0125] in, is the expected cumulative reward of the first evaluation network under the current state and current action, is the output of the action network in the current state, are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0126] At the same time, based on the soft update strategy, update the Target network:

[0127]

[0128] In certain embodiments of the present application, the stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes:

[0129] Inputting the initial state of the single agent into the neural network framework to obtain an initial action;

[0130] generating an initial routing optimization path according to the initial state and the initial action;

[0131] Through deep reinforcement learning, the initial route optimization path is iteratively updated step by step based on the neural network framework to generate a route optimization path; wherein, in each step of generation, the updated state of the single agent is obtained according to the current state of the single agent, and the network parameters of the initial route optimization path and the neural network framework are updated according to the updated state; in the first step of generation, the current state of the single agent is the initial state; in the last step of generation, the updated state of the single agent corresponds to the route optimization path.

[0132] This application first inputs the initial state of a single agent into a neural network framework to obtain an initial action, then generates an initial routing optimization path, and gradually iterates and updates the initial routing optimization path through deep reinforcement learning to generate a routing optimization path. In each step of generation, an updated state is obtained according to the current state of the single agent, and the network parameters of the initial routing optimization path and the neural network framework are updated according to the updated state. By gradually iterating and updating the path through deep reinforcement learning, it can be ensured that the path continuously approaches the target routing node from the starting routing node, and the network parameters of the neural network framework are updated at each step, which can timely update the network parameters and accelerate the convergence speed of the framework.

[0133] In certain embodiments of the present application, obtaining an updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state, specifically includes:

[0134] According to the current state of the single agent, obtaining the current action of the single agent based on the neural network framework;

[0135] Selecting a next-hop routing node of a current routing node based on a ranking mechanism; wherein the current routing node corresponds to a current action of the single agent;

[0136] Obtaining the updated state of the single agent according to the next-hop routing node of the current routing node;

[0137] The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

[0138] This application first obtains the current action based on the current state of the single intelligent agent based on the neural network framework, and then selects the next-hop routing node of the current routing node corresponding to the current action according to the sorting mechanism. It can reasonably select the next-hop routing node to avoid the generation of paths with unreachable routes, and then obtain the updated state according to the next-hop routing node, and update the initial routing optimization path and the network parameters of the neural network framework according to the updated state, which can ensure that the path continuously approaches the target routing node from the starting routing node, and can update the network parameters in time, thereby increasing the convergence speed of the framework.

[0139] In certain embodiments of the present application, selecting the next-hop routing node of the current routing node based on the sorting mechanism specifically includes:

[0140] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0141] Obtaining an adjacent node distance sorting table and an adjacent node weight sorting table based on descending sorting according to the adjacent node target distance list and the adjacent node weight list;

[0142] According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next-hop routing node of the current routing node is selected.

[0143] The present application first obtains an adjacent node target distance list and an adjacent node weight list corresponding to the adjacent routing node list of the current routing node, then obtains a corresponding sorting table based on descending sorting, and obtains an adjacent node comprehensive sorting table based on the sorting table, and then selects the next-hop routing node of the current routing node. Through the target distance list and weight list corresponding to the adjacent routing node list of the current routing node, the potential next-hop routing nodes of the current routing node can be reasonably quantified, and then when performing comprehensive sorting to select the next-hop routing node, the selected next-hop routing node can be made more reasonable, avoiding the generation of paths with unreachable routes.

[0144] In certain embodiments of the present application, updating the network parameters of the neural network framework according to the update state specifically includes:

[0145] Calculating a reward value of the current state according to the current state and the updated state;

[0146] According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on the gradient descent algorithm;

[0147] The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically:

[0148] ;

[0149] in, is the sample set, They are the current state, current action, reward value and updated state respectively. For the The expected cumulative reward of an evaluation network under the current state and current action, For the The network parameters of the evaluation network, is the target value, is the discount factor, For the The target evaluation network is the expected cumulative reward under the updated state and the action corresponding to the updated state, For the The network parameters of the target evaluation network are used.

[0150] This application first calculates the reward value based on the current state and the updated state, and then updates the network parameters of the neural network framework based on the gradient descent algorithm, which can update the corresponding network so that it converges to the range required by the routing optimization task.

[0151] In certain embodiments of the present application, the stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate the route optimization path further includes:

[0152] When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state;

[0153] The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically:

[0154] ;

[0155] in, is the sample set, is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, is the output of the action network in the current state, are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0156] When the number of generated steps reaches a preset generation step threshold, the present application updates the network parameters of the neural network framework based on the gradient ascent algorithm according to the current state, and can update the corresponding network so that it converges to the range required by the routing optimization task.

[0157] Step S103: Control each routing node of the network to be controlled according to the routing optimization path of the last iteration.

[0158] This application first constructs a single intelligent agent based on the node topology matrix of the network to be controlled, and then iterates the routing optimization path output by the single intelligent agent based on the neural network framework. In each round of iteration, the routing optimization path is gradually generated through deep reinforcement learning, which can enable the routing optimization path to continuously approach the target routing node from the starting routing node, thereby completing the task. In each step of generation, the next-hop routing node is selected based on the sorting mechanism to update the routing optimization path. The next-hop routing node is selected through the sorting mechanism, which can reasonably select the next-hop routing node and avoid the generation of paths with unreachable routes. Therefore, when controlling the routing nodes of the network to be controlled according to the routing optimization path of the last iteration, routing control can be performed more reasonably.

[0159] Corresponding to the above method, see Figure 2 , the embodiment of the present application provides a single-agent routing control device, including a single-agent construction module 210, a path optimization module 220 and a routing control module 230;

[0160] The single agent construction module 210 is used to construct a single agent of the network to be controlled according to the node topology matrix of the network to be controlled;

[0161] The path optimization module 220 is configured to iterate the route optimization path output by the single agent based on a preset neural network framework until a preset number of iterations is reached, and output the route optimization path of the final iteration; wherein, during each iteration, the route optimization path is gradually generated based on the neural network framework through deep reinforcement learning; during each generation step, the next hop routing node is selected based on a sorting mechanism to update the route optimization path;

[0162] The routing control module 230 is configured to control each routing node of the network to be controlled according to the routing optimization path of the last iteration.

[0163] In certain embodiments of the present application, the single agent construction module 210 includes a node data acquisition unit, a topology matrix construction unit, and a single agent construction unit;

[0164] The node data acquisition unit is used to obtain the node topology, node location and node traffic of each routing node in the network to be controlled;

[0165] The topology matrix construction unit is configured to construct the node topology matrix according to the node topology, the node position, and the node traffic; wherein the node topology matrix includes node weights of each routing node in the network to be controlled; and the node weights are calculated based on the node topology;

[0166] The single agent construction unit is used to construct a single agent of the network to be controlled according to the node topology matrix.

[0167] In certain embodiments of the present application, the stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes:

[0168] Inputting the initial state of the single agent into the neural network framework to obtain an initial action;

[0169] generating an initial routing optimization path according to the initial state and the initial action;

[0170] Through deep reinforcement learning, the initial route optimization path is iteratively updated step by step based on the neural network framework to generate a route optimization path; wherein, in each step of generation, the updated state of the single agent is obtained according to the current state of the single agent, and the network parameters of the initial route optimization path and the neural network framework are updated according to the updated state; in the first step of generation, the current state of the single agent is the initial state; in the last step of generation, the updated state of the single agent corresponds to the route optimization path.

[0171] In certain embodiments of the present application, obtaining an updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state, specifically includes:

[0172] According to the current state of the single agent, obtaining the current action of the single agent based on the neural network framework;

[0173] Selecting a next-hop routing node of a current routing node based on a ranking mechanism; wherein the current routing node corresponds to a current action of the single agent;

[0174] Obtaining the updated state of the single agent according to the next-hop routing node of the current routing node;

[0175] The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

[0176] In certain embodiments of the present application, selecting the next-hop routing node of the current routing node based on the sorting mechanism specifically includes:

[0177] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0178] Obtaining an adjacent node distance sorting table and an adjacent node weight sorting table based on descending sorting according to the adjacent node target distance list and the adjacent node weight list;

[0179] According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next-hop routing node of the current routing node is selected.

[0180] In certain embodiments of the present application, updating the network parameters of the neural network framework according to the update state specifically includes:

[0181] Calculating a reward value of the current state according to the current state and the updated state;

[0182] According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on the gradient descent algorithm;

[0183] The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically:

[0184] ;

[0185] in, is the sample set, They are the current state, current action, reward value and updated state respectively. For the The expected cumulative reward of an evaluation network under the current state and current action, For the The network parameters of the evaluation network, is the target value, is the discount factor, For the The target evaluation network is the expected cumulative reward under the updated state and the action corresponding to the updated state, For the The network parameters of the target evaluation network are used.

[0186] In certain embodiments of the present application, the stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate the route optimization path further includes:

[0187] When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state;

[0188] The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically:

[0189] ;

[0190] in, is the sample set, is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, is the output of the action network in the current state, are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0191] This application first constructs a single intelligent agent based on the node topology matrix of the network to be controlled, and then iterates the routing optimization path output by the single intelligent agent based on the neural network framework. In each round of iteration, the routing optimization path is gradually generated through deep reinforcement learning, which can enable the routing optimization path to continuously approach the target routing node from the starting routing node, thereby completing the task. In each step of generation, the next-hop routing node is selected based on the sorting mechanism to update the routing optimization path. The next-hop routing node is selected through the sorting mechanism, which can reasonably select the next-hop routing node and avoid the generation of paths with unreachable routes. Therefore, when controlling the routing nodes of the network to be controlled according to the routing optimization path of the last iteration, routing control can be performed more reasonably.

[0192] It should be understood that the device provided in the embodiments of the present application corresponds to the aforementioned method, and the single-agent routing control device provided in the embodiments of the present application can implement the single-agent routing control method provided in any embodiment of the present application.

[0193] Adaptively, the embodiments of the present application further provide a computer device and a computer-readable storage medium.

[0194] The computer device comprises: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor;

[0195] Wherein, when the processor executes the computer program, a single-agent routing control method of the present application is implemented.

[0196] The computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute a single-agent routing control method of the present application.

[0197] The above description is a partial embodiment of the present application, which further describes the purpose, technical solutions, and beneficial effects of the present application in detail. It should be understood that the above description of the partial embodiment of the present application is not to be construed as limiting the present application. In particular, it is pointed out that for those skilled in the art, any changes, modifications, equivalent substitutions, and variations made within the spirit and principles of the present application should be included within the scope of protection of the present application.

Claims

1. A single agent routing control method, characterized in that: include: Construct a single intelligent agent of the network to be controlled according to the node topology matrix of the network to be controlled; Based on a preset neural network framework, the routing optimization path output by the single agent is iterated until a preset number of iterations is reached, and the routing optimization path of the final iteration is output; wherein, during each iteration, the routing optimization path is gradually generated based on the neural network framework through deep reinforcement learning; during each generation step, the next hop routing node is selected based on a sorting mechanism to update the routing optimization path; According to the routing optimization path of the last iteration, each routing node of the control network is controlled; The step of constructing a single intelligent agent of the network to be controlled according to the node topology matrix of the network to be controlled specifically includes: Obtaining the node topology, node location, and node traffic of each routing node in the network to be controlled; constructing the node topology matrix based on the node topology, the node location, and the node traffic; wherein the node topology matrix includes the node weight of each routing node in the network to be controlled; the node weight is calculated based on the node topology; and constructing a single intelligent agent of the network to be controlled based on the node topology matrix; The stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes: The initial state of the single agent is input into the neural network framework to obtain an initial action; an initial routing optimization path is generated based on the initial state and the initial action; the initial routing optimization path is updated step by step through deep reinforcement learning based on the neural network framework to generate a routing optimization path; wherein, in each generation step, the updated state of the single agent is obtained according to the current state of the single agent, and the network parameters of the initial routing optimization path and the neural network framework are updated according to the updated state; in the first generation step, the current state of the single agent is the initial state; in the last generation step, the updated state of the single agent corresponds to the routing optimization path.

2. A single agent routing control method according to claim 1, characterized in that: The step of obtaining an updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state, specifically includes: According to the current state of the single agent, obtaining the current action of the single agent based on the neural network framework; Selecting a next-hop routing node of a current routing node based on a ranking mechanism; wherein the current routing node corresponds to a current action of the single agent; Obtaining the updated state of the single agent according to the next-hop routing node of the current routing node; The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

3. A single agent routing control method according to claim 2, characterized in that: The selecting of the next hop routing node of the current routing node based on the sorting mechanism specifically includes: A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent; Obtaining an adjacent node distance sorting table and an adjacent node weight sorting table based on descending sorting according to the adjacent node target distance list and the adjacent node weight list; According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next-hop routing node of the current routing node is selected.

4. A single agent routing control method according to claim 2, characterized in that: Updating the network parameters of the neural network framework according to the update state specifically includes: Calculating a reward value of the current state according to the current state and the updated state; According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on the gradient descent algorithm; The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically: ； in, is the sample set, They are the current state, current action, reward value and updated state respectively. For the The expected cumulative reward of an evaluation network under the current state and current action, For the The network parameters of the evaluation network, is the target value, is the discount factor, For the The target evaluation network is the expected cumulative reward under the updated state and the action corresponding to the updated state, For the The network parameters of the target evaluation network are used.

5. A single agent routing control method according to claim 2, characterized in that: The step of gradually iteratively updating the initial route optimization path based on the neural network framework through deep reinforcement learning to generate a route optimization path further includes: When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state; The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically: ； in, is the sample set, is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, is the output of the action network in the current state, are the network parameters of the action network, is the gradient of the network parameters of the action network.

6. A single-agent routing control device, characterized in that: It includes single agent building module, path optimization module and routing control module; The single agent construction module is used to construct a single agent of the network to be controlled according to the node topology matrix of the network to be controlled; The path optimization module is configured to iterate the route optimization path output by the single agent based on a preset neural network framework until a preset number of iterations is reached, and output the route optimization path of the final iteration; wherein, during each iteration, the route optimization path is gradually generated based on the neural network framework through deep reinforcement learning; during each generation step, the next hop routing node is selected based on a sorting mechanism to update the route optimization path; The routing control module is used to control each routing node of the network to be controlled according to the routing optimization path of the last iteration; The single agent construction module includes a node data acquisition unit, a topology matrix construction unit and a single agent construction unit; The node data acquisition unit is used to acquire the node topology, node location, and node traffic of each routing node in the network to be controlled; the topology matrix construction unit is used to construct the node topology matrix based on the node topology, the node location, and the node traffic; wherein the node topology matrix includes the node weight of each routing node in the network to be controlled; the node weight is calculated based on the node topology; the single agent construction unit is used to construct a single agent of the network to be controlled based on the node topology matrix; The stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes: The initial state of the single agent is input into the neural network framework to obtain an initial action; an initial routing optimization path is generated based on the initial state and the initial action; the initial routing optimization path is updated step by step through deep reinforcement learning based on the neural network framework to generate a routing optimization path; wherein, in each generation step, the updated state of the single agent is obtained according to the current state of the single agent, and the network parameters of the initial routing optimization path and the neural network framework are updated according to the updated state; in the first generation step, the current state of the single agent is the initial state; in the last generation step, the updated state of the single agent corresponds to the routing optimization path.

7. A single-agent routing control device according to claim 6, characterized in that: The step of obtaining an updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state, specifically includes: According to the current state of the single agent, obtaining the current action of the single agent based on the neural network framework; Selecting a next-hop routing node of a current routing node based on a ranking mechanism; wherein the current routing node corresponds to a current action of the single agent; Obtaining the updated state of the single agent according to the next-hop routing node of the current routing node; The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

8. A single-agent routing control device according to claim 7, characterized in that: The selecting of the next hop routing node of the current routing node based on the sorting mechanism specifically includes: A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent; Obtaining an adjacent node distance sorting table and an adjacent node weight sorting table based on descending sorting according to the adjacent node target distance list and the adjacent node weight list; According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next-hop routing node of the current routing node is selected.

9. A single-agent routing control device according to claim 7, characterized in that: Updating the network parameters of the neural network framework according to the update state specifically includes: Calculating a reward value of the current state according to the current state and the updated state; According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on the gradient descent algorithm; The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically: ； in, is the sample set, They are the current state, current action, reward value and updated state respectively. For the The expected cumulative reward of an evaluation network under the current state and current action, For the The network parameters of the evaluation network, is the target value, is the discount factor, For the The target evaluation network is the expected cumulative reward under the updated state and the action corresponding to the updated state, For the The network parameters of the target evaluation network are used.

10. A single-agent routing control device according to claim 7, characterized in that: The step of gradually iteratively updating the initial route optimization path based on the neural network framework through deep reinforcement learning to generate a route optimization path further includes: When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state; The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically: ； in, is the sample set, is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, is the output of the action network in the current state, are the network parameters of the action network, is the gradient of the network parameters of the action network.

11. A computer device, characterized in that: include: processor; Memory; a computer program stored in the memory and configured to be executed by the processor; When the processor executes the computer program, it implements a single-agent routing control method as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, which are suitable for loading by a processor to execute a single-agent routing control method as described in any one of claims 1 to 5.

Images