Single-agent routing control method, device, equipment and medium

Abstract

The invention discloses a single-agent routing control method, device and equipment and a medium, and relates to the field of routing control optimization, and the method comprises the steps: constructing a single agent of a to-be-controlled network according to a node topology matrix of the to-be-controlled network; on the basis of a preset neural network framework, iterating the route optimization path output by the single agent until a preset iteration round is reached, and outputting the route optimization path of the last iteration; wherein during each iteration, a routing optimization path is gradually generated based on the neural network framework through deep reinforcement learning; in each step of generation, a next-hop routing node is selected based on a sorting mechanism so as to update the routing optimization path; and controlling each routing node of the network to be controlled according to the finally iterated routing optimization path. According to the method and the device, the technical problem that an existing route control method is easy to generate an unreachable route can be solved.

Description

Technical Field

[0001] The present application relates to the field of routing control optimization, and in particular to a single-agent routing control method, device, equipment and medium. Background Art

[0002] The distribution communication network is one of the cores of the power system, responsible for information transmission and communication between different power nodes. With the increasing scale of the power grid, the maintenance method of manual detection is no longer competent for the stable operation of the distribution communication network, and a more effective distribution communication network maintenance method is needed. In recent years, the maintenance method of intelligently adjusting the flow of the distribution communication network and planning the route of nodes through AI algorithms has become the focus of research, which can maintain the distribution communication network efficiently and with high quality.

[0003] In the prior art, the main method of maintaining the distribution communication network through AI algorithms is deep reinforcement learning routing optimization based on AI-constructed intelligent agents. Among them, several prominent studies are: first, the service quality-aware routing algorithm using deep reinforcement learning in knowledge-defined networks, in which the intelligent agent determines the best strategy through interactive learning of domain data networks and maximizing the expected reward function, thereby optimizing the routing configuration; second, by jointly learning the network environment and dynamic changes, maximizing the reward function universal to all routes; third, a two-stage joint routing selection and network resource scheduling algorithm based on reinforcement learning, which can solve the transmission reliability and efficiency problems of large-scale mesh networks; fourth, a mobile ad hoc network routing protocol based on deep Q-network (DQN), which determines the node cost by calculating the residual energy and moving speed of the node, and performs routing control based on the Q value of each node calculated by DQN, which can reduce control overhead and improve data packet transmission rate. The above algorithms can optimize routing to a certain extent, thereby achieving the effect of power grid maintenance, but the above algorithms are prone to unreachable routing in large-scale routing scenarios. Summary of the invention

[0004] The present application provides a single-agent routing control method, device, equipment and medium to solve the technical problem that existing routing control methods are prone to produce unreachable routes.

[0005] According to a first aspect of the embodiments of the present application, a single-agent routing control method is provided, comprising:

[0006] According to the node topology matrix of the network to be controlled, a single intelligent agent of the network to be controlled is constructed;

[0007] Based on a preset neural network framework, the route optimization path output by the single agent is iterated until a preset iteration round is reached, and the route optimization path of the last iteration is output; wherein, in each iteration round, the route optimization path is gradually generated based on the neural network framework through deep reinforcement learning; in each step of generation, the next hop routing node is selected based on a sorting mechanism to update the route optimization path;

[0008] According to the routing optimization path of the last iteration, each routing node of the network to be controlled is controlled.

[0009] The present application first constructs a single intelligent agent according to the node topology matrix of the network to be controlled, and then iterates the routing optimization path output by the single intelligent agent based on the neural network framework. In each round of iteration, the routing optimization path is gradually generated through deep reinforcement learning, which can make the routing optimization path continuously approach the target routing node from the starting routing node, thereby completing the task, and in each step of generation, the next-hop routing node is selected based on the sorting mechanism to update the routing optimization path. The next-hop routing node is selected by the sorting mechanism, and the next-hop routing node can be reasonably selected to avoid the generation of paths with unreachable routes. Then, when controlling each routing node of the network to be controlled according to the routing optimization path of the last iteration, routing control can be performed more reasonably.

[0010] In certain embodiments of the present application, constructing a single agent of the network to be controlled according to the node topology matrix of the network to be controlled specifically includes:

[0011] Obtain the node topology, node location and node traffic of each routing node in the network to be controlled;

[0012] According to the node topology, the node position and the node traffic, the node topology matrix is ​​constructed; wherein the node topology matrix includes the node weight of each routing node in the network to be controlled; the node weight is calculated based on the node topology;

[0013] According to the node topology matrix, a single intelligent agent of the network to be controlled is constructed.

[0014] This application first obtains the node data of each routing node in the network to be controlled, then constructs a node topology matrix with node weights, and then constructs a single intelligent agent, which facilitates the subsequent iteration of the routing optimization path output by the single intelligent agent and is more in line with the actual routing optimization.

[0015] In certain embodiments of the present application, the stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes:

[0016] Inputting the initial state of the single agent into the neural network framework to obtain an initial action;

[0017] generating an initial routing optimization path according to the initial state and the initial action;

[0018] Through deep reinforcement learning, the initial routing optimization path is iteratively updated step by step based on the neural network framework to generate a routing optimization path; wherein, in each generation step, the updated state of the single agent is obtained according to the current state of the single agent, and the initial routing optimization path and the network parameters of the neural network framework are updated according to the updated state; in the first generation step, the current state of the single agent is the initial state; in the last generation step, the updated state of the single agent corresponds to the routing optimization path.

[0019] This application first inputs the initial state of a single agent into a neural network framework to obtain an initial action, then generates an initial routing optimization path, and gradually iterates and updates the initial routing optimization path through deep reinforcement learning to generate a routing optimization path. In each step of generation, an updated state is obtained according to the current state of the single agent, and the network parameters of the initial routing optimization path and the neural network framework are updated according to the updated state. By gradually iterating and updating the path through deep reinforcement learning, it can be ensured that the path continuously approaches the target routing node from the starting routing node, and the network parameters of the neural network framework are updated at each step, so that the network parameters can be updated in time and the framework convergence speed can be accelerated.

[0020] In certain embodiments of the present application, obtaining the updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state specifically include:

[0021] According to the current state of the single agent, the current action of the single agent is obtained based on the neural network framework;

[0022] Selecting a next-hop routing node of a current routing node based on a sorting mechanism; wherein the current routing node corresponds to a current action of the single agent;

[0023] According to the next-hop routing node of the current routing node, the update state of the single agent is obtained;

[0024] The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

[0025] This application is first based on the neural network framework, obtains the current action according to the current state of the single intelligent agent, and then selects the next-hop routing node of the current routing node corresponding to the current action according to the sorting mechanism, which can reasonably select the next-hop routing node to avoid the generation of paths with unreachable routes, and then obtains the updated state according to the next-hop routing node, and updates the initial routing optimization path and the network parameters of the neural network framework according to the updated state, which can ensure that the path continuously approaches the target routing node from the starting routing node, and can update the network parameters in time, thereby increasing the convergence speed of the framework.

[0026] In certain embodiments of the present application, selecting the next hop routing node of the current routing node based on the sorting mechanism specifically includes:

[0027] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0028] According to the adjacent node target distance list and the adjacent node weight list, based on descending sorting, an adjacent node distance sorting table and an adjacent node weight sorting table are obtained;

[0029] According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next hop routing node of the current routing node is selected.

[0030] The present application first obtains an adjacent node target distance list and an adjacent node weight list corresponding to the adjacent routing node list of the current routing node, then obtains a corresponding sorting table based on descending sorting, and obtains an adjacent node comprehensive sorting table based on the sorting table, and then selects the next-hop routing node of the current routing node. Through the target distance list and weight list corresponding to the adjacent routing node list of the current routing node, the potential next-hop routing nodes of the current routing node can be reasonably quantified, and then when the next-hop routing node is selected by comprehensive sorting, the selected next-hop routing node can be made more reasonable to avoid the generation of paths with unreachable routes.

[0031] In certain embodiments of the present application, updating the network parameters of the neural network framework according to the update state specifically includes:

[0032] Calculating a reward value of the current state according to the current state and the updated state;

[0033] According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on a gradient descent algorithm;

[0034] The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically:

[0035]

[0036] Among them, N B is the sample set, s t , a t , r t ,s t+1 They are the current state, current action, reward value and updated state respectively. is the expected cumulative reward of the i-th evaluation network under the current state and current action, is the network parameter of the i-th evaluation network, is the target value, γ is the discount factor, For the i The target evaluation network is the expected cumulative reward under the updated state and the action corresponding to the updated state, For the i The network parameters of the target evaluation network.

[0037] This application first calculates the reward value based on the current state and the updated state, and then updates the network parameters of the neural network framework based on the gradient descent algorithm, which can update the corresponding network so that it converges to the range required for the routing optimization task.

[0038] In certain embodiments of the present application, the stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate the route optimization path further includes:

[0039] When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state;

[0040] The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically:

[0041]

[0042] Among them, N B is the sample set, s t , a t is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, μ(s t ,θ μ ) is the output of the action network in the current state, θ μ are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0043] When the number of generated steps reaches a preset generation step threshold, the present application updates the network parameters of the neural network framework based on the gradient ascent algorithm according to the current state, and can update the corresponding network so that it converges to the range required for the routing optimization task.

[0044] According to a second aspect of the implementation of the present application, there is provided a single-agent routing control device, comprising a single-agent construction module, a path optimization module and a routing control module;

[0045] The single agent construction module is used to construct a single agent of the network to be controlled according to the node topology matrix of the network to be controlled;

[0046] The routing optimization module is used to iterate the routing optimization path output by the single agent based on a preset neural network framework until a preset iteration round is reached, and output the routing optimization path of the last iteration; wherein, in each iteration round, the routing optimization path is gradually generated based on the neural network framework through deep reinforcement learning; in each step of generation, the next hop routing node is selected based on a sorting mechanism to update the routing optimization path;

[0047] The routing control module is used to control each routing node of the network to be controlled according to the routing optimization path of the last iteration.

[0048] In certain embodiments of the present application, the single agent construction module includes a node data acquisition unit, a topology matrix construction unit, and a single agent construction unit;

[0049] The node data acquisition unit is used to acquire the node topology, node location and node flow of each routing node in the network to be controlled;

[0050] The topology matrix construction unit is used to construct the node topology matrix according to the node topology, the node position and the node traffic; wherein the node topology matrix includes the node weight of each routing node in the network to be controlled; the node weight is calculated based on the node topology;

[0051] The single agent construction unit is used to construct a single agent of the network to be controlled according to the node topology matrix.

[0052] In certain embodiments of the present application, the stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes:

[0053] Inputting the initial state of the single agent into the neural network framework to obtain an initial action;

[0054] generating an initial routing optimization path according to the initial state and the initial action;

[0055] Through deep reinforcement learning, the initial routing optimization path is iteratively updated step by step based on the neural network framework to generate a routing optimization path; wherein, in each generation step, the updated state of the single agent is obtained according to the current state of the single agent, and the initial routing optimization path and the network parameters of the neural network framework are updated according to the updated state; in the first generation step, the current state of the single agent is the initial state; in the last generation step, the updated state of the single agent corresponds to the routing optimization path.

[0056] In certain embodiments of the present application, obtaining the updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state specifically include:

[0057] According to the current state of the single agent, the current action of the single agent is obtained based on the neural network framework;

[0058] Selecting a next-hop routing node of a current routing node based on a sorting mechanism; wherein the current routing node corresponds to a current action of the single agent;

[0059] According to the next-hop routing node of the current routing node, the update state of the single agent is obtained;

[0060] The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

[0061] In certain embodiments of the present application, selecting the next hop routing node of the current routing node based on the sorting mechanism specifically includes:

[0062] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0063] According to the adjacent node target distance list and the adjacent node weight list, based on descending sorting, an adjacent node distance sorting table and an adjacent node weight sorting table are obtained;

[0064] According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next hop routing node of the current routing node is selected.

[0065] In certain embodiments of the present application, updating the network parameters of the neural network framework according to the update state specifically includes:

[0066] Calculating a reward value of the current state according to the current state and the updated state;

[0067] According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on a gradient descent algorithm;

[0068] The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically:

[0069]

[0070] Among them, N B is the sample set, s t , a t , r t ,s t+1 They are the current state, current action, reward value and updated state respectively. is the expected cumulative reward of the i-th evaluation network under the current state and current action, is the network parameter of the i-th evaluation network, is the target value, γ is the discount factor, Evaluate the expected cumulative reward of the network for the i-th target under the updated state and the action corresponding to the updated state, Evaluate the network parameters of the network for the i-th target.

[0071] In certain embodiments of the present application, the stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate the route optimization path further includes:

[0072] When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state;

[0073] The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically:

[0074]

[0075] Among them, N B is the sample set, s t , a t is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, μ(s t ,θ μ) is the output of the action network in the current state, θ μ are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0076] The present application first constructs a single intelligent agent according to the node topology matrix of the network to be controlled, and then iterates the routing optimization path output by the single intelligent agent based on the neural network framework. In each round of iteration, the routing optimization path is gradually generated through deep reinforcement learning, which can make the routing optimization path continuously approach the target routing node from the starting routing node, thereby completing the task, and in each step of generation, the next-hop routing node is selected based on the sorting mechanism to update the routing optimization path. The next-hop routing node is selected by the sorting mechanism, and the next-hop routing node can be reasonably selected to avoid the generation of paths with unreachable routes. Then, when controlling each routing node of the network to be controlled according to the routing optimization path of the last iteration, routing control can be performed more reasonably.

[0077] According to a third aspect of the implementation scheme of the present application, a computer device is provided, comprising: a processor; a memory; a computer program stored in the memory and configured to be executed by the processor; wherein when the processor executes the computer program, a single-agent routing control method as described in the present application is implemented.

[0078] According to a fourth aspect of the implementation of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute a single-agent routing control method described in the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 : A flow chart of a single-agent routing control method shown in certain embodiments of the present application;

[0080] Figure 2 : A module structure diagram of a single-agent routing control device shown in certain embodiments of the present application. DETAILED DESCRIPTION

[0081] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below in conjunction with the accompanying drawings are exemplary and are only used to explain some embodiments of the present application, and should not be construed as limitations on the embodiments of the present application. Based on the embodiments shown in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0082] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, unless otherwise clearly and specifically defined, "multiple" and "several" mean two or more.

[0083] The deep reinforcement learning routing optimization method based on AI constructed intelligent agent in the prior art includes: first, a service quality-aware routing algorithm using deep reinforcement learning in a knowledge-defined network, in which the intelligent agent determines the best strategy through interactive learning of the domain data network and maximizing the expected reward function, thereby optimizing the routing configuration; second, by jointly learning the network environment and dynamic changes, maximizing the reward function common to all routes; third, a two-stage joint routing selection and network resource scheduling algorithm based on reinforcement learning; fourth, a mobile self-organizing network routing protocol based on a deep Q network DQN, which determines the node cost by calculating the remaining energy and moving speed of the node, and performs routing control based on the Q value of each node calculated by DQN. All of the above methods can optimize the routing to a certain extent, but in the scenario of large-scale routing, all of the above methods are prone to produce unreachable routes. Therefore, how to avoid the routing control method from producing unreachable routes is still a technical problem that needs to be solved urgently in the prior art.

[0084] Based on the above technical background, please refer to Figure 1 The present application embodiment provides a single-agent routing control method, including steps S101 to S103, each of which is as follows:

[0085] Step S101: construct a single intelligent agent of the network to be controlled according to the node topology matrix of the network to be controlled.

[0086] In certain embodiments of the present application, constructing a single agent of the network to be controlled according to the node topology matrix of the network to be controlled specifically includes:

[0087] Obtain the node topology, node location and node traffic of each routing node in the network to be controlled;

[0088] According to the node topology, the node position and the node traffic, the node topology matrix is ​​constructed; wherein the node topology matrix includes the node weight of each routing node in the network to be controlled; the node weight is calculated based on the node topology;

[0089] According to the node topology matrix, a single intelligent agent of the network to be controlled is constructed.

[0090] This application first obtains the node data of each routing node in the network to be controlled, then constructs a node topology matrix with node weights, and then constructs a single intelligent agent, which facilitates the subsequent iteration of the routing optimization path output by the single intelligent agent and is more in line with the actual routing optimization.

[0091] In certain embodiments of the present application, the node traffic includes node inbound traffic and node inbound bandwidth utilization.

[0092] Specifically, suppose the network to be controlled has N routing nodes, then the node topology matrix is ​​constructed according to the node topology, the node position and the node traffic, specifically:

[0093] The node position includes a node abscissa and a node ordinate;

[0094] Constructing a node information matrix according to the node inbound bandwidth utilization, node inbound traffic, node abscissa and node ordinate of all routing nodes; wherein the dimension of the node information matrix is ​​N×4;

[0095] The node topology, the node information matrix and the path performance index are input into a preset network model, and the node weight of each routing node is obtained through dimension reduction coding;

[0096] The node topology matrix is ​​constructed according to the node topology, the node information matrix and the node weight of each routing node.

[0097] Step S102: Based on a preset neural network framework, the routing optimization path output by the single agent is iterated until a preset iteration round is reached, and the routing optimization path of the last iteration is output; wherein, in each round of iteration, the routing optimization path is gradually generated based on the neural network framework through deep reinforcement learning; in each step of generation, the next hop routing node is selected based on a sorting mechanism to update the routing optimization path.

[0098] Specifically, in the current routing optimization path, the path length is represented by D r The sum of the inbound bandwidth utilization of all routing nodes is expressed as R t , the sum of the inflows of all routing nodes is expressed as O t , then the state space of the single agent is expressed as (D r , R t , O t ); The node weight of each routing node is expressed as (I1, I2, ..., I N ) and serves as the action space of the single agent.

[0099] In certain embodiments of the present application, the neural network framework includes but is not limited to a DQN network, a DDPG network and a TD3 network, and the preferred embodiment is a TD3 network. Specifically, the neural network framework includes an action network, two evaluation networks, a target action network and two target evaluation networks.

[0100] Specifically, in each round of training of the neural network framework, specifically:

[0101] Step 1: Initialize the parameters θ of the action network Actor μ and evaluate the parameters of the network Cr it ic The parameters of the Actor network and the Cr it ic network are copied to the target action network Target Actor and the target evaluation network Target Cr it ic, that is, θ μ′ =θ μ , Initialize the replay buffer capacity to N Bmax ; Initialize discount factor γ, learning rate lr, soft update coefficient τ, maximum step size N steps and the maximum number of rounds Ne posode .

[0102] Step 2: Based on the single agent and neural network framework, the routing optimization path is updated step by step. In each step of training, the specific steps are:

[0103] Step 2.1: Input the current state of the single agent into the Actor network to get the current action:

[0104] a t =clip(μ(s t |θ μ )+k,α Low , α High ) where μ(s t |θ μ ) is the output of the Actor network, θ μ are the network parameters of the Actor network, is additive noise, σ a is the standard deviation of the normal distribution of additive noise, α Low , α High are the minimum and maximum values ​​of the output action respectively, and clip(x, min, max) means clipping x to the range of (min, max). Actions generated by the perturbation policy network.

[0105] Step 2.2: According to the current action a tGet the corresponding current routing node, and select the next hop routing node of the current routing node based on the sorting mechanism, and get the next state (update state) according to the next hop routing node of the current routing node t+1 , and according to the next state s t+1 Calculate the reward value r for the current action t .

[0106] The next hop routing node of the current routing node is selected based on the sorting mechanism, specifically:

[0107] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0108] According to the adjacent node target distance list and the adjacent node weight list, based on descending sorting, an adjacent node distance sorting table and an adjacent node weight sorting table are obtained;

[0109] Adding the adjacent node distance ranking table and the adjacent node weight ranking table item by item to obtain an adjacent node comprehensive ranking table;

[0110] According to the adjacent node comprehensive ranking table, a next-hop routing node of the current routing node is selected.

[0111] For example, if the adjacent routing node list of the current routing node 0 is N L = [1, 2, 3], the corresponding adjacent node target distance list is d = [1200, 1000, 1100], then the adjacent node distance sorting table is O d =[2, 0, 1]; the corresponding adjacent node weight list is w = [0.9, 0.2, 0.4], then the adjacent node weight sorting table is O w =[2,0,1]. At this time, the adjacent node comprehensive sorting table is O a =[4, 0, 2], the next hop routing node of the current routing node 0 obtained according to the comprehensive ranking table of adjacent nodes should be routing 2.

[0112] Step 2.3: Set the reward value r t 、Current action a t , current status t and the next state s t+1 Stored in the replay buffer. When updating the parameters of the neural network framework, the total number of samples in the replay buffer is N. B <N Bmax A sample of , and the target action is calculated through the Target Actor network:

[0113] a t+1=clip((μ′(s t+1 |θ μ′ ))+clip(κ′,−c,c),α Low , α High )

[0114] Among them, μ′(s t+1 |θ μ′ ) is the output of the Target Actor network, θ μ′ is the network parameter of the Target Actor network, is additive noise, is the standard deviation of the normal distribution of additive noise. Specifically, the additive noise Actions used to perturb the target policy network.

[0115] Based on the target action, the minimum value of the two target evaluation network outputs is selected to calculate the target value:

[0116]

[0117] in, is the reward value, γ is the discount factor, is the expected cumulative reward of the ith Target Critic network under the updated state and the action corresponding to the updated state, is the network parameter of the i-th Target Critic network.

[0118] Specifically, whether a "dead route" is generated is judged by a detection function. If the detection function detects that there are no duplicate nodes on the current route optimization path, there is no "dead route"; if the detection function detects that there are duplicate nodes on the current route optimization path, there is a "dead route".

[0119] Step 2.4: Use the gradient descent method to update the network parameters of the critical network to minimize the loss function:

[0120]

[0121] Among them, N B is the sample set, s t , a t , r t ,s t+1 They are the current state, current action, reward value and updated state respectively. is the expected cumulative reward of the i-th evaluation network under the current state and current action, are the network parameters of the i-th evaluation network.

[0122] Since the update frequency of the Target network and the Actor network is lower than that of the Critic network, whenever the number of training steps reaches the preset generation step threshold, the gradient ascent method is used to update the network parameters of the Actor network:

[0123]

[0124] in, is the expected cumulative reward of the first evaluation network under the current state and current action, μ(s t ,θ μ ) is the output of the action network in the current state, θ μ are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0125] At the same time, based on the soft update strategy, update the Target network:

[0126] θ μ′ ←τθ μ +(1-τ)θ μ′

[0127]

[0128] In certain embodiments of the present application, the stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes:

[0129] Inputting the initial state of the single agent into the neural network framework to obtain an initial action;

[0130] generating an initial routing optimization path according to the initial state and the initial action;

[0131] Through deep reinforcement learning, the initial routing optimization path is iteratively updated step by step based on the neural network framework to generate a routing optimization path; wherein, in each generation step, the updated state of the single agent is obtained according to the current state of the single agent, and the initial routing optimization path and the network parameters of the neural network framework are updated according to the updated state; in the first generation step, the current state of the single agent is the initial state; in the last generation step, the updated state of the single agent corresponds to the routing optimization path.

[0132] This application first inputs the initial state of a single agent into a neural network framework to obtain an initial action, then generates an initial routing optimization path, and gradually iterates and updates the initial routing optimization path through deep reinforcement learning to generate a routing optimization path. In each step of generation, an updated state is obtained according to the current state of the single agent, and the network parameters of the initial routing optimization path and the neural network framework are updated according to the updated state. By gradually iterating and updating the path through deep reinforcement learning, it can be ensured that the path continuously approaches the target routing node from the starting routing node, and the network parameters of the neural network framework are updated at each step, so that the network parameters can be updated in time and the framework convergence speed can be accelerated.

[0133] In certain embodiments of the present application, obtaining the updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state specifically include:

[0134] According to the current state of the single agent, the current action of the single agent is obtained based on the neural network framework;

[0135] Selecting a next-hop routing node of a current routing node based on a sorting mechanism; wherein the current routing node corresponds to a current action of the single agent;

[0136] According to the next-hop routing node of the current routing node, the update state of the single agent is obtained;

[0137] The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

[0138] This application is first based on the neural network framework, obtains the current action according to the current state of the single intelligent agent, and then selects the next-hop routing node of the current routing node corresponding to the current action according to the sorting mechanism, which can reasonably select the next-hop routing node to avoid the generation of paths with unreachable routes, and then obtains the updated state according to the next-hop routing node, and updates the initial routing optimization path and the network parameters of the neural network framework according to the updated state, which can ensure that the path continuously approaches the target routing node from the starting routing node, and can update the network parameters in time, thereby increasing the convergence speed of the framework.

[0139] In certain embodiments of the present application, selecting the next hop routing node of the current routing node based on the sorting mechanism specifically includes:

[0140] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0141] According to the adjacent node target distance list and the adjacent node weight list, based on descending sorting, an adjacent node distance sorting table and an adjacent node weight sorting table are obtained;

[0142] According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next hop routing node of the current routing node is selected.

[0143] The present application first obtains an adjacent node target distance list and an adjacent node weight list corresponding to the adjacent routing node list of the current routing node, then obtains a corresponding sorting table based on descending sorting, and obtains an adjacent node comprehensive sorting table based on the sorting table, and then selects the next-hop routing node of the current routing node. Through the target distance list and weight list corresponding to the adjacent routing node list of the current routing node, the potential next-hop routing nodes of the current routing node can be reasonably quantified, and then when the next-hop routing node is selected by comprehensive sorting, the selected next-hop routing node can be made more reasonable to avoid the generation of paths with unreachable routes.

[0144] In certain embodiments of the present application, updating the network parameters of the neural network framework according to the update state specifically includes:

[0145] Calculating a reward value of the current state according to the current state and the updated state;

[0146] According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on a gradient descent algorithm;

[0147] The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically:

[0148]

[0149] Among them, N B is the sample set, s t , a t , r t ,s t+1 They are the current state, current action, reward value and updated state respectively. is the expected cumulative reward of the i-th evaluation network under the current state and current action, is the network parameter of the i-th evaluation network, is the target value, γ is the discount factor, For the i The target evaluation network is the expected cumulative reward under the updated state and the action corresponding to the updated state, For the iThe network parameters of the target evaluation network.

[0150] This application first calculates the reward value based on the current state and the updated state, and then updates the network parameters of the neural network framework based on the gradient descent algorithm, which can update the corresponding network so that it converges to the range required for the routing optimization task.

[0151] In certain embodiments of the present application, the stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate the route optimization path further includes:

[0152] When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state;

[0153] The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically:

[0154]

[0155] Among them, N B is the sample set, s t , a t is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, μ(s t ,θ μ ) is the output of the action network in the current state, θ μ are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0156] When the number of generated steps reaches a preset generation step threshold, the present application updates the network parameters of the neural network framework based on the gradient ascent algorithm according to the current state, and can update the corresponding network so that it converges to the range required for the routing optimization task.

[0157] Step S103: Control each routing node of the network to be controlled according to the routing optimization path of the last iteration.

[0158] The present application first constructs a single intelligent agent according to the node topology matrix of the network to be controlled, and then iterates the routing optimization path output by the single intelligent agent based on the neural network framework. In each round of iteration, the routing optimization path is gradually generated through deep reinforcement learning, which can make the routing optimization path continuously approach the target routing node from the starting routing node, thereby completing the task, and in each step of generation, the next-hop routing node is selected based on the sorting mechanism to update the routing optimization path. The next-hop routing node is selected by the sorting mechanism, and the next-hop routing node can be reasonably selected to avoid the generation of paths with unreachable routes. Then, when controlling each routing node of the network to be controlled according to the routing optimization path of the last iteration, routing control can be performed more reasonably.

[0159] Corresponding to the previous method, see Figure 2 , the embodiment of the present application provides a single-agent routing control device, including a single-agent construction module 210, a path optimization module 220 and a routing control module 230;

[0160] The single agent construction module 210 is used to construct a single agent of the network to be controlled according to the node topology matrix of the network to be controlled;

[0161] The routing optimization module 220 is used to iterate the routing optimization path output by the single agent based on a preset neural network framework until a preset iteration round is reached, and output the routing optimization path of the last iteration; wherein, in each iteration round, the routing optimization path is gradually generated based on the neural network framework through deep reinforcement learning; in each step of generation, the next hop routing node is selected based on a sorting mechanism to update the routing optimization path;

[0162] The routing control module 230 is used to control each routing node of the network to be controlled according to the routing optimization path of the last iteration.

[0163] In certain embodiments of the present application, the single agent construction module 210 includes a node data acquisition unit, a topology matrix construction unit, and a single agent construction unit;

[0164] The node data acquisition unit is used to acquire the node topology, node location and node flow of each routing node in the network to be controlled;

[0165] The topology matrix construction unit is used to construct the node topology matrix according to the node topology, the node position and the node traffic; wherein the node topology matrix includes the node weight of each routing node in the network to be controlled; the node weight is calculated based on the node topology;

[0166] The single agent construction unit is used to construct a single agent of the network to be controlled according to the node topology matrix.

[0167] In certain embodiments of the present application, the stepwise generation of a routing optimization path based on the neural network framework through deep reinforcement learning specifically includes:

[0168] Inputting the initial state of the single agent into the neural network framework to obtain an initial action;

[0169] generating an initial routing optimization path according to the initial state and the initial action;

[0170] Through deep reinforcement learning, the initial routing optimization path is iteratively updated step by step based on the neural network framework to generate a routing optimization path; wherein, in each generation step, the updated state of the single agent is obtained according to the current state of the single agent, and the initial routing optimization path and the network parameters of the neural network framework are updated according to the updated state; in the first generation step, the current state of the single agent is the initial state; in the last generation step, the updated state of the single agent corresponds to the routing optimization path.

[0171] In certain embodiments of the present application, obtaining the updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state specifically include:

[0172] According to the current state of the single agent, the current action of the single agent is obtained based on the neural network framework;

[0173] Selecting a next-hop routing node of a current routing node based on a sorting mechanism; wherein the current routing node corresponds to a current action of the single agent;

[0174] According to the next-hop routing node of the current routing node, the update state of the single agent is obtained;

[0175] The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

[0176] In certain embodiments of the present application, selecting the next hop routing node of the current routing node based on the sorting mechanism specifically includes:

[0177] A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent;

[0178] According to the adjacent node target distance list and the adjacent node weight list, based on descending sorting, an adjacent node distance sorting table and an adjacent node weight sorting table are obtained;

[0179] According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next hop routing node of the current routing node is selected.

[0180] In certain embodiments of the present application, updating the network parameters of the neural network framework according to the update state specifically includes:

[0181] Calculating a reward value of the current state according to the current state and the updated state;

[0182] According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on a gradient descent algorithm;

[0183] The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically:

[0184]

[0185] Among them, N B is the sample set, s t , a t , r t ,s t+1 They are the current state, current action, reward value and updated state respectively. is the expected cumulative reward of the i-th evaluation network under the current state and current action, is the network parameter of the i-th evaluation network, is the target value, γ is the discount factor, Evaluate the expected cumulative reward of the network for the i-th target under the updated state and the action corresponding to the updated state, Evaluate the network parameters of the network for the i-th target.

[0186] In certain embodiments of the present application, the stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate the route optimization path further includes:

[0187] When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state;

[0188] The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically:

[0189]

[0190] Among them, N B is the sample set, s t , a t is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, μ(s t ,θ μ ) is the output of the action network in the current state, θ μ are the network parameters of the action network, is the gradient of the network parameters of the action network.

[0191] The present application first constructs a single intelligent agent according to the node topology matrix of the network to be controlled, and then iterates the routing optimization path output by the single intelligent agent based on the neural network framework. In each round of iteration, the routing optimization path is gradually generated through deep reinforcement learning, which can make the routing optimization path continuously approach the target routing node from the starting routing node, thereby completing the task, and in each step of generation, the next-hop routing node is selected based on the sorting mechanism to update the routing optimization path. The next-hop routing node is selected by the sorting mechanism, and the next-hop routing node can be reasonably selected to avoid the generation of paths with unreachable routes. Then, when controlling each routing node of the network to be controlled according to the routing optimization path of the last iteration, routing control can be performed more reasonably.

[0192] It should be understood that the device provided in the embodiments of the present application corresponds to the aforementioned method, and a single-agent routing control device provided in the embodiments of the present application can implement a single-agent routing control method provided in any embodiment of the present application.

[0193] Adaptively, the embodiments of the present application also provide a computer device and a computer-readable storage medium.

[0194] The computer device comprises: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor;

[0195] Wherein, when the processor executes the computer program, a single-agent routing control method of the present application is implemented.

[0196] The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute a single-agent routing control method of the present application.

[0197] The above are some implementation methods of the present application, and the purpose, technical scheme and beneficial effects of the present application are further described in detail. It should be clear that the above-mentioned partial implementation methods of the present application cannot be understood as limiting the present application. It is particularly pointed out that for those skilled in the art, any changes, modifications, equivalent substitutions and variations made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A single agent routing control method, characterized in that: include: According to the node topology matrix of the network to be controlled, a single intelligent agent of the network to be controlled is constructed; Based on a preset neural network framework, the route optimization path output by the single agent is iterated until a preset iteration round is reached, and the route optimization path of the last iteration is output; wherein, in each iteration round, the route optimization path is gradually generated based on the neural network framework through deep reinforcement learning; In each generation step, a next-hop routing node is selected based on a sorting mechanism to update the routing optimization path; According to the routing optimization path of the last iteration, each routing node of the network to be controlled is controlled.

2. A single agent routing control method according to claim 1, characterized in that: The step of constructing a single intelligent agent of the network to be controlled according to the node topology matrix of the network to be controlled specifically includes: Obtain the node topology, node location and node traffic of each routing node in the network to be controlled; According to the node topology, the node position and the node traffic, the node topology matrix is ​​constructed; wherein the node topology matrix includes the node weight of each routing node in the network to be controlled; the node weight is calculated based on the node topology; According to the node topology matrix, a single intelligent agent of the network to be controlled is constructed.

3. A single agent routing control method according to claim 1, characterized in that: The stepwise generation of the routing optimization path based on the neural network framework through deep reinforcement learning specifically includes: Inputting the initial state of the single agent into the neural network framework to obtain an initial action; generating an initial routing optimization path according to the initial state and the initial action; Through deep reinforcement learning, the initial routing optimization path is iteratively updated step by step based on the neural network framework to generate a routing optimization path; wherein, in each generation step, the updated state of the single agent is obtained according to the current state of the single agent, and the initial routing optimization path and the network parameters of the neural network framework are updated according to the updated state; in the first generation step, the current state of the single agent is the initial state; in the last generation step, the updated state of the single agent corresponds to the routing optimization path.

4. A single agent routing control method according to claim 3, characterized in that: The step of obtaining an updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state, specifically includes: According to the current state of the single agent, the current action of the single agent is obtained based on the neural network framework; Selecting a next-hop routing node of a current routing node based on a sorting mechanism; wherein the current routing node corresponds to a current action of the single agent; According to the next-hop routing node of the current routing node, the update state of the single agent is obtained; The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

5. A single agent routing control method according to claim 4, characterized in that: The selecting the next hop routing node of the current routing node based on the sorting mechanism specifically includes: A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent; According to the adjacent node target distance list and the adjacent node weight list, based on descending sorting, an adjacent node distance sorting table and an adjacent node weight sorting table are obtained; According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next hop routing node of the current routing node is selected.

6. A single agent routing control method according to claim 4, characterized in that: The updating of the network parameters of the neural network framework according to the update state specifically includes: Calculating a reward value of the current state according to the current state and the updated state; According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on a gradient descent algorithm; The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically: Among them, N B is the sample set, s t , a t , r t ,s t+1 They are the current state, current action, reward value and updated state respectively. is the expected cumulative reward of the i-th evaluation network under the current state and current action, is the network parameter of the i-th evaluation network, is the target value, γ is the discount factor, Evaluate the expected cumulative reward of the network for the i-th target under the updated state and the action corresponding to the updated state, Evaluate the network parameters of the network for the i-th target.

7. A single agent routing control method according to claim 4, characterized in that: The stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate a route optimization path also includes: When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state; The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically: Among them, N B is the sample set, s t , a t is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, μ(s t ,θ μ ) is the output of the action network in the current state, θ μ are the network parameters of the action network, is the gradient of the network parameters of the action network.

8. A single-agent routing control device, characterized in that: It includes single agent building module, path optimization module and routing control module; The single agent construction module is used to construct a single agent of the network to be controlled according to the node topology matrix of the network to be controlled; The routing optimization module is used to iterate the routing optimization path output by the single agent based on a preset neural network framework until a preset iteration round is reached, and output the routing optimization path of the last iteration; wherein, in each iteration round, the routing optimization path is gradually generated based on the neural network framework through deep reinforcement learning; in each step of generation, the next hop routing node is selected based on a sorting mechanism to update the routing optimization path; The routing control module is used to control each routing node of the network to be controlled according to the routing optimization path of the last iteration.

9. A single-agent routing control device according to claim 8, characterized in that: The single agent construction module includes a node data acquisition unit, a topology matrix construction unit and a single agent construction unit; The node data acquisition unit is used to acquire the node topology, node location and node flow of each routing node in the network to be controlled; The topology matrix construction unit is used to construct the node topology matrix according to the node topology, the node position and the node traffic; wherein the node topology matrix includes the node weight of each routing node in the network to be controlled; The node weight is calculated based on the node topology; The single agent construction unit is used to construct a single agent of the network to be controlled according to the node topology matrix.

10. A single-agent routing control device according to claim 8, characterized in that: The stepwise generation of the routing optimization path based on the neural network framework through deep reinforcement learning specifically includes: Inputting the initial state of the single agent into the neural network framework to obtain an initial action; generating an initial routing optimization path according to the initial state and the initial action; Through deep reinforcement learning, the initial routing optimization path is iteratively updated step by step based on the neural network framework to generate a routing optimization path; wherein, in each generation step, the updated state of the single agent is obtained according to the current state of the single agent, and the initial routing optimization path and the network parameters of the neural network framework are updated according to the updated state; in the first generation step, the current state of the single agent is the initial state; in the last generation step, the updated state of the single agent corresponds to the routing optimization path.

11. A single-agent routing control device according to claim 10, characterized in that: The step of obtaining an updated state of the single agent according to the current state of the single agent, and updating the initial routing optimization path and the network parameters of the neural network framework according to the updated state, specifically includes: According to the current state of the single agent, the current action of the single agent is obtained based on the neural network framework; Selecting a next-hop routing node of a current routing node based on a sorting mechanism; wherein the current routing node corresponds to a current action of the single agent; According to the next-hop routing node of the current routing node, the update state of the single agent is obtained; The initial routing optimization path is updated according to the updated state, and the network parameters of the neural network framework are updated according to the updated state.

12. A single-agent routing control device according to claim 11, characterized in that: The selecting the next hop routing node of the current routing node based on the sorting mechanism specifically includes: A list of adjacent routing nodes corresponding to the current routing node is used to obtain an adjacent node target distance list and an adjacent node weight list; wherein the adjacent node weight list is obtained by the single agent; According to the adjacent node target distance list and the adjacent node weight list, based on descending sorting, an adjacent node distance sorting table and an adjacent node weight sorting table are obtained; According to the adjacent node distance sorting table and the adjacent node weight sorting table, an adjacent node comprehensive sorting table is obtained, and according to the adjacent node comprehensive sorting table, a next hop routing node of the current routing node is selected.

13. A single-agent routing control device according to claim 11, characterized in that: The updating of the network parameters of the neural network framework according to the update state specifically includes: Calculating a reward value of the current state according to the current state and the updated state; According to the current state, the updated state and the reward value, the network parameters of the neural network framework are updated based on a gradient descent algorithm; The network parameters of the neural network framework are updated based on the gradient descent algorithm, specifically: Among them, N B is the sample set, s t , a t , r t ,s t+1 They are the current state, current action, reward value and updated state respectively. is the expected cumulative reward of the i-th evaluation network under the current state and current action, is the network parameter of the i-th evaluation network, is the target value, γ is the discount factor, Evaluate the expected cumulative reward of the network under the updated state and the action corresponding to the updated state for the i-th target, Evaluate the network parameters of the network for the i-th target.

14. A single-agent routing control device according to claim 11, characterized in that: The stepwise iterative updating of the initial route optimization path based on the neural network framework through deep reinforcement learning to generate a route optimization path also includes: When the number of generated steps reaches a preset generation step threshold, the network parameters of the neural network framework are updated based on the gradient ascent algorithm according to the current state; The network parameters of the neural network framework are updated based on the gradient ascent algorithm, specifically: Among them, N B is the sample set, s t , a t is the current state and current action, is the expected cumulative reward of the first evaluation network under the current state and current action, μ(s t ,θ μ ) is the output of the action network in the current state, θ μ are the network parameters of the action network, is the gradient of the network parameters of the action network.

15. A computer device, characterized in that: include: processor; Memory; a computer program stored in the memory and configured to be executed by the processor; When the processor executes the computer program, it implements a single-agent routing control method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute a single-agent routing control method as described in any one of claims 1 to 7.

Images