A software-defined network routing method based on deep reinforcement learning

Abstract

The present invention discloses a software-defined network routing method based on deep reinforcement learning, comprising: based on actual network operating parameters, performing state design and action design on network flow information according to the principle of large flow priority scheduling and task objectives; building a network congestion model based on network bandwidth data obtained by a software-defined network (SDN) controller; performing reward design based on the network congestion model and network QoS data, and training a reinforcement learning model in combination with state design and action design; the reinforcement learning model implements route selection based on the state value of the actual network flow. The present invention uses reinforcement learning as the main framework of the entire method, by prioritizing the scheduling of large flows in the network, and based on the original bandwidth data, performing state design of the reinforcement model based on large flow priority scheduling, thereby improving various network QoS indicators, and further combining the network congestion model to improve the training effect of the reinforcement learning model.

Description

Technical Field

[0001] The present invention relates to the technical field of software-defined network routing, and in particular to a software-defined network routing method based on deep reinforcement learning. Background Art

[0002] Routing is a key task in software-defined networking (SDN). It aims to select the most appropriate forwarding path for network flows within SDN and has widespread application in network management. Traditional approaches primarily focus on implementing traditional routing strategies within SDN networks and formulating routing problems as optimization problems and designing heuristic algorithms to solve them. Compared to traditional approaches, machine learning-based methods are more flexible and can achieve better QoS metrics. Because reinforcement learning (RL)-based methods do not require labeled data and can learn autonomously, they are more suitable for solving routing problems than supervised learning-based methods.

[0003] Key steps in RL-based routing tasks are designing the state space, action space, and reward values, and selecting an appropriate reinforcement learning algorithm. Most current approaches are based on value functions or policy gradient-based reinforcement learning algorithms. Some approaches consider the entire network state to design the state space, using the link weight matrix as the action output. Routing methods based on deep deterministic policy gradients (DDPGs) have been proposed, which account for the periodicity of network state and use recurrent neural networks (RNNs) to capture these characteristics. Other approaches consider the limitations of TCAM resources in the network, designing the state space using allocated resources and resource requirements as part of the state space, and implementing them using DDQNs. These approaches consider the entire network state, resulting in an excessively large state space and difficulty in convergence. Other approaches consider only flows or switches in the network as state, using the selection of the next hop or routing path for the flow as the action, and designing reinforcement learning models using network QoS as the reward value. This reduces the difficulty of model training and further improves scalability. However, these models fail to consider the impact of large flows on the environment and fail to exploit the potential information in existing network data during training, resulting in poor model training performance. Summary of the Invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a software-defined network routing method based on deep reinforcement learning to solve the problem that existing training learning methods cannot consider the impact of large flows on the environment and cannot mine the potential information of existing network data during training, resulting in poor training performance of the learning model.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] An embodiment of the present invention provides a software-defined network routing method based on deep reinforcement learning, comprising:

[0009] Based on the actual network operation parameters, the state design and action design of network flow information are carried out according to the principle of large flow priority scheduling and task objectives;

[0010] Build a network congestion model based on the network bandwidth data obtained by the software-defined network (SDN) controller;

[0011] Perform reward design based on the network congestion model and network QoS data, and train a reinforcement learning model in combination with the state design and the action design;

[0012] The reinforcement learning model implements route selection based on the state value of the actual network flow.

[0013] As a preferred solution of the software-defined network routing method based on deep reinforcement learning described in the present invention, the state design includes: designing a state for each network flow in the network based on the principle of large flow priority scheduling;

[0014] The state design can be expressed as:

[0015] s i =(src,dst,seq)

[0016] Among them, src and dst are the source node and destination node of the flow respectively, and seq is the relative size of the flow among all flows.

[0017] As a preferred solution of the software-defined network routing method based on deep reinforcement learning described in the present invention, the action design includes: selecting a path for each network flow based on the task goal, calculating several paths for each flow in advance through the Dijkstra algorithm, and selecting a path from the alternative paths for output.

[0018] As a preferred solution of the software-defined network routing method based on deep reinforcement learning described in the present invention, building a network congestion model based on network bandwidth data includes:

[0019] The network congestion model is defined as:

[0020] {θmax ,(link ij ,α ij ,β ij ,γ ij )}

[0021] Among them, θ max is the congestion threshold, link ij For any link in the network, α ij is the link utilization, β ij is the instantaneous traffic of the link, γ ij is the indirect congestion degree, i is the first switch, and j is the second switch.

[0022] As a preferred solution of the software-defined network routing method based on deep reinforcement learning described in the present invention, it further includes: calculating the link utilization and instantaneous flow of links in the network based on network bandwidth data;

[0023] The link utilization is the ratio of the existing bandwidth to the link bandwidth;

[0024] The instantaneous link traffic is the difference between the link bandwidth used at the current moment and the link bandwidth used at the previous moment divided by the interval time.

[0025] As a preferred solution of the software-defined network routing method based on deep reinforcement learning described in the present invention, wherein: calculating the indirect congestion of the link based on the link utilization and the instantaneous flow of the link includes:

[0026] The indirect congestion calculation of the target link can be expressed as the sum of the indirect congestion of the adjacent edges corresponding to the two endpoints;

[0027] The indirect congestion of the corresponding adjacent edge is the sum of the link utilization of the edge, the instantaneous link flow, and the indirect congestion of the adjacent edge of the edge;

[0028] The final indirect congestion value of the corresponding adjacent edge is obtained by recursively calculating the target link through breadth-first search.

[0029] As a preferred solution of the software-defined network routing method based on deep reinforcement learning described in the present invention, wherein: reward design is performed based on the network congestion model and network QoS data, including:

[0030] When the QoS indicators of the action path meet the constraints, normalization is performed to obtain the corresponding reward values, which are bandwidth reward, delay reward, and packet loss rate reward;

[0031] When the QoS indicator of the action path does not meet the constraints, a penalty value is obtained;

[0032] Based on the network congestion model, the congestion reward is calculated by calculating the impact of link utilization and link instantaneous traffic on link congestion.

[0033] The reward value corresponding to the action is the sum of the bandwidth reward, delay reward, packet loss rate reward and congestion reward of the action path.

[0034] As a preferred solution of the software-defined network routing method based on deep reinforcement learning described in the present invention, it also includes:

[0035] The reward value is expressed as:

[0036]

[0037] Among them, r_bwd path is the bandwidth reward, r_delay path is the delayed reward, r_loss path is the packet loss rate reward, r_congest path Congestion reward.

[0038] As a preferred solution of the software-defined network routing method based on deep reinforcement learning described in the present invention, wherein: reinforcement learning model training is performed according to the reward design combined with state design and action design, including:

[0039] Initialize reinforcement learning model parameters;

[0040] The reinforcement learning model is trained for several rounds, and at the beginning of each round, the environment parameters are initialized to obtain the initial state;

[0041] In each state, the action is selected based on the model output and the corresponding cumulative reward Q value is calculated;

[0042] Select the action output when the Q value is the smallest, store the state action and the reward value in the experience replay, and complete one round of training;

[0043] After several rounds of training, the reinforcement learning model is further trained based on the action output, stored state action, and reward value in the experience replay until the reward value converges.

[0044] As a preferred solution of the software-defined network routing method based on deep reinforcement learning described in the present invention, wherein: the reinforcement learning model performs routing selection on the actual network flow, including:

[0045] Once the reward value converges, the reinforcement learning model completes training;

[0046] Obtain the actual network flow information and generate the corresponding state value consisting of the source node, target node and the relative size of the flow;

[0047] The route selection is realized by outputting the action in each state obtained as a routing strategy.

[0048] Compared with the existing technology, the present invention has the following beneficial effects: the present invention uses reinforcement learning as the main framework of the entire method, and through the priority scheduling of large flows in the network, based on the original bandwidth data, the state design of the reinforcement model is performed based on the priority scheduling of large flows, thereby improving various QoS indicators of the network, and further combining the network congestion model to improve the training effect of the reinforcement learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0050] Figure 1 This is a schematic diagram of the overall process of a software-defined network routing method based on deep reinforcement learning according to an embodiment of the present invention;

[0051] Figure 2 The overall architecture of the RSDRL model of a software-defined network routing method based on deep reinforcement learning according to an embodiment of the present invention;

[0052] Figure 3 This is a diagram illustrating indirect congestion of a software-defined network routing method based on deep reinforcement learning according to an embodiment of the present invention;

[0053] Figure 4 This is a diagram showing experimental results of model parameter selection for a software-defined network routing method based on deep reinforcement learning according to an embodiment of the present invention;

[0054] Figure 5 This is a graph showing the comparative experimental results of the QoS indicators of the RSDRL model of a software-defined network routing method based on deep reinforcement learning and the traditional method according to an embodiment of the present invention, wherein: Figure 5 (a) is a schematic diagram of network throughput. Figure 5 (b) is a diagram of network load balancing. Figure 5 (c) is a schematic diagram of the average link delay. Figure 5 (d) is a schematic diagram of the average link packet loss rate;

[0055] Figure 6This is a graph showing experimental results comparing the training time of an RSDRL model of a software-defined network routing method based on deep reinforcement learning according to an embodiment of the present invention and a traditional method;

[0056] Figure 7 This is a graph showing the ablation experiment results of a software-defined network routing method based on deep reinforcement learning according to an embodiment of the present invention; wherein, Figure 7 (a) is a schematic diagram of network throughput. Figure 7 (b) is a diagram of network load balancing. Figure 7 (c) is a schematic diagram of the average link delay. Figure 7 (d) is a schematic diagram of the average link packet loss rate. DETAILED DESCRIPTION

[0057] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0059] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0060] The present invention is described in detail with reference to schematic diagrams. For ease of illustration, cross-sectional views of device structures may be partially enlarged and not to scale when describing embodiments of the present invention. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0061] In the description of the present invention, it should be noted that the terms "upper, lower, inner, and outer" and other references to orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0062] In this disclosure, unless otherwise specified or limited, the terms "mounted," "connected," and "connected" should be interpreted broadly. For example, they may refer to fixed, removable, or integral connections. They may also refer to mechanical, electrical, or direct connections, indirect connections through an intermediary, or internal communication between two components. Those skilled in the art will understand the specific meanings of these terms in this disclosure.

[0063] Example 1

[0064] Reference Figures 1 to 3 , is an embodiment of the present invention, which provides a software-defined network routing method based on deep reinforcement learning, including:

[0065] S102: Based on the actual network operation parameters, according to the principle of large flow priority scheduling and task objectives, the network flow information is designed for state and action respectively;

[0066] Furthermore, state design includes: designing a state for each network flow in the network based on the principle of high-volume flow priority scheduling;

[0067] State design can be expressed as:

[0068] s i =(src,dst,seq)

[0069] Among them, src and dst are the source node and destination node of the flow respectively, and seq is the relative size of the flow among all flows.

[0070] Furthermore, the action design includes: selecting a path for each network flow based on the task goal, calculating several paths for each flow in advance through the Dijkstra algorithm, and selecting a path from the alternative paths for output.

[0071] It should be noted that during training, the reinforcement learning model randomly generates a seq state for each flow and performs state migration based on the seq, migrating from relatively large flows to smaller flows. Since a path has been selected for the OD flow, the state migration process will also change the environment, that is, the available bandwidth, packet loss rate, and delay data of the SDN environment will change, thereby affecting the forwarding of subsequent flows. When actually forwarding traffic, the controller will monitor the data plane to obtain all OD flow information, sort all OD flows to generate a seq state, add it to the flow state, and then input it into the model to obtain the optimal path, reducing the complexity of training.

[0072] It should also be noted that the network topology diagram is defined as:

[0073] G=(V,E)

[0074] Among them, V and E are the sets that identify network nodes and links respectively.

[0075] For any link in the network ij ∈E, let the bandwidth, transmission delay and packet loss rate of the link be b respectively ij ,d ij ,l ij .

[0076] The set of network flows is defined as:

[0077] S={S 0 ,S 1 ,S 2 …S n-1}

[0078] For any flow S in the network k ,k∈[0,n), and its bandwidth requirement, i.e., traffic size, delay requirement, and packet loss rate requirement are B k ,D k ,L k ,use Represents stream S k The set of links passed through, The value is 0 or 1, indicating the flow S k Whether it passes through the link link ij .

[0079] S104: Building a network congestion model based on network bandwidth data acquired by the software-defined network (SDN) controller;

[0080] Furthermore, building a network congestion model based on network bandwidth data includes:

[0081] The network congestion model is defined as:

[0082] {θmax ,(link ij ,α ij ,β ij ,γ ij )}

[0083] Among them, θ max is the congestion threshold, link ij For any link in the network, α ij is the link utilization, β ij is the instantaneous traffic of the link, γ ij is the indirect congestion degree, i is the first switch, and j is the second switch.

[0084] It should be noted that, ij and β ij Considering the impact of traffic volume on link congestion, the calculation formula is:

[0085]

[0086]

[0087] in, is the link bandwidth used, γ in the network congestion model ij Indicates the possibility that other links in the network will cause congestion to the target link. To better describe γ ij The calculation process of link ij An adjacent edge e∈E with degree n: if and only if e and link ij The shortest path between them passes through n nodes in the network.

[0088] Specifically, it also includes: calculating the link utilization and instantaneous flow of links in the network based on the network bandwidth data;

[0089] Link utilization is the ratio of existing bandwidth to link bandwidth;

[0090] The instantaneous link traffic is the difference between the link bandwidth used at the current moment and the link bandwidth used at the previous moment divided by the interval time.

[0091] Furthermore, the indirect congestion of the link is calculated based on the link utilization and the instantaneous traffic of the link, including:

[0092] The indirect congestion calculation of the target link can be expressed as the sum of the indirect congestion of the adjacent edges corresponding to the two endpoints;

[0093] The indirect congestion of the corresponding adjacent edge is the sum of the link utilization of the edge, the instantaneous link traffic, and the indirect congestion of the adjacent edge of the edge;

[0094] The final indirect congestion value of the corresponding adjacent edge is obtained by recursively calculating the target link through breadth-first search.

[0095] It should be noted that for the target link, if there is a congested link in the network, it is easy to know that the target link is also prone to congestion; γ ij Characterizes the possible impact of congestion of other links in the network on the target link; for the input link, its γ value is equal to the average of the γ values ​​of all adjacent edges with a degree of 1; and for link e ′ ≠e∈E, its γ value is equal to the sum of α and β of the edge and the γ values ​​of all adjacent edges with degree 1; link link ij The congestion value is calculated as α ij , β ij and γ ij The harmony.

[0096] S106: Design rewards based on the network congestion model and network QoS data, and train a reinforcement learning model in combination with state design and action design;

[0097] Furthermore, rewards are designed based on network congestion models and network QoS data, including:

[0098] When the QoS indicators of the action path meet the constraints, they are normalized to obtain the corresponding reward values, which are bandwidth reward, delay reward, and packet loss rate reward.

[0099] When the QoS indicator of the action path does not meet the constraints, a penalty value is obtained;

[0100] Based on the network congestion model, the congestion reward is calculated by calculating the impact of link utilization and link instantaneous traffic on link congestion.

[0101] The reward value corresponding to an action is the sum of the bandwidth reward, delay reward, packet loss rate reward, and congestion reward of the action path.

[0102] It should be noted that normalization is the process of normalizing the constraints to [0, 100], with the penalty value being a maximum of 100. The reinforcement learning model considers QoS indicators and network congestion during training to improve the performance of the entire network. The model first calculates the bandwidth, latency, and packet loss rate of the path and determines whether the constraints are met.

[0103] It should also be noted that the optimization problem is constructed as follows:

[0104]

[0105] Among them, TotalThroughput, TotalDelay, and TotalLoss are the total network throughput, total delay, and total packet loss rate, respectively.

[0106] The constraint that a single flow passes through the link in both directions simultaneously is expressed as:

[0107]

[0108] in, Represents stream S k Will it pass the link link ij ;

[0109] Link congestion constraint, expressed as:

[0110]

[0111] Among them, B k Represents stream S k The flow rate, b ij Indicates link ij bandwidth;

[0112] The network satisfies the bandwidth requirement constraints for each flow, expressed as:

[0113]

[0114] Among them, S k0 Represents any flow in the network, Represents stream S k0 Will it pass the link link ij , B k0 Represents stream S k0 The flow rate;

[0115] The network meets the delay requirement constraints of each flow, expressed as:

[0116]

[0117] Among them, D k Represents stream S k The delay requirement, d ij Indicates link ij transmission delay;

[0118] The network meets the packet loss rate requirement constraint for each flow, expressed as:

[0119]

[0120] Among them, L k Represents stream S k Packet loss rate requirement, l ij Indicates link ij Packet loss rate;

[0121] Specifically, it also includes:

[0122] The reward value is expressed as:

[0123]

[0124] Among them, r_bwd path is the bandwidth reward, r_delay path is the delayed reward, r_loss path is the packet loss rate reward, r_congest path Congestion reward.

[0125] It should be noted that r_congest path The congestion value of the most congested link on the path based on the network congestion model.

[0126] Furthermore, the reinforcement learning model training is carried out based on reward design combined with state design and action design, including:

[0127] Initialize reinforcement learning model parameters;

[0128] The reinforcement learning model will be trained for several rounds. At the beginning of each round, the environment parameters will be initialized to obtain the initial state.

[0129] In each state, the action is selected based on the model output and the corresponding cumulative reward Q value is calculated;

[0130] Select the action output when the Q value is the smallest, store the state action and reward value in the experience replay, and complete one round of training;

[0131] After several rounds of training, the reinforcement learning model is further trained based on the action output in the experience replay, the stored state action and the reward value until the reward value converges.

[0132] S108: The reinforcement learning model implements route selection based on the state value of the actual network flow;

[0133] Furthermore, the reinforcement learning model performs routing on actual network flows, including:

[0134] Once the reward value converges, the reinforcement learning model completes training;

[0135] Obtain the actual network flow information and generate the corresponding state value consisting of the source node, target node and the relative size of the flow;

[0136] The route selection is implemented based on the action obtained in each state as the routing strategy output.

[0137] Example 2

[0138] Reference Figures 4 to 7, which is an embodiment of the present invention. In order to verify the beneficial effects of the present invention, scientific verification is carried out through comparative experiments.

[0139] This embodiment is based on the actual experimental environment, using Linux, Mininet, and Ryu software platforms to set the SDN controller and model parameters, as shown in Table 1 and Table 2:

[0140] Table 1 Controller parameters

[0141] Unit / second Monitoring cycle 450 Topology discovery cycle 5 Network QoS data collection cycle 8

[0142] Table 2 Model parameters

[0143]

[0144]

[0145] The data plane uses the GEANT topology. To better verify the effectiveness of the network congestion model, we explore the impact of edge degree n on the model. Edge degree n describes the potential for congestion caused by other links in the network on the target link. We compare three sets of experiments with n = 1, 2, and 3, and evaluate the model based on the reward value obtained during training and the model calculation time.

[0146] pass Figure 4 It can be seen that when n = 1, 2, and 3, the model obtains reward values ​​of 924.42, 907.61, and 908.24, respectively, and the time spent is 1.22s, 21.09s, and 52.94s, respectively; when n increases from 1 to 2, the model obtains a lower reward value, indicating that the model can more fully utilize the congestion information of the link to obtain the optimal path, but when n increases from 2 to 3, the reward value does not change much, indicating that for the GEANT topology, there are fewer links with an edge degree of 3 and less information that can be further utilized; however, when n increases, the need to perform recursive operations on each edge will bring about greater computational overhead, and the model calculation time will be significantly increased; considering all factors, the result of choosing n = 1 is good.

[0147] In the comparative experiment, the embodiment of the present invention uses TMGen to generate data plane traffic and compares it with other traditional methods in terms of network throughput, network load balancing, link delay, and link packet loss rate. This embodiment selects 14 time points to generate 14 traffic matrices of different sizes, with an average traffic value of 5000kbps.

[0148] Comparative experimental results such as Figure 5As shown in the figure, it can be seen from the experimental results that DSDRL has certain advantages compared with the DQN strategy DRSIR, the Q-learning strategy RSIR, or the Dijkstra strategies Dijkstra_comp and Dijkstra_hop. Compared with DRSIR, Q-learning, Dijkstra_comp, and Dijkstra_hop, RSDRL has an improvement of 8.9%, 16.2%, 36.4%, and 27.2% in network throughput, 11.6%, 28.8%, 21.3%, and 19.3% in network load balancing, 11.3%, 23.9%, 25.2%, and 11.8% in link delay, and 8.3%, 14.9%, 40.2%, and 16.9% in link packet loss rate. It can be seen that the method of the present invention can effectively improve the routing effect under SDN, and RSDRL can optimize the scheduling of large flows. When a large flow appears in the network, the model considers giving priority to selecting paths for the large flow to avoid the situation where multiple small flows are concentrated on certain links due to unreasonable routing scheduling, resulting in a decrease in the available bandwidth of the link, and thus an inability to select a suitable path for the large flow. At the same time, the model will consider the degree of congestion in the network and select links with lower congestion for the next flow to avoid the generation of bottleneck links.

[0149] pass Figure 6 As shown in the figure, since RSDRL needs to additionally calculate the indirect congestion during training, RSDRL has the longest training time of 58.865s. The training time of the DQN-based model DRSIR is also longer than that of the Q-learning-based model RSIR, which is mainly due to the different sizes of the model state space. The Dijkstra-based method has the lowest time overhead because it does not require network training.

[0150] In order to further verify the beneficial effects, this embodiment conducts ablation learning on the model, removing the two modules of high-flow priority scheduling and network congestion model from the model for experimental analysis. Figure 7 As shown in the figure, removing the large flow priority scheduling can obtain DRSIR_model, and removing the network congestion model can obtain DRSIR_seq; compared with DRSIR_seq and DRSIR_model, RSDRL has an improvement of 14.86% and 2.12% in network throughput, 9.49% and 2.06% in network load balancing, 22.79% and 2.68% in link delay, and 27.81% and 10.25% in link packet loss rate; it can be seen that the large flow priority scheduling module and the network congestion model module have certain improvements on the model.

[0151] Compared with the existing commonly used routing methods, the method of the present invention has high network service quality in a network environment with more large flows, and the network congestion model can also make full use of the original network data, thereby improving the model training effect.

[0152] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0153] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiment of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal translation scripting language JavaScript, etc.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0157] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0158] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A software-defined network routing method based on deep reinforcement learning, characterized in that: include: Based on the actual network operation parameters, the state design and action design of network flow information are carried out according to the principle of large flow priority scheduling and task objectives; Build a network congestion model based on the network bandwidth data obtained by the software-defined network (SDN) controller; Perform reward design based on the network congestion model and network QoS data, and train a reinforcement learning model in combination with the state design and the action design; Reinforcement learning model training is performed based on the reward design combined with the state design and action design, including: Initialize reinforcement learning model parameters; The reinforcement learning model is trained for several rounds, and at the beginning of each round, the environment parameters are initialized to obtain the initial state; In each state, the action is selected based on the model output and the corresponding cumulative reward Q value is calculated; Select the action output when the Q value is the smallest, store the state action and the reward value in the experience replay, and complete one round of training; After several rounds of training, the reinforcement learning model is further trained based on the action outputs, stored state actions, and reward values ​​in the experience replay until the reward values ​​converge. The reinforcement learning model implements route selection based on the state value of the actual network flow.

2. The software-defined network routing method based on deep reinforcement learning according to claim 1, characterized in that: The state design includes: designing a state for each network flow in the network based on the principle of large flow priority scheduling; The state design can be expressed as: s i =(src,dst,seq) Among them, src and dst are the source node and destination node of the flow respectively, and seq is the relative size of the flow among all flows.

3. The software-defined network routing method based on deep reinforcement learning according to claim 2, characterized in that: Action design includes: selecting a path for each network flow based on the task goal, calculating several paths for each flow in advance through the Dijkstra algorithm, and selecting a path from the alternative paths for output.

4. The software-defined network routing method based on deep reinforcement learning according to claim 3, characterized in that Building a network congestion model based on network bandwidth data includes: The network congestion model is defined as: {i} max ,(link ij ,a ij ,b ij ,c ij )} Among them, θ max is the congestion threshold, link ij For any link in the network, α ij is the link utilization, β ij is the instantaneous traffic of the link, γ ij is the indirect congestion degree, i is the first switch, and j is the second switch.

5. The software-defined network routing method based on deep reinforcement learning according to claim 4, characterized in that: Also includes: Calculate the link utilization and instantaneous traffic of links in the network based on network bandwidth data; The link utilization is the ratio of the existing bandwidth to the link bandwidth; The instantaneous link traffic is the difference between the link bandwidth used at the current moment and the link bandwidth used at the previous moment divided by the interval time.

6. The software-defined network routing method based on deep reinforcement learning according to claim 5, characterized in that: Calculating the indirect congestion degree of the link according to the link utilization and the instantaneous flow of the link, including: The indirect congestion calculation of the target link can be expressed as the sum of the indirect congestion of the adjacent edges corresponding to the two endpoints; The indirect congestion of the corresponding adjacent edge is the sum of the link utilization of the edge, the instantaneous link flow, and the indirect congestion of the adjacent edge of the edge; The final indirect congestion value of the corresponding adjacent edge is obtained by recursively calculating the target link through breadth-first search.

7. The software-defined network routing method based on deep reinforcement learning according to any one of claims 4, 5, and 6, characterized in that: Reward design is performed based on the network congestion model and network QoS data, including: When the QoS indicators of the action path meet the constraints, normalization is performed to obtain the corresponding reward values, which are bandwidth reward, delay reward, and packet loss rate reward; When the QoS indicator of the action path does not meet the constraints, a penalty value is obtained; Based on the network congestion model, the congestion reward is calculated by calculating the impact of link utilization and link instantaneous traffic on link congestion. The reward value corresponding to the action is the sum of the bandwidth reward, delay reward, packet loss rate reward and congestion reward of the action path.

8. The software-defined network routing method based on deep reinforcement learning according to claim 7, characterized in that: Also includes: The reward value is expressed as: Among them, r_bwd path is the bandwidth reward, r_delay path is the delayed reward, r_loss path is the packet loss rate reward, r_congest path Congestion reward.

9. The software-defined network routing method based on deep reinforcement learning according to claim 8, characterized in that: The reinforcement learning model performs routing on actual network flows, including: Once the reward value converges, the reinforcement learning model completes training; Obtain the actual network flow information and generate the corresponding state value consisting of the source node, target node and the relative size of the flow; The route selection is implemented based on the action obtained in each state as the routing strategy output.

Images