Route policy optimization method and system, electronic device and storage medium

Abstract

The application provides a routing strategy optimization method and system, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: receiving current IP address state information transmitted by a router; inputting the current IP address state information into a decision model to obtain a current optimal decision action parameter corresponding to a maximum value function value; and sending the current optimal decision action parameter to the router, so that the router obtains a next-hop IP address according to the current optimal decision action parameter and performs routing forwarding on the next-hop IP address. The decision model is obtained by training a DQN model according to each sample data, including sample IP address state information, sample decision action parameters, a reward value corresponding to the sample decision action parameters, and next IP address state information corresponding to the sample decision action parameters, which are obtained by cyclically acquiring from an experience pool. The application improves the adaptability, accuracy and efficiency of routing strategy optimization.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a routing strategy optimization method, system, electronic device, and storage medium. Background Technology

[0002] In computer network communication, routing technology is a key and core technology. Therefore, how to efficiently and accurately adjust and optimize network routing strategies to improve network performance, reduce network latency, and alleviate network congestion is a pressing technical problem that needs to be solved.

[0003] In related technologies, traditional heuristic optimization methods are typically used to optimize routing strategies. For example, a routing protocol algorithm based on ant colony agent optimization has been demonstrated to be effective through simulation experiments using evaluation metrics such as routing cost and average packet delay. This method solves the problem of high time complexity in the algorithm search process. However, the routing strategies found are prone to getting trapped in local optima and cannot cope with model-free environments and dynamically changing environments, resulting in high optimization time complexity and poor adaptability of the routing strategies. Summary of the Invention

[0004] This invention provides a routing policy optimization method, system, electronic device, and storage medium to address the shortcomings of high time complexity and poor adaptability in existing routing optimization technologies, thereby improving the adaptability and efficiency of routing policy optimization.

[0005] This invention provides a routing policy optimization method, comprising:

[0006] Receive the current IP address status information transmitted by the router;

[0007] The current IP address status information is input into the decision model to obtain the current optimal decision action parameters corresponding to the maximum value function value;

[0008] The current optimal decision action parameters are sent to the router so that the router can obtain the next-hop IP address corresponding to the current IP address status information based on the current optimal decision action parameters, and perform routing forwarding on the next-hop IP address.

[0009] This invention provides a routing strategy optimization method, wherein the decision model is trained based on the following steps:

[0010] For the current iteration of training, if it is determined that the number of sample data in the experience pool stored in the memory is greater than or equal to a preset number, then multiple sample data are randomly sampled from the experience pool stored in the memory to construct the training dataset for the current iteration of training.

[0011] Based on the training dataset of the current iteration, the DQN model trained in the previous iteration is trained.

[0012] Based on the training results of each iteration, the decision model is constructed.

[0013] The sample data includes sample IP address status information, sample decision actions, the reward value corresponding to the sample decision actions, and the next IP address status information corresponding to the sample decision actions.

[0014] According to the present invention, a routing strategy optimization method is provided, wherein the DQN model includes a state-action value network and a target value network;

[0015] The step of training the DQN model from the previous iteration based on the training dataset of the current iteration includes:

[0016] The state information of each sample IP address and the decision action parameters of each sample in the training dataset are input into the state action value network trained in the previous iteration, and the state information of each next IP address in the training dataset is input into the target value network trained in the previous iteration, so as to train the DQN model trained in the previous iteration.

[0017] According to the present invention, a routing policy optimization method is provided, wherein the next IP address state information corresponding to the sample decision action parameter is generated by the router obtaining the next-hop IP address index corresponding to the sample IP address state information based on the sample decision action parameter, obtaining the next-hop IP address corresponding to the sample IP address state information based on the next-hop IP address index, performing routing forwarding on the next-hop IP address corresponding to the sample IP address state information, and performing IP address state transition based on the routing forwarding result.

[0018] According to the present invention, a routing policy optimization method is provided. The reward value corresponding to the sample decision action parameter is determined by the router matching the next-hop IP address corresponding to the sample decision action parameter with the target IP address. If the next-hop IP address corresponding to the sample decision action parameter matches the target IP address, the reward value is determined based on the target reward value. Alternatively, if the next-hop IP address corresponding to the sample decision action parameter does not match the target IP address, the routing metric between the next-hop IP address corresponding to the sample decision action parameter and the current IP address is searched in a pre-constructed routing path matrix, and the reward value is determined based on the routing metric.

[0019] The present invention also provides a routing policy optimization method, comprising:

[0020] Transmit the current IP address status information to the server;

[0021] Receive the current optimal decision action parameters transmitted by the server;

[0022] Based on the current optimal decision action parameters, obtain the next-hop IP address corresponding to the current IP address status information, and perform routing forwarding on the next-hop IP address;

[0023] The current optimal decision action parameters are the decision action parameters corresponding to the maximum value function value obtained by the server inputting the current IP address status information into the decision model.

[0024] The present invention also provides a routing policy optimization system, including a server, a router, and a storage device;

[0025] The server is configured to receive current IP address status information transmitted by the router; input the current IP address status information into the decision model to obtain the current optimal decision action parameters corresponding to the maximum value function value; and send the current optimal decision action parameters to the router.

[0026] The router is configured to obtain the next-hop IP address corresponding to the current IP address status information based on the current optimal decision action parameters, and to perform routing forwarding on the next-hop IP address.

[0027] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the routing strategy optimization method as described above.

[0028] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the routing strategy optimization method as described above.

[0029] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the routing strategy optimization method as described above.

[0030] The routing strategy optimization method, system, electronic device, and storage medium provided by this invention utilize sample IP address state information, sample decision action parameters, reward values ​​corresponding to the sample decision action parameters, and next IP address state information corresponding to the sample decision action parameters, generated through environmental interaction between the server and the router, to train a DQN model. This allows the resulting decision model to better interact, learn, and optimize with the environment based on the current IP address state information, thereby achieving better global optimization to obtain the current optimal decision action parameters and ultimately the globally optimal routing optimization strategy. This provides the optimal next-hop IP address under the current IP address state information, guiding the router's routing and forwarding, improving the rationality of routing strategy optimization, and enabling efficient and low-cost routing strategy implementation. Furthermore, the method of learning through trial and error via direct interaction between the server and the router solves the problems of difficult system modeling and lack of models. Moreover, by continuously interacting with the router to obtain new sample data for strategy training, it exhibits good adaptability. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is one of the flowcharts illustrating the routing strategy optimization method provided by the present invention;

[0033] Figure 2 This is a schematic diagram of the routing policy optimization system provided by the present invention;

[0034] Figure 3 This is the second flowchart of the routing strategy optimization method provided by the present invention;

[0035] Figure 4 This is the third flowchart of the routing strategy optimization method provided by the present invention;

[0036] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] Currently, the routing table of the OSPF (Open Shortest Path First) protocol is constructed using Dijkstra's algorithm (a typical shortest path algorithm, mainly implemented using breadth-first search, greedy algorithms, and dynamic programming). Because Dijkstra's algorithm is optimized through traversal, its time complexity is generally high. Although its complexity can be reduced by using different data structures, it is generally stable at O(n^2), where n is the number of IP (Internet Protocol) addresses to be forwarded. Furthermore, with the increasing scale of networks and the growing number of IP addresses, the time required for this algorithm to construct the routing table is also increasing. Newer solutions and technical approaches to address this drawback include:

[0039] Some related technologies use nodes with network communication quality exceeding a preset standard as candidate nodes for updating the routing table. Then, when the current node in the routing table meets the preset update conditions, a node is selected from the candidate nodes to replace the current node in the routing table, thus completing the routing table update. This technical solution updates the routing table by adding preset conditions, resulting in low time complexity. However, the construction of these preset conditions relies on expert experience, leading to relatively poor optimization accuracy.

[0040] In response, other technologies propose using traditional heuristic optimization methods to optimize routing strategies. For example, a routing protocol algorithm based on ant colony proxy optimization has been demonstrated for effectiveness through simulation experiments using evaluation metrics such as routing cost and average packet delay. This method addresses the high time complexity required during the algorithm search process and reduces reliance on expert experience. However, traditional heuristic optimization methods often tend to reach local optima, require pre-defined models, and cannot handle dynamically changing models well. They fail to efficiently and accurately obtain the global optimum, resulting in persistently high time complexity and poor adaptability in routing strategy optimization.

[0041] Therefore, to address the shortcomings of existing technologies that cannot efficiently and accurately obtain the globally optimal solution, resulting in high time complexity and poor adaptability in routing strategy optimization, this embodiment provides a routing strategy optimization method. Specifically, it trains a DQN (Deep Q Network) model using sample data generated through environmental interaction with the router to obtain a decision model. Based on the decision model, the path is optimized according to the reward, ultimately forming a routing strategy. This eliminates the need to consult the routing table for forwarding; instead, the forwarding is determined directly by the trained strategy. This reduces reliance on expert experience and eliminates the need to know the model principles of the environment. The interactive trial-and-error process also makes the algorithm more adaptable, achieving globally optimal routing strategy optimization, improving the rationality of the route, and thus enhancing the adaptability, robustness, and efficiency of routing strategy optimization. Furthermore, the distributed reinforcement learning deployment method offers better scalability and portability in application.

[0042] Figure 1 This is one of the flowcharts illustrating the routing policy optimization method provided by the present invention; for example... Figure 1 As shown, this method can be used for routing in protocol scenarios such as OSPF, and this embodiment does not specifically limit it; this method can be applied to a routing policy optimization system; the system includes a routing policy optimization server, a router with an interactive environment, and a memory with an experience pool storage.

[0043] Figure 2 This is a schematic diagram of the routing policy optimization system provided by the present invention; as shown below. Figure 2 As shown, the server in this system primarily carries reinforcement learning algorithms and algorithm tuning software for upper-layer applications. By using the server, the traditional method of querying addresses through routing tables can be abandoned, and a routing strategy based on neural networks can be used for address selection. This strategy is integrated into the server, making installation, adjustment, and migration easier. Simultaneously, the server uses the algorithm tuning software to adjust parameters such as network structure, learning rate, discount rate, and number of training iterations in the reinforcement learning algorithm, resulting in better learning and decision-making performance. Besides the algorithm tuning software for upper-layer applications facilitating parameter tuning, the server also receives and forwards data tuples sent by routers. This data is not stored on the server itself. When the amount of data tuples stored in the memory reaches a set value, the server randomly samples the data tuples in the memory, extracting a set amount for training the reinforcement learning model. The trained policy network can provide the optimal next-hop IP address based on all current IP addresses sent by the router, guiding the router's forwarding. The routing policy parameters can be adjusted in real time through the upper-layer tuning software, facilitating changes to the policy network for different network environments.

[0044] Routers with an interactive environment are distributed deployments that interact with servers and forward IP addresses. The forwarding of IP addresses can be achieved through multiple routing devices that concurrently forward IP addresses to various network devices, giving the system better flexibility, fault tolerance, and scalability. The interactive environment can involve the interaction forming the states, actions, and reward functions required for model training.

[0045] Compared to traditional routers that integrate the OSPF routing protocol and make decisions by building routing tables, this embodiment modifies the traditional router. It retains the synchronization process with the link-state database in the OSPF protocol, but removes the process of building the routing table using Dijkstra's algorithm. Instead, it builds the switching environment based on the link-state database. This construction process mainly consists of three steps: the index-IP address mapping relationship, the routing path matrix, and the construction of the interaction environment. The index-IP address mapping relationship is used to conveniently represent different IP addresses and is beneficial for network training and data forwarding. The routing path matrix represents the reward from the current IP address to other IP addresses, facilitating the construction of the reward function. The construction of the three elements represents the current state of all IP addresses, the action to execute the next-hop IP address, and the reward for evaluation, facilitating interaction with the server. Route optimization and decision-making are entirely handled by the algorithm within the server; the router is only responsible for environment interaction and forwarding.

[0046] The storage device saves the data tuples generated by the continuous interaction between the server and the router to the experience pool. When the amount of data reaches a certain quantity, the server will randomly sample from the experience pool.

[0047] In summary, this system adopts a distributed deployment approach, distributing the algorithm, environment, and data across servers, routers, and storage devices, respectively. Compared to the traditional approach of placing everything on a single device, this method offers advantages such as better scalability, easier migration, and modular design for easier development. Furthermore, externalizing the routing strategy on the server allows for the support of deeper neural networks and larger data volumes through increased hardware configuration, facilitating modifications to the neural network for different network structures. Externalizing the routing strategy also allows the router to allocate more resources for data aggregation and verification, ensuring the accuracy and security of data packets. Storing interactive data tuples in storage allows for the retention of more data, ensuring the policy network exhibits better exploratory power and robustness.

[0048] The method is executed by a server, and includes:

[0049] Step 110: Receive the current IP address status information transmitted by the router;

[0050] Here, the current IP address state information is determined by the router based on the state space information; the state space information is used to indicate whether the current IP address and other IP addresses have been reached (routed), and can be represented by the following matrix:

[0051] s t =[route_onehot1route_onehot2…route_onehot n ];

[0052]

[0053] Among them, route_onehot i This indicates whether the i-th IP address has been passed through (routing forwarded) in cycle t. If it has, set it to 1; otherwise, set it to 0. When the target IP address is reached, set it to 1 to indicate the end of the round.

[0054] Optionally, the server can monitor the router in real time to obtain the current IP address status information transmitted by the router, and thereby optimize the routing policy based on the current IP address status information;

[0055] The so-called current IP address state information is obtained by the router by collecting the routing state information of all IP addresses under the current period's routing based on the state space information when routing policy optimization is required.

[0056] Step 120: Input the current IP address status information into the decision model to obtain the current optimal decision action parameters corresponding to the maximum value function value;

[0057] Step 130: Send the current optimal decision action to the router so that the router can obtain the next-hop IP address corresponding to the current IP address status information according to the current optimal decision action parameters, and perform routing forwarding on the next-hop IP address;

[0058] The decision model is obtained by training the DQN model based on sample data obtained cyclically from the experience pool stored in the memory. The sample data includes sample IP address status information, sample decision action parameters, reward values ​​corresponding to the sample decision action parameters, and next IP address status information corresponding to the sample decision action parameters.

[0059] It should be noted that, considering the characteristics of the interaction between the two devices, the optimal decision action parameters selected by the server can include the index of the best IP address that the decision router needs to forward to the next hop, indicating the next hop IP address.

[0060] Alternatively, routing policy optimization can be based on the following steps:

[0061] The server receives the current IP address status information s sent by the router. t Then, based on the trained decision-making model and the current IP address status information s t Obtain the value function values, i.e. Q values, corresponding to all pre-configured candidate actions, and select the action a with the largest corresponding Q value. t As the parameters for the current optimal decision action, it is forwarded to the router.

[0062] After receiving the current optimal decision action parameters, the router looks up the mapping next-hop IP address index to find the next-hop IP address, performs route forwarding on the next-hop IP address, and after forwarding, the IP address changes and a state transition occurs, thus obtaining the next IP address state information s. t+1 The system checks whether the route reaches the target IP address, that is, whether the next-hop IP address matches the target IP address. If they match, the route reaches the target IP address; otherwise, the route does not reach the target IP address. If the route reaches the target IP address, the routing process ends; otherwise, the next IP address status information is sent. t+1 The new current IP address status information is transmitted to the server so that the server can iteratively optimize the route based on the new current IP address status information until the route reaches the target IP address.

[0063] The training steps for the decision model here include:

[0064] The server collects a large number of data tuples generated through multiple interactions with the router's environment model. These tuples include at least sample IP address status information generated in each historical period, sample decision action parameters, the corresponding reward values ​​for the sample decision action parameters, and the next IP address status information corresponding to the sample decision action parameters. This large number of data tuples are forwarded and stored in an experience pool in memory. During each iteration of training, a large amount of sample data is repeatedly retrieved from the experience pool, and the DQN model is iteratively trained based on this training set. This trained DQN model then constructs a decision model that can output the corresponding optimal decision action parameters based on IP address status information.

[0065] The environment model here refers to the setting where the agent learns and makes decisions. It defines the state space the agent can observe, the action space it can execute, and the state transition rules. The environment model also provides feedback based on the agent's actions, usually in the form of rewards. The environment model plays a crucial role in reinforcement learning; it not only provides the learning environment but also provides feedback on the agent's performance through rewards and state transitions, enabling the agent to learn and optimize its behavioral strategies through interaction with the environment. Therefore, establishing a reasonable environment model tailored to the specific circumstances is particularly important.

[0066] The sample IP address status information here can be the IP address status information generated by each router during the historical routing process; for specific acquisition steps, please refer to step 110.

[0067] The sample decision action parameters here can be the decision action parameters output by the DQN model after inputting the sample IP address status information into the DQN model, or they can be randomly selected decision action parameters. This embodiment does not specifically limit them.

[0068] The reward value corresponding to the sample decision action parameter and the next IP address state information corresponding to the sample decision action parameter can be the action reward value and state change value generated by the execution of the sample decision action parameter obtained by interacting with the environment model.

[0069] The routing policy optimization method provided in this invention uses sample IP address state information, sample decision action parameters, reward values ​​corresponding to sample decision action parameters, and next IP address state information corresponding to sample decision action parameters, obtained through environmental interaction between the server and the router, to train a DQN model. This allows the resulting decision model to better interact, learn, and optimize with the environment based on the current IP address state information, thereby obtaining the current optimal decision action parameters globally and ultimately the globally optimal routing optimization policy. This provides the optimal next-hop IP address under the current IP address state information, guiding the router's routing and forwarding, improving the rationality of routing policy optimization, and enabling efficient and low-cost routing policy implementation. Furthermore, the method of learning through trial and error via server-router interaction solves the problems of difficult system modeling and lack of models. By continuously interacting with the router to obtain new sample data for policy training, it exhibits good adaptability.

[0070] In some embodiments, the decision model is trained based on the following steps:

[0071] For the current iteration of training, if it is determined that the number of sample data in the experience pool stored in the memory is greater than or equal to a preset number, then multiple sample data are randomly sampled from the experience pool stored in the memory to construct the training dataset for the current iteration of training.

[0072] Based on the training dataset of the current iteration, the DQN model trained in the previous iteration is trained.

[0073] Based on the training results of each iteration, the decision model is constructed.

[0074] The sample data includes sample IP address status information, sample decision actions, the reward value corresponding to the sample decision actions, and the next IP address status information corresponding to the sample decision actions.

[0075] The preset quantity here can be set adaptively based on model performance requirements and computational complexity, or it can be set according to actual needs.

[0076] Optionally, the specific training steps for the decision model include:

[0077] For each iteration of training, the following steps are performed:

[0078] First, determine whether the number of sample data in the experience pool of the memory has reached (greater than or equal to) the preset number. If it has, then start the current iteration of training based on the following steps:

[0079] Multiple sample data are randomly drawn from the experience pool to construct the training dataset for the current iteration of training;

[0080] Train the DQN model based on the training dataset from the previous iteration;

[0081] After the current iteration of training ends, a new round of decision-making and environment interaction is conducted. Following this new round of interaction, the experience pool is updated. When the conditions for iterative training are met, sample data is drawn from the experience pool for a new round of training, until the current round ends. The so-called experience pool update can mean that after acquiring new sample data, if the experience pool has reached its maximum capacity limit, the oldest stored sample data is deleted from the experience pool, and the new sample data is added to the experience pool.

[0082] Finally, the routing performance of the DQN trained through multiple iterations was tested to select the DQN with the best routing performance and build a decision model.

[0083] The method provided in this embodiment effectively improves the performance of the decision model through a multi-iteration training method, resulting in better performance and generalization ability in practical applications. In some embodiments, the DQN model includes a state-action value network and a target value network;

[0084] The DQN model trained in the previous iteration is trained based on the training dataset of the current iteration.

[0085] The state information of each sample IP address and the decision action parameters of each sample in the training dataset are input into the state action value network trained in the previous iteration, and the state information of each next IP address in the training dataset is input into the target value network trained in the previous iteration, so as to train the DQN model trained in the previous iteration.

[0086] Optionally, the training steps of the decision model further include:

[0087] For the current iteration of training, if it is determined that the number of sample data in the experience pool is less than a preset number, then sample data is generated based on the state-action value network of the previous iteration of training, and the generated sample data is cyclically stored in the experience pool until it is determined that the number of sample data in the experience pool is greater than or equal to the preset number. Then, multiple sample data are randomly sampled from the experience pool to construct the training dataset for the current iteration of training.

[0088] The state information of each sample IP address and the decision action parameters of each sample in the training dataset are input into the state action value network trained in the previous iteration to obtain the first value function value of multiple pre-configured candidate decision action parameters corresponding to the state information of each sample IP address, and the value function value of each sample decision action parameter is obtained from the multiple first value function values.

[0089] The next IP address status information in the training dataset is input into the target value network trained in the previous iteration to obtain the second value function value of multiple candidate decision action parameters corresponding to each next IP address status information, and the optimal value function value is determined among the multiple second value function values.

[0090] Based on the reward value corresponding to each sample decision action parameter, the value function value of each sample decision action parameter, and the optimal value function value, the loss function value is obtained.

[0091] Based on the loss function value, the state-action value network and the target value network trained in the previous iteration are iteratively trained.

[0092] Based on the results of iterative training, the state-action value network with the best routing performance is selected from the state-action value networks trained through multiple iterations, and the decision model is constructed.

[0093] Figure 3 This is a second flowchart illustrating the routing policy optimization method provided by the present invention; as shown below. Figure 3 As shown, the DQN model includes a state-action value network (or Q network) and a target value network (or Target Q network); the Q network and the Target Q network are two neural networks with the same structure; the Q network is used to output the next-hop IP address, and the Target Q network is used to assist the training of the Q network. The sample data required for training is obtained by periodically sampling from the resource pool of the memory according to the algorithm settings.

[0094] The Q network input is the sample IP address and status information s. i Used to calculate sample IP address status information s i The corresponding value function values ​​of each candidate decision action parameter, and the Target Q network input is the next IP address state information s. i+1 The Q-network is used to calculate the value function of the target decision action parameters (i.e., the optimal decision action parameters). Using a dedicated network to calculate the target value can better reduce the correlation between the target value and the current value. The loss calculated based on the value function values ​​of the two decision action parameters is used to update the Q-network parameters through backpropagation. The target Q-network parameter update is achieved by determining a set value and directly copying the Q-network parameters when the training iteration count reaches the set value.

[0095] The preset quantity here can be set adaptively based on model performance requirements and computational complexity, or it can be set according to actual needs.

[0096] Optionally, the specific training steps for the decision model include:

[0097] For each iteration of training, the following steps are performed:

[0098] First, determine if the number of sample data in the experience pool of the memory has reached the preset number. If not, give an initial sample IP address state information s0. Based on the exploration rate configured for each iteration of training and a randomly determined random number, obtain the sample decision action parameter a0 corresponding to the sample IP address state information s0 from multiple pre-configured candidate decision action parameters. Input the sample IP address state information s0 and the sample decision action parameter a0 into the router, interact with the router, and generate sample data based on the tuple data returned by the router. Then, based on the next IP address state information s1 corresponding to the sample decision action parameter a0 contained in the tuple data, iteratively input it into the state action value network of the previous iteration of training, and iteratively interact with the router based on the sample decision action parameters output by the state action value network to form new sample data. Continue until the number of sample data reaches the preset number, then stop the sample data generation process of the current iteration of training, and start the current iteration of training based on the following steps:

[0099] Multiple sample data are randomly drawn from the experience pool to construct the training dataset for the current iteration of training;

[0100] The IP address status information s of each sample in the training dataset i and sample decision action parameter a i Input is fed into the state-action value network trained in the previous iteration to select the value function value Q(s) of the sample decision action parameter from the first value function values ​​of multiple pre-configured candidate decision action parameters learned. i ,a i ;ω);and will sample IP address status information s i Next IP address status information s i+1 The input is fed into the target value network trained in the previous iteration. The maximum value function value is selected from the second value function values ​​of multiple candidate decision action parameters corresponding to the next IP address state information as the optimal value function value, i.e., maxQ(s). i+1 ,a i+1 ;θ);

[0101] Subsequently, the reward value r corresponding to the sample decision action parameters is used. i+1 The value function value Q(s) of the decision action parameters for each sample i ,a i ;ω), and the optimal value function value maxQ(s) i+1 ,a i+1 ;θ), to obtain the loss function value L(ω), the specific calculation formula is as follows:

[0102]

[0103] Where γ is the discount rate, and represents the decision action parameters a0, a1…a i-1 The degree of impact on the current return; θ and ω are the model parameters of the target value network and the state-action value network, respectively.

[0104] Subsequently, gradient descent is performed on the state-action-value network from the previous training iteration based on the loss function, and the model parameters are updated. When the current training iteration reaches the preset value for updating the parameters of the Target Q network, the updated Q network parameters are copied to the Target Q network to synchronize the update of the Target Q network.

[0105] After the current iteration of training ends, a new round of decision-making and environment interaction is conducted. Following this new round of interaction, the experience pool is updated. When the conditions for iterative training are met, sample data is drawn from the experience pool for a new round of training, until the current round ends. The so-called experience pool update can mean that after acquiring new sample data, if the experience pool has reached its maximum capacity limit, the oldest stored sample data is deleted from the experience pool, and the new sample data is added to the experience pool.

[0106] Finally, the routing performance of the state-action value network trained through multiple iterations is tested to select the state-action value network with the best routing performance and construct a decision model.

[0107] The method provided in this embodiment effectively improves the performance of the model by using a decision model training method based on multiple iterations of training and loss function optimization, enabling it to have better effects and generalization ability in practical applications.

[0108] In some embodiments, sample data is generated based on the state-action value network trained in the previous iteration, and the generated sample data is cyclically stored in the experience pool until it is determined that the number of sample data in the experience pool is greater than or equal to the preset number, including:

[0109] Obtain the status information of the sample IP address transmitted by the router;

[0110] Based on the exploration rate and random number corresponding to the current iteration of training, the sample decision action parameters of the sample IP address status information are obtained from multiple candidate decision action parameters;

[0111] The sample IP address status information and the sample decision action parameters are sent to the router;

[0112] The router receives a data tuple transmitted by the router. The data tuple is generated by the router based on the sample IP address status information and the sample decision action parameters. The data tuple includes the sample IP address status information, the sample decision action, the reward value corresponding to the sample decision action parameters, and the next IP address status information corresponding to the sample decision action parameters.

[0113] Sample data is generated based on the data tuples, and the generated sample data is stored in the experience pool.

[0114] The next IP address status information in the data tuple is used as the new sample IP address status information. Sample data is generated iteratively for the new sample IP address status information, and the generated sample data is cyclically stored in the experience pool until it is determined that the number of sample data in the experience pool is greater than or equal to the preset number.

[0115] Optionally, if the sample data in the experience pool does not reach the set quantity, the server and router continuously interact with the environment, obtaining tuple data after each interaction and saving it to the storage to update the experience pool; wherein the update steps of the experience pool include:

[0116] The router will send the sample IP address status information s i The status information is sent to the server. After receiving the status information, the server can randomly select a sample decision action parameter 'a' from multiple candidate decision action parameters based on the exploration rate and random number corresponding to the current iteration of training. i Forwarded to the router, or the corresponding sample decision action parameter 'a' can be output through the state-action-value network decision-making process. i Forward to the router;

[0117] In some embodiments, obtaining the sample decision action parameters of the sample IP address status information from a plurality of candidate decision action parameters based on the exploration rate and random number corresponding to the current iteration of training includes:

[0118] Based on the exploration rate and random number corresponding to the current iteration of training, any one of the candidate decision action parameters is randomly selected as the sample decision action parameter for the sample IP address status information; or...

[0119] The sample IP address status information is input into the state action value network trained in the previous iteration to obtain the first value function value of multiple candidate decision action parameters. Based on the first value function value, the decision action parameter corresponding to the optimal value function value is determined from the multiple candidate decision action parameters as the sample decision action parameter of the sample IP address status information.

[0120] Optionally, the random number selected in the current iteration of training can be compared with the exploration rate set in the current iteration of training. If the random number corresponding to the current iteration of training is less than the exploration rate corresponding to the current iteration of training, a decision action parameter can be randomly selected from multiple candidate decision action parameters as the sample decision action parameter a. i Forward to the router; if the random number selected in the current iteration of training is greater than or equal to the exploration rate corresponding to the current iteration of training, input the sample IP address state information into the state-action value network of the previous iteration of training to obtain the first value function value of multiple candidate decision action parameters, and then select maxQ(s) from them. i ,a i The corresponding action is used as the sample decision action parameter a. i Forward to the router. It should be noted that after each decision, the exploration rate ε is decreased linearly or exponentially until the exploration rate ε reaches 1e-3, at which point the updating of the exploration rate ε stops.

[0121] The router receives the sample decision action parameter a i Then, by looking up the mapping index_IP, the next-hop IP address is found, and routing forwarding for the next-hop IP address is executed. After forwarding, the IP address changes, and a state transition occurs, changing the state to s. i+1 The router evaluates the above behaviors based on the reward function to obtain the reward value r corresponding to the sample decision action parameters. i+1 The router will send the sample IP address status information s i Sample decision action parameter a i Reward value r i+1 After an IP address change, the next IP address state information s after a state transition occurs. i+1 Four variables constitute a data tuple (s) i ,a i ,r i+1 ,s i+1 The data is sent to the server, which forwards it and stores it in the memory's resource pool. When the number of sample data in the memory's resource pool reaches a set value, the server begins training the Q-network by randomly sampling data; otherwise, it continues with the experience pool update step described above. Where r... i+1 This indicates the status information s of the sample IP address. i In this case, the parameter a of the sample decision action is executed. i The reward received later. i+1 This indicates the status information s of the sample IP address. i In this case, the parameter a of the sample decision action is executed. i The state transition that occurs afterward.

[0122] The method provided in this embodiment updates the sample data in the experience pool in real time based on the trained model during the training process, making the samples in the experience pool more diverse and comprehensive. This helps to improve the model's generalization ability and learning effect, and allows the model to gradually adapt to changes and uncertainties in the environment, improving the model's robustness and adaptability. This enables the model to make more accurate decisions when facing new situations, thereby promoting the learning and optimization process of the reinforcement learning model and improving the model's performance and coping ability.

[0123] In some embodiments, the next IP address state information corresponding to the sample decision action parameters is generated by the router obtaining the next-hop IP address index corresponding to the sample IP address state information based on the sample decision action parameters, obtaining the next-hop IP address corresponding to the sample IP address state information based on the next-hop IP address index, performing routing forwarding on the next-hop IP address corresponding to the sample IP address state information, and performing IP address state transition based on the routing forwarding result.

[0124] It's important to note that after establishing neighbor and adjacency relationships, the OSPF protocol synchronizes a link-state database. This database records the next-hop IP address reachable by each routing node, along with the method and cost of reaching that address. The optimization algorithm then uses this information to find the optimal path. The index-IP address mapping table is constructed to more conveniently meet the basic requirements for interaction.

[0125] Since IP addresses used in actual network forwarding processes are quite long, directly using them would be detrimental to data transmission and neural network training. Therefore, to conveniently represent different IP addresses, this patent constructs a mapping between an index and IP addresses, as shown in the following formula:

[0126]

[0127] Where n represents the total number of IP addresses, IP n This represents the nth IP address.

[0128] Furthermore, considering the training of a practical Q network, directly using IP addresses with a large number of bits is not conducive to network training. Therefore, based on the mapping relationship of the above formula, the decision action parameters are defined as the index of the IP address. Accordingly, the action space in the router can be represented by the following formula:

[0129] A = [a1 a2 … a t ];

[0130] Among them, a t Represented as: a t =index_IP i0i∈n, representing the current IP address status information s t The next-hop IP address index is a t .

[0131] Optionally, after the server sends the sample decision action parameters to the router, the router can obtain the next-hop IP address index corresponding to the sample IP address status information based on the sample decision action parameters. Then, it can obtain the next-hop IP address corresponding to the sample IP address status information by associating the index contained in the next-hop IP address index, and perform distributed parallel forwarding on the next-hop IP address. After forwarding, the IP address changes and an IP address state transition occurs, thereby obtaining the next IP address status information.

[0132] The method provided in this embodiment achieves state transition through interaction with the router, which reduces reliance on expert experience and eliminates the need to know the model principles of the environment. At the same time, the interactive trial and error process makes the algorithm more adaptable, achieves globally optimal routing strategy optimization, improves the rationality of routing, and thus improves the adaptability, robustness and efficiency of routing strategy optimization.

[0133] In some embodiments, the reward value corresponding to the sample decision action parameter is determined by the router matching the next-hop IP address corresponding to the sample decision action parameter with the target IP address. If the next-hop IP address corresponding to the sample decision action parameter matches the target IP address, the reward value is determined based on the target reward value. Alternatively, if the next-hop IP address corresponding to the sample decision action parameter does not match the target IP address, the router searches for the routing metric between the next-hop IP address corresponding to the sample decision action parameter and the current IP address in a pre-built routing path matrix and determines the reward value based on the routing metric.

[0134] It should be noted that after synchronizing information in the link-state database, the OSPF protocol constructs a routing path matrix based on the information in this database. This matrix represents the information between any two IP addresses and facilitates the subsequent construction of the state space, action space, and rewards. The formula for calculating this routing path matrix is ​​as follows:

[0135]

[0136] The route matrix represents the connection state and routing metric between any two IP addresses, where n and m represent the IP address index numbers, and cost... nm This represents the routing metric (such as routing cost) from the nth IP address to the mth IP address. If the two IP addresses cannot be reached directly, take infinity.

[0137] As a unique entity in reinforcement learning, rewards can evaluate each action and serve as the basis for solving state values and state-action values. Therefore, the quality of the reward function design is crucial for the outcome of the entire reinforcement learning decision-making process.

[0138] Optionally, the reward value corresponding to the sample decision action parameter can be implemented by the router performing the following steps:

[0139] After obtaining the sample decision action parameter, match the next-hop IP address corresponding to the sample decision action parameter with the target IP address. If the next-hop IP address corresponding to the sample decision action parameter matches the target IP address, use the pre-set target reward value as the reward value corresponding to the sample decision action parameter; if the next-hop IP address corresponding to the sample decision action parameter does not match the target IP address, look up the routing metric value between the next-hop IP address corresponding to the sample decision action parameter and the current IP address in the routing path matrix as the reward value corresponding to the sample decision action parameter. The calculation formula for the reward value is as follows:

[0140]

[0141] Rewawrd is the reward value from the i-th IP address to the j-th IP address, where i < j to prevent loops.

[0142] In the method provided in this embodiment, the setting of the reward is also adaptively generated through the path cost in the link state database, reducing the reliance on expert experience, thereby improving the accuracy and efficiency of routing policy optimization.

[0143] The following uses specific embodiments to specifically describe the process of training the decision model provided in this embodiment.

[0144] As shown in Table 1, the complete steps of decision model training include:

[0145] Step 1), initial parameters, including but not limited to initializing the Q-network structure, target Q-network structure, network parameters, experience pool, routing path matrix, index-IP address mapping relationship matrix, reward function, environmental parameters, and the index of the current routing IP address;

[0146] Step 2), for each episode, initialize the environment to obtain the state s0;

[0147] Step 3), the router sends the IP address state information s t to the server. After receiving this state, the server may randomly select a decision action parameter a t and forward it to the router, or obtain the Q values Q(st The algorithm selects the decision action parameter a with the largest corresponding Q value (e.g., a; ε). t Forward it to the router.

[0148] Step 4), the router receives the decision action parameter a. t Then, by looking up the mapping index_IP, the next-hop IP address is found, and forwarding is performed.

[0149] Step 5): After the router performs forwarding, the IP address changes and a state transition occurs, with the IP address state information changing to s. t+1 The router determines the current index_IP. at+1 Is it the target IP address? If not, calculate the reward value based on the reward function.

[0150] Step 6), the router sends the IP address status information s t Decision action parameter a t Reward value r t+1 After an IP address change, the new IP address status information s that undergoes a state transition t+1 Four variables constitute a data tuple (s) t ,a t ,r t+1 ,s t+1 The data is sent to the server, which forwards it and stores it in memory. When the amount of data in memory reaches a set value, step 7) is executed; otherwise, steps 3)-6) are continued.

[0151] Step 7): The server randomly samples the data tuples in the experience pool of the memory to obtain a small batch of sample data (s). t ,a t ,r t+1 ,s t+1 ).

[0152] Step 8) Based on the output results of the target value network and the state-action value network obtained from the sample data, a loss function is calculated, and gradient descent is performed using the loss function to update the model parameters of the Q network. When the number of training iterations reaches the preset value C for updating the parameters of the Target Q network, the updated Q network parameters are copied to the Target Q network.

[0153] Step 9): If the target IP address is reached in this round, return to step 2) to start the next round; otherwise, return to step 3) to continue the current round.

[0154] Step 10) The training process ends when the set number of training rounds is reached.

[0155] Table 1. Pseudocode for Decision Model Training Algorithm

[0156]

[0157]

[0158] In some embodiments, this embodiment also provides a routing policy optimization method, wherein the execution subject of the method is a router; such as Figure 4 As shown, the method includes:

[0159] Step 410: Transmit the current IP address status information to the server;

[0160] Step 420: Receive the current optimal decision action parameters transmitted by the server;

[0161] Step 430: Based on the current optimal decision action parameters, obtain the next-hop IP address corresponding to the current IP address status information, and perform routing forwarding on the next-hop IP address;

[0162] The current optimal decision action parameters are the decision action parameters corresponding to the maximum value function value obtained by the server inputting the current IP address status information into the decision model.

[0163] Optionally, the router determines the current IP address status information based on the state space information and transmits the current IP address status information to the server; the state space information is used to indicate whether the current IP address and other IP addresses have been reached (routed).

[0164] The server receives the current IP address status information s sent by the router. t Then, based on the trained decision-making model and the current IP address status information s t Obtain the value function values, i.e. Q values, corresponding to all pre-configured candidate actions, and select the action a with the largest corresponding Q value. t As the parameters for the current optimal decision action, it is forwarded to the router.

[0165] After receiving the current optimal decision action parameters, the router looks up the mapping next-hop IP address index to find the next-hop IP address, performs route forwarding on the next-hop IP address, and after forwarding, the IP address changes and a state transition occurs, thus obtaining the next IP address state information s. t+1The system checks whether the route reaches the target IP address, that is, whether the next-hop IP address matches the target IP address. If they match, the route reaches the target IP address; otherwise, the route does not reach the target IP address. If the route reaches the target IP address, the routing process ends; otherwise, the next IP address status information is sent. t+1 The new current IP address status information is transmitted to the server so that the server can iteratively optimize the route based on the new current IP address status information until the route reaches the target IP address.

[0166] The routing policy optimization method provided in this invention uses sample IP address state information, sample decision action parameters, reward values ​​corresponding to sample decision action parameters, and next IP address state information corresponding to sample decision action parameters, obtained through interaction between the router and server with the environment, to train a DQN model. This allows the resulting decision model to better interact, learn, and optimize with the environment based on the current IP address state information, thereby obtaining the current optimal decision action parameters globally and ultimately the globally optimal routing optimization policy. This provides the optimal next-hop IP address under the current IP address state information, guiding the router's routing and forwarding, improving the rationality of routing policy optimization, and enabling efficient and low-cost routing policy implementation. Furthermore, the method of learning through trial and error via interaction between the server and router solves the problems of difficult system modeling and lack of models. By continuously interacting with the router to obtain new sample data for policy training, it exhibits good adaptability.

[0167] The routing policy optimization system provided by the present invention is described below. The routing policy optimization system described below can be referred to in correspondence with the routing policy optimization method described above.

[0168] like Figure 2 As shown, the routing policy optimization system includes a server, a router, and a storage device;

[0169] The server is configured to receive the current IP address status information transmitted by the router, input the current IP address status information into the decision model to obtain the current optimal decision action parameters corresponding to the maximum value function value, and send the current optimal decision action parameters to the router.

[0170] Optionally, the server can monitor the router in real time to obtain the current IP address status information transmitted by the router, and then optimize the routing policy based on the current IP address status information. The so-called current IP address status information is obtained by the router by collecting the routing status information of all IP addresses in the current period according to the state space information when routing policy optimization is required.

[0171] The server receives the current IP address status information s sent by the router. t Then, based on the trained decision-making model and the current IP address status information s t Obtain the value function value, or Q value, for all candidate actions, and select the action a with the largest Q value. t As the parameters for the current optimal decision action, it is forwarded to the router.

[0172] The router is configured to obtain the next-hop IP address corresponding to the current IP address status information based on the current optimal decision action parameters, and to perform routing forwarding on the next-hop IP address.

[0173] Optionally, after receiving the current optimal decision action parameters, the router looks up the mapping next-hop IP address index to find the next-hop IP address, performs route forwarding on the next-hop IP address, and after forwarding, the IP address changes and a state transition occurs, thereby obtaining the next IP address state information s. t+1 The system checks whether the route reaches the target IP address, that is, whether the next-hop IP address matches the target IP address. If they match, the route reaches the target IP address; otherwise, the route does not reach the target IP address. If the route reaches the target IP address, the routing process ends; otherwise, the next IP address status information is sent. t+1 The new current IP address status information is transmitted to the server so that the server can iteratively optimize the route based on the new current IP address status information until the route reaches the target IP address.

[0174] This embodiment addresses the issues of high time complexity and poor adaptability in path optimization within the OSPF routing protocol by proposing a routing policy optimization system. This system deploys algorithms, data, and the environment across different network devices. Through multiple interactions between devices, the algorithm is trained, resolving the need for pre-guided models when using heuristic optimization algorithms. Real-time parameter tuning via software also avoids the problem of local optima. Furthermore, when adding or removing devices or dynamically changing the network structure, the algorithm on the server directly interacts with the router to obtain new data, guiding policy updates without requiring model reconstruction or algorithm re-traversal. This facilitates rapid learning of routing policies and provides good adaptability and robustness. The distributed reinforcement learning deployment method also offers better scalability, portability, and ease of installation. Deploying algorithms, the environment, and data separately on servers, routers, and storage devices maximizes resource utilization for each device. Externalizing the routing policy allows for more resources for policy training and learning, while saving router resources for packet aggregation and verification.

[0175] The system provided by this invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0176] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can invoke logical instructions in the memory 530 to execute a routing policy optimization method. This method includes: receiving current IP address status information transmitted by a router; inputting the current IP address status information into a decision model to obtain current optimal decision action parameters corresponding to the maximum value function value; sending the current optimal decision action parameters to the router so that the router can obtain the next-hop IP address corresponding to the current IP address status information based on the current optimal decision action parameters, and route the next-hop IP address, or transmit the current IP address status information to a server; receiving current optimal decision action parameters transmitted by the server; obtaining the next-hop IP address corresponding to the current IP address status information based on the current optimal decision action parameters, and route the next-hop IP address; wherein the current optimal decision action parameters are the decision action parameters corresponding to the maximum value function value obtained by the server inputting the current IP address status information into the decision model.

[0177] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the routing strategy optimization method provided by the above methods. The method includes: receiving current IP address status information transmitted by a router; inputting the current IP address status information into a decision model to obtain current optimal decision action parameters corresponding to the maximum value function value; sending the current optimal decision action parameters to the router so that the router can obtain the next-hop IP address corresponding to the current IP address status information according to the current optimal decision action parameters and route the next-hop IP address, or transmit the current IP address status information to a server; receiving current optimal decision action parameters transmitted by the server; obtaining the next-hop IP address corresponding to the current IP address status information according to the current optimal decision action parameters and route the next-hop IP address; wherein, the current optimal decision action parameters are the decision action parameters corresponding to the maximum value function value obtained by the server inputting the current IP address status information into the decision model.

[0179] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the routing strategy optimization method provided by the above methods. The method includes: receiving current IP address status information transmitted by a router; inputting the current IP address status information into a decision model to obtain current optimal decision action parameters corresponding to the maximum value function value; sending the current optimal decision action parameters to the router, so that the router can obtain the next-hop IP address corresponding to the current IP address status information according to the current optimal decision action parameters, and route the next-hop IP address, or transmit the current IP address status information to a server; receiving current optimal decision action parameters transmitted by the server; obtaining the next-hop IP address corresponding to the current IP address status information according to the current optimal decision action parameters, and route the next-hop IP address; wherein the current optimal decision action parameters are the decision action parameters corresponding to the maximum value function value obtained by the server inputting the current IP address status information into the decision model.

[0180] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0181] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A routing strategy optimization method, characterized in that, include: Receive the current IP address status information transmitted by the router; The current IP address status information is input into the decision model to obtain the current optimal decision action parameters corresponding to the maximum value function value; The current optimal decision action parameters are sent to the router so that the router can obtain the next-hop IP address corresponding to the current IP address status information based on the current optimal decision action parameters, and perform routing forwarding on the next-hop IP address; The decision model was trained based on the following steps: For the current iteration of training, if it is determined that the number of sample data in the experience pool stored in the memory is greater than or equal to a preset number, then multiple sample data are randomly sampled from the experience pool stored in the memory to construct the training dataset for the current iteration of training. Based on the training dataset of the current iteration, the DQN model trained in the previous iteration is trained. Based on the training results of each iteration, the decision model is constructed. The sample data includes sample IP address status information, sample decision actions, the reward value corresponding to the sample decision actions, and the next IP address status information corresponding to the sample decision actions.

2. The routing strategy optimization method according to claim 1, characterized in that, The DQN model includes a state-action value network and a target value network; The step of training the DQN model from the previous iteration based on the training dataset of the current iteration includes: The state information of each sample IP address and the decision action parameters of each sample in the training dataset are input into the state action value network trained in the previous iteration, and the state information of each next IP address in the training dataset is input into the target value network trained in the previous iteration, so as to train the DQN model trained in the previous iteration.

3. The routing strategy optimization method according to any one of claims 1-2, characterized in that, The next IP address state information corresponding to the sample decision action parameters is generated by the router obtaining the next-hop IP address index corresponding to the sample IP address state information based on the sample decision action parameters, obtaining the next-hop IP address corresponding to the sample IP address state information based on the next-hop IP address index, performing routing forwarding on the next-hop IP address corresponding to the sample IP address state information, and performing IP address state transition based on the routing forwarding result.

4. The routing strategy optimization method according to any one of claims 1-2, characterized in that, The reward value corresponding to the sample decision action parameter is determined by the router matching the next-hop IP address corresponding to the sample decision action parameter with the target IP address. If the next-hop IP address corresponding to the sample decision action parameter matches the target IP address, the reward value is determined based on the target reward value. Alternatively, if the next-hop IP address corresponding to the sample decision action parameter does not match the target IP address, the router searches for the routing metric between the next-hop IP address corresponding to the sample decision action parameter and the current IP address in a pre-built routing path matrix and determines the reward value based on the routing metric.

5. A routing strategy optimization method, characterized in that, include: Transmit the current IP address status information to the server; Receive the current optimal decision action parameters transmitted by the server; Based on the current optimal decision action parameters, obtain the next-hop IP address corresponding to the current IP address status information, and perform routing forwarding on the next-hop IP address; The current optimal decision action parameters are the decision action parameters corresponding to the maximum value function value obtained by the server inputting the current IP address status information into the decision model; The decision model was trained based on the following steps: For the current iteration of training, if it is determined that the number of sample data in the experience pool stored in the memory is greater than or equal to a preset number, then multiple sample data are randomly sampled from the experience pool stored in the memory to construct the training dataset for the current iteration of training. Based on the training dataset of the current iteration, the DQN model trained in the previous iteration is trained. Based on the training results of each iteration, the decision model is constructed. The sample data includes sample IP address status information, sample decision actions, the reward value corresponding to the sample decision actions, and the next IP address status information corresponding to the sample decision actions.

6. A routing policy optimization system, characterized in that, Includes servers, routers, and storage; The server is configured to receive current IP address status information transmitted by the router; input the current IP address status information into the decision model to obtain the current optimal decision action parameters corresponding to the maximum value function value; and send the current optimal decision action parameters to the router. The router is configured to obtain the next-hop IP address corresponding to the current IP address status information based on the current optimal decision action parameters, and to perform routing forwarding on the next-hop IP address; The decision model was trained based on the following steps: For the current iteration of training, if it is determined that the number of sample data in the experience pool stored in the memory is greater than or equal to a preset number, then multiple sample data are randomly sampled from the experience pool stored in the memory to construct the training dataset for the current iteration of training. Based on the training dataset of the current iteration, the DQN model trained in the previous iteration is trained. Based on the training results of each iteration, the decision model is constructed. The sample data includes sample IP address status information, sample decision actions, the reward value corresponding to the sample decision actions, and the next IP address status information corresponding to the sample decision actions.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the routing policy optimization method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the routing policy optimization method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the routing policy optimization method as described in any one of claims 1 to 5.

Images