Multi-path routing method, apparatus, device, storage medium and computer program product

Abstract

The application discloses a kind of multi-path routing method, device, equipment, storage medium and computer program product, the method includes: the network routing model of random scene trigger reinforcement learning is constructed, model defines the learning period and stable period of network operation;In learning period, periodically according to network state based on reinforcement learning algorithm update action value function, and the multi-path sub-flow distribution proportion between source node and target node is calculated, and the multi-path routing strategy is determined;In stable period, whether trigger model update action value function is determined based on random scene trigger mechanism, and the multi-path sub-flow distribution proportion is recalculated, and the multi-path routing strategy is optimized.The application designs multi-path routing based on dynamic value function, effectively utilizes temporary resource by sub-flow distribution strategy, improves overall resource utilization;Random scene trigger mechanism is introduced, immediately trigger model update when network occurs sudden event, shorten response time, improve system dynamic adaptability.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a multi-path routing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Currently, reinforcement learning-based intelligent routing algorithms, such as routing strategies based on Q-learning to optimize traffic engineering or adaptive routing algorithms based on DQN, while improving the intelligence of network routing to some extent, suffer from two main technical drawbacks: First, weak dynamic adaptability. These algorithms typically use fixed update cycles or model updates based on a certain amount of interaction accumulation, making them inadequate when faced with rapid changes in network conditions. Sudden network events, such as node failures, link interruptions, or traffic surges, lead to lengthy model adjustment times, resulting in network performance degradation, reduced service quality, and even congestion. Furthermore, in highly dynamic network environments, frequent event triggering can cause excessive algorithm updates and adjustments, increasing computational burden, causing policy instability and performance degradation, and becoming a performance bottleneck for resource-constrained systems. Second, limited resource utilization efficiency. Existing technologies, in designing and training reinforcement learning network routing models, often focus on single-path transmission, exhibiting low sensitivity to local network events. For example, when local resources change, the model may fail to utilize them in a timely manner, leading to resource waste or over-reliance on fixed paths, affecting overall resource utilization, limiting network flexibility and robustness, and hindering efficient and precise resource scheduling. Summary of the Invention

[0003] The purpose of this invention is to provide a multi-path routing method, apparatus, device, storage medium, and computer program product. It designs a multi-path routing based on a dynamic value function, effectively utilizes temporary resources through a sub-flow splitting strategy, improves overall resource utilization, and achieves efficient and flexible resource management. It introduces a random scenario triggering mechanism to immediately trigger model updates when sudden events occur in the network, ensuring that the routing strategy quickly adapts to changes in network status, shortens response time, and improves the system's dynamic adaptability.

[0004] To achieve the above objectives, embodiments of the present invention provide a multi-path routing method, including: A network routing model for reinforcement learning triggered by random scenarios is constructed, wherein the network routing model defines the learning period and the stabilization period of the network operation; During the learning period, the action value function is periodically updated based on the network state using a reinforcement learning algorithm, and the multi-path sub-flow allocation ratio between the source node and the target node is calculated based on the updated action value function to determine the multi-path routing strategy. During the stable period, a random scenario triggering mechanism is used to determine whether to trigger the model to update the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

[0005] As an improvement to the above scheme, the construction of a network routing model for reinforcement learning triggered by random scenarios includes: Define the state space, action space, and reward function of the network nodes, and configure the random scenario triggering mechanism, which divides the network operation process into the learning period and the stable period; Initialize the action value function table, where the row index in the action value function table represents the target node and the column index represents the next hop node.

[0006] As an improvement to the above scheme, the network routing model enters the learning period by default in the initial stage of operation; when the fluctuation value of the maximum cumulative reward is detected to be less than the preset convergence threshold for N consecutive periods, the network routing model is determined to have converged and automatically switches to the stable period.

[0007] As an improvement to the above scheme, the step of calculating the multi-path sub-flow allocation ratio between the source node and the target node based on the updated action value function includes: The row vector corresponding to the target node is determined based on the updated action value function table; Based on the action value of each next-hop node in the row vector, calculate the subflow allocation ratio to each next-hop node.

[0008] As an improvement to the above scheme, the step of determining whether to trigger a model update of the action value function based on a random scenario triggering mechanism and recalculating the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy includes: Real-time monitoring of the rate of change of network status parameters; If the rate of change does not exceed the preset random scenario trigger threshold, then the current multi-path sub-flow allocation ratio is directly applied for traffic forwarding. If the rate of change exceeds a preset random scenario trigger threshold, the model is triggered to update the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

[0009] As an improvement to the above scheme, the action value function is updated using the Q-Learning algorithm, and the update formula for the action value function is: ; In the formula, Indicates the first i After +1 iterations of update, the current node j The target node isk The value of the action; Indicates the first i Before the next iteration update, the current node j The target node is k The historical value of this action; α Indicates the learning rate; r This represents the immediate reward received by traffic after it is forwarded at a node; γ Indicates the discount factor; n Indicates the current node j Candidate next-hop nodes; F ( j ) represents the current node j The set of all available next-hop nodes; Indicates the current node j Among all available next-hop nodes, select the action value with the minimum cost / optimal forwarding overhead; Indicates the current node j The target node in the Q table is k And the next hop node is n The value of the action.

[0010] This invention also provides a multi-path routing device, comprising: The model building module is used to build a network routing model for reinforcement learning triggered by random scenarios. The network routing model defines the learning period and the stabilization period of the network operation. The strategy learning module is used to periodically update the action value function based on the network state according to the reinforcement learning algorithm during the learning period, and calculate the multi-path sub-flow allocation ratio between the source node and the target node according to the updated action value function to determine the multi-path routing strategy. The routing execution module is used to determine whether to trigger the model to update the action value function based on a random scenario triggering mechanism during the stable period, and to recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

[0011] This invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the multipath routing method described in any of the preceding claims.

[0012] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the multi-path routing method described above.

[0013] This invention also provides a computer program product, which includes a computer program or computer instructions, wherein the computer program or computer instructions, when executed by a processor, implement the multi-path routing method described above.

[0014] Compared to existing technologies, the beneficial effects of the multi-path routing method, apparatus, device, storage medium, and computer program product provided by this invention are as follows: By constructing a network routing model based on random scenario-triggered reinforcement learning, the network routing model defines a learning period and a stable period for network operation; during the learning period, the action value function is periodically updated according to the network state based on the reinforcement learning algorithm, and the multi-path sub-flow allocation ratio between the source node and the target node is calculated based on the updated action value function to determine the multi-path routing strategy; during the stable period, a random scenario triggering mechanism is used to determine whether to trigger the model to update the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy. This invention designs a multi-path routing based on a dynamic value function, effectively utilizing temporary resources through sub-flow splitting strategies, improving overall resource utilization, and achieving efficient and flexible resource management; the introduction of a random scenario triggering mechanism immediately triggers model updates when sudden events occur in the network, ensuring that the routing strategy quickly adapts to changes in network state, shortening response time, and improving the system's dynamic adaptability. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a preferred embodiment of a multi-path routing method provided by the present invention; Figure 2 This is a schematic diagram of a network routing model for random scenario-triggered reinforcement learning in a multi-path routing method provided by the present invention; Figure 3 This is a schematic diagram of periodic partitioning in a multi-path routing method provided by the present invention; Figure 4 This is a schematic diagram of multi-path routing in a multi-path routing method provided by the present invention; Figure 5 This is a schematic diagram of periodic transformation in a multi-path routing method provided by the present invention; Figure 6 This is a schematic diagram of a preferred embodiment of a multi-path routing device provided by the present invention; Figure 7 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 , Figure 1 This is a flowchart illustrating a preferred embodiment of a multi-path routing method provided by the present invention. The multi-path routing method includes: S1, Construct a network routing model for reinforcement learning triggered by random scenarios, wherein the network routing model defines the learning period and the stabilization period of the network operation; S2, during the learning period, the action value function is periodically updated according to the network state based on the reinforcement learning algorithm, and the multi-path sub-flow allocation ratio between the source node and the target node is calculated according to the updated action value function to determine the multi-path routing strategy. S3, during the stable period, based on the random scenario triggering mechanism, determine whether to trigger the model to update the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

[0018] Specifically, in this embodiment of the invention, a network routing model for reinforcement learning triggered by random scenarios is first constructed, which defines the learning period and the stabilization period of the network operation.

[0019] It should be noted that in traditional reinforcement learning (RL), the agent observes and acts on the environment, and the environment provides rewards, completing each learning step periodically. This cyclical process is key to reinforcement learning; through continuous trial and error and adjustment, the agent gradually learns to acquire the optimal behavioral strategy in a specific environment. However, the stochastic scenario-triggered reinforcement learning provided in this embodiment of the invention differs from this approach; its learning process is intermittent. Please refer to... Figure 2 , Figure 2This is a schematic diagram of a network routing model for random scenario-triggered reinforcement learning in a multi-path routing method provided by this invention. In the network routing model of random scenario-triggered reinforcement learning in this embodiment of the invention, the state and reward are still periodic, that is, the agent receives the current state and corresponding reward signal from the environment at each time step. However, unlike traditional reinforcement learning, the actions of this model become non-periodic, and action transitions are achieved through random scenario events. The value of this change is that non-periodic action does not mean that the agent sometimes does not act. On the contrary, it means that the agent does not perform policy search, but directly uses the previous action. This strategy aims to reduce the computational burden. After all, in practical applications, frequent policy search may consume a lot of computing resources and increase the communication burden between nodes. By reducing the policy search frequency, the model can reduce computational costs while maintaining performance. It is worth noting that the fluctuations in the network environment and the fluctuations in the reinforcement learning situation coexist, which brings challenges to the model. To minimize computational costs, updates need to be triggered randomly, and periodic action transitions need to be performed when the overall situation is unstable to ensure training efficiency. Based on this, this embodiment of the invention, taking into account the characteristics of network routing and the need for continuous learning of the value function of actions in the early stages, divides the network operation period into a learning period and a stable period. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of periodic partitioning in a multi-path routing method provided by the present invention.

[0020] During the learning phase, the network routing model periodically transitions its behavior without triggering events. Based on reinforcement learning algorithms, the action value function is periodically updated according to the network state. The updated action value function is then used to calculate the multi-path sub-flow allocation ratio between the source and target nodes, determining the multi-path routing strategy. The purpose of this stage is to allow the network routing model to build a relatively accurate action value function table through continuous trial and adjustment. This table records the rewards for taking different actions under various states. This stage is indispensable because the action value function table is a core element in reinforcement learning algorithms, guiding the agent's decision-making.

[0021] During the stable period, random scenario triggering is activated. Based on this mechanism, it is determined whether to trigger a model update of the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy. At this stage, the network routing model has established a relatively reliable action value function table through the learning period, so frequent policy searches are no longer necessary. When the network environment is relatively stable, i.e., no sudden events occur, the triggering conditions are not met, and the agent does not need to perform a policy search, continuing to execute the current action. This strategy reduces computational resource consumption while maintaining stable network operation. However, when sudden events occur in the network, such as node failures, link interruptions, or traffic surges, these events can significantly impact network performance, requiring the agent to react promptly. In this case, the triggering conditions are met, and the agent will perform a policy search and update. In this way, the agent can maintain network stability while responding quickly to sudden events, thereby improving network reliability and robustness.

[0022] This invention divides the network operation period into a learning period and a stable period, and combines this with a random scenario triggering mechanism to achieve efficient learning and stable operation in network routing. This method not only improves network performance and reliability but also reduces computational resource consumption, ensures that routing strategies quickly adapt to changes in network state, shortens response time, and enhances the system's dynamic adaptability, providing a novel solution for network routing. Furthermore, this invention designs a multi-path routing system based on a dynamic value function, effectively utilizing temporary resources through a sub-flow splitting strategy, improving overall resource utilization, and achieving efficient and flexible resource management.

[0023] In another preferred embodiment, step S1, constructing a network routing model for random scenario-triggered reinforcement learning, includes: S11, Define the state space, action space and reward function of the network node, and configure the random scenario triggering mechanism, which divides the network operation process into the learning period and the stable period; S12, initialize the action value function table, where the row index in the action value function table represents the target node and the column index represents the next hop node.

[0024] Specifically, in constructing a network routing model for reinforcement learning triggered by random scenarios, this embodiment of the invention defines the state space, action space, and reward function of network nodes, and configures a random scenario triggering mechanism. This mechanism divides the network operation process into a learning period and a stabilization period. The state space includes network topology and traffic characteristics; the action space includes path selection and sub-flow splitting ratios; and the reward function includes latency and bandwidth utilization. Subsequently, an action value function table is initialized, where the row index represents the target node and the column index represents the next-hop node.

[0025] For example, this embodiment of the invention employs the Q-Learning algorithm to implement multi-path routing. During the learning period, the Q-table is quickly determined through periodic updates, ensuring the model possesses basic decision-making capabilities. During the stabilization period, a random triggering mechanism significantly reduces unnecessary computation. Each network node is configured with a Q-table as a decision agent, recording the state-action value from that node's perspective, and forwarding traffic according to the network routing model. Please refer to... Figure 4 Figure 4 is a schematic diagram of multi-path routing in the multi-path routing method provided by the present invention. In the Q table, the row index represents the state, i.e., the target node, and the column index represents the action, i.e., the next hop node that the forwarding task can choose at that node. During the stable period and when no model update is triggered, the node only queries the local Q table for forwarding and does not perform policy search; when a model update is triggered, the node only updates the local routing entries affected by network events, rather than updating the entire table.

[0026] In another preferred embodiment, the network routing model enters the learning period by default in the initial stage of operation; when the fluctuation value of the maximum cumulative reward is detected to be less than the preset convergence threshold for N consecutive periods, the network routing model is determined to have converged and automatically switches to the stable period.

[0027] Specifically, this invention designs a random scenario triggering mechanism for network routing scenarios. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of periodic transformation in a multi-path routing method provided by the present invention. Figure 5 In this model, the maximum cumulative reward reflects the interaction between reinforcement learning and the network environment. Based on this metric, the entire network operation period is divided into two phases. The first is the learning phase: the network environment is unstable, and reinforcement learning has not yet learned a stable policy. During this phase, each learning step needs to be completed periodically. The second is the stable phase: the maximum cumulative reward fluctuates smoothly, indicating that the reinforcement learning has converged and the interaction with the network environment is stable. During this phase, model updates are triggered randomly to reduce computational costs.

[0028] In yet another preferred embodiment, the step of calculating the multipath sub-flow allocation ratio between the source node and the target node based on the updated action value function includes: The row vector corresponding to the target node is determined based on the updated action value function table; Based on the action value of each next-hop node in the row vector, calculate the subflow allocation ratio to each next-hop node.

[0029] Specifically, in this embodiment of the invention, when a transmission task needs to be forwarded upon arrival at node V1, the corresponding path splitting ratio is, for example... Figure 4 As shown in the table, rows represent target nodes, and columns represent the flow distribution ratio between a node and its neighboring nodes. Rows can be selected using the target nodes, and then the Q-value for each neighboring node is determined. If node V1 is the source node of the transmission task, the sub-flow allocation ratio for forwarding the transmission task to each neighboring node is calculated based on the Q-values ​​of each neighboring node. It should be noted that if node V1 is the forwarding node of the transmission task, the corresponding next-hop node is selected from several neighboring nodes using an ε-greedy strategy. In this way, the number of sub-paths and the sub-flow allocation ratio between the source and target node pairs can be well determined, thus obtaining a multi-path routing and forwarding method.

[0030] This invention presents a Q-value-based multi-path routing design. By finely modeling path selection and sub-flow splitting strategies, and defining corresponding reward functions and action spaces, the model can effectively utilize temporary resources such as idle bandwidth and idle nodes, thereby improving the overall utilization efficiency of network resources.

[0031] In another preferred embodiment, the step of determining whether to trigger a model update of the action value function and recalculate the multi-path sub-flow allocation ratio based on a random scenario triggering mechanism to optimize the multi-path routing strategy includes: Real-time monitoring of the rate of change of network status parameters; If the rate of change does not exceed the preset random scenario trigger threshold, then the current multi-path sub-flow allocation ratio is directly applied for traffic forwarding. If the rate of change exceeds a preset random scenario trigger threshold, the model is triggered to update the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

[0032] Specifically, in this embodiment of the invention, after the agent completes observation, it can determine whether an event needs to be triggered based on the rate of change between the previous observation and the current observation. For example, the rate of change of network state parameters in real time can be used to determine whether a sudden event has occurred. If the rate of change of network state parameters does not exceed a preset random scenario triggering threshold, it indicates that no sudden event has occurred, the network environment is not affected by external interference, and it can usually operate smoothly as expected. In this case, there is no need to traverse the routing strategy, and the current multi-path sub-flow allocation ratio can be directly applied for traffic forwarding. If the rate of change of network state parameters exceeds the preset random scenario triggering threshold, it indicates that a sudden event has occurred, such as node failure, link interruption, or traffic burst. At this time, the model is triggered to update the action value function, and the multi-path sub-flow allocation ratio between the source node and the target node is recalculated based on the updated action value function to optimize the multi-path routing strategy.

[0033] This invention proposes a random scenario triggering mechanism. When specific events occur in the network, such as node joining or leaving, link failure recovery or disconnection, or traffic surges or declines, a model update is immediately triggered. This mechanism ensures that routing policies quickly adapt to changes in network conditions. This mechanism enables the intelligent routing system to respond in real time to any significant changes in network conditions, greatly shortening the response time for policy adjustments, significantly enhancing the system's dynamic adaptability, and ensuring that efficient and stable routing performance is maintained even in the face of complex and rapidly changing network environments.

[0034] As a preferred embodiment, the action value function is updated using the Q-Learning algorithm, and the update formula for the action value function is: ; In the formula, Indicates the first i After +1 iterations of update, the current node j The target node is k The value of the action; Indicates the first i Before the next iteration update, the current node j The target node is k The historical value of this action; α Indicates the learning rate; r This represents the immediate reward received by traffic after it is forwarded at a node; γ Indicates the discount factor; n Indicates the current node j Candidate next-hop nodes; F ( j ) represents the current node j The set of all available next-hop nodes; Indicates the current nodej Among all available next-hop nodes, select the action value with the minimum cost / optimal forwarding overhead; Indicates the current node j The target node in the Q table is k And the next hop node is n The value of the action.

[0035] Specifically, in this embodiment of the invention, the action value function is updated using the Q-Learning update rule, and the update formula for the action value function is: ; In the formula, Indicates the first i After +1 iterations of update, the current node j The target node is k The action value, i.e., Q value; Indicates the first i Before the next iteration update, the current node j The target node is k The historical value of action, i.e., the old Q value; α The learning rate is used to control the weight of the new reward on the old Q value, and its value ranges from 0 to 1. α ≤1; r This represents the immediate reward received by traffic after it is forwarded at a node; γ This represents the discount factor, used to measure the importance of future long-term returns relative to current immediate rewards, with a value ranging from 0 to 1. γ ≤1; n Indicates the current node j Candidate next-hop neighbor nodes; F ( j ) represents the current node j The set of all available next-hop neighbor nodes; Indicates the current node j Among all available next-hop neighbor nodes, select the action value with the minimum cost / optimal forwarding overhead; Indicates the current node j The target node in the Q table is k And the next hop node is n The value of the action.

[0036] This invention addresses the problems of wasted network resources and insufficient redundancy in existing technologies by designing a Q-value-based multi-path routing system. Through detailed modeling of path selection and sub-flow splitting strategies, the model effectively utilizes temporary resources such as idle bandwidth and idle nodes, thereby improving the overall utilization efficiency of network resources. Furthermore, based on reinforcement learning, a stochastic scenario triggering mechanism is introduced to optimize for specific events occurring during the stable period of the network environment. This significantly shortens the response time for policy adjustments, substantially enhances the system's dynamic adaptability, and reduces corresponding computational overhead.

[0037] Accordingly, the present invention also provides a multi-path routing apparatus capable of implementing all the processes of the multi-path routing method in the above embodiments.

[0038] Please see Figure 6 , Figure 6 This is a schematic diagram of a preferred embodiment of a multi-path routing device provided by the present invention. The multi-path routing device includes: The model building module 601 is used to build a network routing model for reinforcement learning triggered by random scenarios. The network routing model defines the learning period and the stabilization period of the network operation. The strategy learning module 602 is used to periodically update the action value function based on the network state according to the reinforcement learning algorithm during the learning period, and calculate the multi-path sub-flow allocation ratio between the source node and the target node according to the updated action value function to determine the multi-path routing strategy. The routing execution module 603 is used to determine whether to trigger the model to update the action value function based on a random scenario triggering mechanism during the stable period, and to recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

[0039] Preferably, the model building module 601 is specifically used for: Define the state space, action space, and reward function of the network nodes, and configure the random scenario triggering mechanism, which divides the network operation process into the learning period and the stable period; Initialize the action value function table, where the row index in the action value function table represents the target node and the column index represents the next hop node.

[0040] Preferably, in the initial stage of operation, the network routing model defaults to the learning period; when the fluctuation value of the maximum cumulative reward is detected to be less than the preset convergence threshold for N consecutive periods, the network routing model is determined to have converged and automatically switches to the stable period.

[0041] Preferably, the step of calculating the multipath subflow allocation ratio between the source node and the target node based on the updated action value function includes: The row vector corresponding to the target node is determined based on the updated action value function table; Based on the action value of each next-hop node in the row vector, calculate the subflow allocation ratio to each next-hop node.

[0042] Preferably, the step of determining whether to trigger a model update of the action value function and recalculate the multi-path sub-flow allocation ratio based on a random scenario triggering mechanism to optimize the multi-path routing strategy includes: Real-time monitoring of the rate of change of network status parameters; If the rate of change does not exceed the preset random scenario trigger threshold, then the current multi-path sub-flow allocation ratio is directly applied for traffic forwarding. If the rate of change exceeds a preset random scenario trigger threshold, the model is triggered to update the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

[0043] Preferably, the action value function is updated using the Q-Learning algorithm, and the update formula for the action value function is: ; In the formula, Indicates the first i After +1 iterations of update, the current node j The target node is k The value of the action; Indicates the first i Before the next iteration update, the current node j The target node is k The historical value of this action; α Indicates the learning rate; r This represents the immediate reward received by traffic after it is forwarded at a node; γ Indicates the discount factor; n Indicates the current node j Candidate next-hop nodes; F ( j ) represents the current node j The set of all available next-hop nodes; Indicates the current node j Among all available next-hop nodes, select the action value with the minimum cost / optimal forwarding overhead; Indicates the current node j The target node in the Q table is k And the next hop node is n The value of the action.

[0044] In specific implementation, the working principle, control process and technical effects of the multi-path routing device provided in the embodiments of the present invention are the same as those of the multi-path routing method in the above embodiments, and will not be repeated here.

[0045] Please see Figure 7 , Figure 7 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 701, a memory 702, and a computer program stored in the memory 702 and configured to be executed by the processor 701. When the processor 701 executes the computer program, it implements the multi-path routing method described in any of the above embodiments.

[0046] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 702 and executed by the processor 701 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0047] The processor 701 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 701 can be any conventional processor. The processor 701 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0048] The memory 702 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 702 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, and a flash card, etc., or the memory 702 can also be other volatile solid-state storage devices.

[0049] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 7 The structural diagram is merely an example of the terminal device described above and does not constitute a limitation on the terminal device described above. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components.

[0050] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the multi-path routing method described in any of the above embodiments.

[0051] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the multi-path routing method described in any of the above embodiments.

[0052] This invention provides a multi-path routing method, apparatus, device, storage medium, and computer program product. It constructs a network routing model based on random scenario-triggered reinforcement learning, defining a learning period and a stable period for network operation. During the learning period, an action value function is periodically updated based on the network state using a reinforcement learning algorithm. The multi-path sub-flow allocation ratio between the source node and the target node is calculated based on the updated action value function to determine the multi-path routing strategy. During the stable period, a random scenario triggering mechanism determines whether to trigger the model to update the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy. This invention designs a multi-path routing based on a dynamic value function. Through sub-flow allocation strategies, it effectively utilizes temporary resources, improves overall resource utilization, and achieves efficient and flexible resource management. The introduction of a random scenario triggering mechanism immediately triggers model updates when sudden events occur in the network, ensuring that the routing strategy quickly adapts to changes in network state, shortening response time, and improving the system's dynamic adaptability.

[0053] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0054] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-path routing method, characterized by, include: A network routing model for reinforcement learning triggered by random scenarios is constructed, wherein the network routing model defines the learning period and the stabilization period of the network operation; During the learning period, the action value function is periodically updated based on the network state using a reinforcement learning algorithm, and the multi-path sub-flow allocation ratio between the source node and the target node is calculated based on the updated action value function to determine the multi-path routing strategy. During the stable period, a random scenario triggering mechanism is used to determine whether to trigger the model to update the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

2. The multi-path routing method of claim 1, wherein, The construction of a network routing model for reinforcement learning triggered by random scenarios includes: Define the state space, action space, and reward function of the network nodes, and configure the random scenario triggering mechanism, which divides the network operation process into the learning period and the stable period; Initialize the action value function table, where the row index in the action value function table represents the target node and the column index represents the next hop node.

3. The multi-path routing method as described in claim 2, characterized in that, In the initial stage of operation, the network routing model defaults to the learning period; when the fluctuation value of the maximum cumulative reward is detected to be less than the preset convergence threshold for N consecutive periods, the network routing model is determined to have converged and automatically switches to the stable period.

4. The multi-path routing method as described in claim 3, characterized in that, The step of calculating the multipath sub-flow allocation ratio between the source node and the target node based on the updated action value function includes: The row vector corresponding to the target node is determined based on the updated action value function table; Based on the action value of each next-hop node in the row vector, calculate the subflow allocation ratio to each next-hop node.

5. The multi-path routing method as described in claim 4, characterized in that, The step of determining whether to trigger a model update of the action value function and recalculate the multi-path sub-flow allocation ratio based on a random scenario triggering mechanism to optimize the multi-path routing strategy includes: Real-time monitoring of the rate of change of network status parameters; If the rate of change does not exceed the preset random scenario trigger threshold, then the current multi-path sub-flow allocation ratio is directly applied for traffic forwarding. If the rate of change exceeds a preset random scenario trigger threshold, the model is triggered to update the action value function and recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

6. The multi-path routing method as described in claim 5, characterized in that, The action value function is updated using the Q-Learning algorithm, and the update formula for the action value function is: ； In the formula, Indicates the first i After +1 iterations of update, the current node j The target node is k The value of the action; Indicates the first i Before the next iteration update, the current node j The target node is k The historical value of this action; α Indicates the learning rate; r This represents the immediate reward received by traffic after it is forwarded at a node; γ Indicates the discount factor; n Indicates the current node j Candidate next-hop nodes; F ( j ) represents the current node j The set of all available next-hop nodes; Indicates the current node j Among all available next-hop nodes, select the action value with the minimum cost / optimal forwarding overhead; Indicates the current node j The target node in the Q table is k And the next hop node is n The value of the action.

7. A multi-path routing device, characterized in that, include: The model building module is used to build a network routing model for reinforcement learning triggered by random scenarios. The network routing model defines the learning period and the stabilization period of the network operation. The strategy learning module is used to periodically update the action value function based on the network state according to the reinforcement learning algorithm during the learning period, and calculate the multi-path sub-flow allocation ratio between the source node and the target node according to the updated action value function to determine the multi-path routing strategy. The routing execution module is used to determine whether to trigger the model to update the action value function based on a random scenario triggering mechanism during the stable period, and to recalculate the multi-path sub-flow allocation ratio to optimize the multi-path routing strategy.

8. A terminal device, characterized in that, The system includes a processor and a memory, the memory storing a computer program configured to be executed by the processor, the processor executing the computer program to implement the multipath routing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the multi-path routing method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions that, when executed by a processor, implement the multipath routing method as described in any one of claims 1 to 6.

Images