Heterogeneous dynamic network-oriented adaptive traffic scheduling method and device

Abstract

The invention provides a self-adaptive traffic scheduling method and device for a heterogeneous dynamic network. The method comprises two core modules, namely an adaptive path selector and a flow priority manager: the adaptive path selector dynamically adjusts a path scheduling strategy by adopting a reinforcement learning method based on network state indexes, such as round-trip delay, packet loss rate and congestion window of a path, and explores a low-delay path in combination with a greedy method so as to avoid falling into local optimum; the flow priority manager dynamically distributes the data sending amount in each round of scheduling according to the flow priority proportion and the sending window, and in addition, after the high-priority flow is sent, unconfirmed data packets with the same priority are preferentially scheduled, so that the transmission continuity of the key service flow is guaranteed; the bandwidth utilization rate of multi-path transmission and the transmission reliability of key services can be improved, and the method adapts to dynamic changes in a complex network environment.

Description

Technical Field

[0001] The present invention relates to the technical field of wireless communication equipment, and in particular to an adaptive traffic scheduling method and device for heterogeneous dynamic networks. Background Art

[0002] With the rapid development of the internet and multimedia services, more and more network applications are placing higher demands on transmission latency, such as web page loading, high-definition video transmission, virtual reality, and cloud gaming. To improve web page loading performance, mechanisms such as multi-stream multiplexing, header compression, and server push have been introduced, achieving some success in reducing the number of connections and protocol overhead. Multi-stream multiplexing technology supports the simultaneous transmission of multiple resource streams over a single connection and prioritizes important resource streams based on a priority mechanism, effectively improving page loading performance.

[0003] To further enhance the low-latency service experience, Google proposed a solution with advantages such as low handshake latency, support for multi-stream multiplexing, and end-to-end encryption. Its extended version, the Multipath Protocol, aggregates bandwidth and improves throughput by using multiple available paths simultaneously, making it one of the important transmission protocols in heterogeneous wireless network environments. However, existing multipath scheduling methods still face performance bottlenecks in complex dynamic environments. First, existing methods do not fully utilize the priority mechanism of multi-stream multiplexing, resulting in uneven resource allocation between multiple concurrent streams and causing congestion and blocking between streams. Second, due to significant differences in the states of multiple paths, data packets of the same data stream may be received out of order, resulting in increased buffer pressure on the end side and increased overall transmission delay.

[0004] Existing technologies have attempted to optimize the path selection process by introducing methods such as reinforcement learning and path scoring. However, most methods still focus on the path dimension as the core scheduling unit and lack comprehensive consideration of flow-level priorities. This makes it difficult to strike a balance between ensuring the real-time performance of different flows and overall scheduling efficiency in highly dynamic environments. Furthermore, existing re-injection methods suffer from scheduling discontinuities and high recovery delays when high-priority flows are lost, impacting the transmission continuity of high-priority services.

[0005] Therefore, the existing technology still needs to be improved in the multi-flow and multi-path scheduling scenario. There is an urgent need for an efficient multi-path scheduling solution that is oriented towards heterogeneous dynamic network environments and has flow-level differentiated scheduling capabilities, so as to take into account the multi-flow multiplexing characteristics and path state differences, and achieve balanced optimization of transmission guarantee of high-priority flows and overall performance. Summary of the Invention

[0006] The purpose of the present invention is to provide an adaptive traffic scheduling method and device for heterogeneous dynamic networks, characterized in that the device is suitable for heterogeneous and dynamic network environments.

[0007] In a first aspect, the present invention provides a multipath scheduling device, comprising:

[0008] The adaptive path selector module is used to select a valid path set from the currently available paths, and dynamically adjust the path selection strategy based on the network status parameters such as the round-trip delay, packet loss rate and congestion window of each path using the reinforcement learning method; in order to avoid local optimality, the greedy strategy is further combined to explore new paths with minimum delay; the adaptive path selector module further includes a path evaluation submodule, which is used to construct a path based on multiple dimensions such as the round-trip delay, packet loss rate, congestion window and path bandwidth.

[0009] The flow priority manager module is used to dynamically allocate the data transmission volume of each flow in each scheduling cycle according to the priority weight and sending window size of each flow; and after the high-priority flow is sent, it gives priority to processing unconfirmed data packets with the same priority to improve the continuity and real-time performance of key business flows; the flow priority manager module includes a flow status monitoring unit and a window scheduling unit. The flow status monitoring unit is used to monitor the congestion window and sending rate of each business flow, and the window scheduling unit dynamically adjusts the allocation rate according to the priority ratio of the flow, and supports scheduling the unconfirmed packets with the same priority immediately after the high-priority flow is completed to reduce interruption delay.

[0010] The device is adapted to the multi-stream multiplexing mechanism, the path selection and priority scheduling strategy can be executed independently among multiple sub-connections, and the scheduling result can be bound to the transmission path based on the connection identifier, thereby realizing the decoupling of multi-path scheduling and multi-stream priority control.

[0011] The adaptive path selector module further includes a path exploration submodule and a path score dynamic update submodule; the path exploration submodule is used to comprehensively evaluate path quality based on latency, congestion window, and packet loss rate using a greedy strategy and reinforcement learning feedback, and to explore and activate new paths with better performance; the path score dynamic update submodule is used to update the evaluation value of each path based on a reinforcement learning strategy, and the evaluation value changes dynamically with the path status to respond to dynamic changes in heterogeneous network environments in real time, thereby improving the dynamic adaptability of path selection and bandwidth utilization.

[0012] The flow priority manager module further includes a data flow mapping unit and a flow granularity adjustment unit; the data flow mapping unit is used to parse the data packets transmitted by the protocol and map them to the corresponding priority queues according to the priority information marked on the data packets; the flow granularity adjustment unit dynamically adjusts the data sending granularity of each priority queue based on the real-time monitored network round-trip delay and the sending status of each flow. When the length of the high-priority queue is less than a given threshold, the data sending frequency of the high-priority queue is increased, thereby reducing the waiting delay of critical business data packets; the data flow mapping unit and the flow granularity adjustment unit jointly realize differentiated scheduling of data packets of different priorities, thereby meeting the real-time transmission requirements of delay-sensitive services.

[0013] A second aspect of the present invention further provides a multi-path scheduling method based on reinforcement learning, comprising the following steps:

[0014] Obtain network status information, including transmission round-trip delay, packet loss rate, congestion window and other indicators of each path;

[0015] Scoring paths based on reinforcement learning strategy and selecting the optimal path set in combination with greedy strategy;

[0016] Dynamically allocate data transmission volume based on the priority and sending window of each flow;

[0017] After the high-priority flow is sent, the unacknowledged packets of the same priority are scheduled first;

[0018] Bind the flow scheduling results to the protocol sub-connection to achieve multi-path and multi-flow differentiated transmission.

[0019] According to a third aspect of the present invention, a computer-readable storage medium is provided, comprising a computer program, wherein the computer program is configured to execute the multipath scheduling method according to claim 6 when running on one or more processors.

[0020] In a fourth aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, and a computer program; wherein the computer program is stored in the memory and executed by the processor to execute the operating instructions of the multipath scheduling method as claimed in claim 6. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 Schematic diagram of a dependency tree example according to an embodiment of the present invention.

[0023] Figure 2 Schematic diagram of a weighted round-robin multi-flow scheduling method according to an embodiment of the present invention.

[0024] Figure 3 Schematic diagram of the framework of the multi-flow priority differentiated multi-path scheduling method according to an embodiment of the present invention.

[0025] Figure 4 Schematic diagram of a valid path set according to an embodiment of the present invention.

[0026] Figure 5Schematic diagram of end-to-end delay in an embodiment of the present invention.

[0027] Figure 6 Schematic diagram of a priority-based data packet scheduling method according to an embodiment of the present invention.

[0028] Figure 7 Schematic diagram of the original re-injection strategy according to an embodiment of the present invention.

[0029] Figure 8 Schematic diagram of the improved re-injection strategy according to an embodiment of the present invention. Figure 9 This is a diagram showing the overall architecture of an adaptive traffic scheduling method for heterogeneous dynamic networks according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is merely an illustrative description, intended to help understand the technical content of the present invention, and does not constitute a limitation on the scope of protection of the present invention.

[0031] This embodiment provides an adaptive traffic scheduling method and device for heterogeneous and dynamic networks. This method is suitable for heterogeneous and dynamic network environments, particularly for latency-sensitive multi-flow services. The device is designed to jointly optimize two core issues: path dynamics and flow priority. It has the following features:

[0032] (1) It can dynamically select efficient path combinations for parallel transmission based on the real-time status of the network environment, improve transmission stability and utilization, and adapt to dynamic characteristics such as path packet loss and congestion window changes;

[0033] (2) Supports multi-stream priority scheduling, adopts round-robin scheduling and sending window allocation mechanism, dynamically adjusts the sending order according to the priority weight of the flow, and prioritizes the scheduling of unconfirmed packets after the high-priority flow is completed to ensure the continuity of critical services;

[0034] (3) By integrating reinforcement learning with greedy strategies, we can evaluate the round-trip delay, packet loss rate, congestion window and other indicators of each path online, and select the optimal path set through the path scoring function;

[0035] (5) Adopting an extensible structure, it supports the introduction of other traffic prediction mechanisms and path cost modeling modules in the future to further improve the intelligence and adaptability of the system.

[0036] In a specific implementation, the multipath scheduling device includes the following modules:

[0037] Network status collection module: used to collect key indicators such as round-trip transmission delay, congestion window, packet loss rate, etc. of multiple paths; path scoring and selection module: evaluates path scores and selects path sets based on reinforcement learning and greedy strategies; multi-flow priority management module: dynamically allocates sending windows and data volume to each flow in rounds, and provides a priority re-injection mechanism for key business flows; protocol adaptation module: used to map scheduling decision results to the protocol layer to realize path selection and multi-flow scheduling results.

[0038] The system can be deployed on terminals or edge nodes and is suitable for high-real-time scenarios such as high-definition video conferencing, telemedicine, and virtual reality interaction. It can effectively improve the multi-stream data transmission performance and network service quality assurance capabilities in heterogeneous network environments.

[0039] The scheduler of this embodiment consists of two components: an adaptive path selector based on network heterogeneity and dynamics, and a priority-based flow manager. The overall schematic diagram of this embodiment is shown in the accompanying abstract. The flow manager determines which flows should be prioritized based on scheduling rules and, based on priority and send window size constraints, optimizes network resource allocation and reduces congestion between flows.

[0040] For each data packet, the path selector first explores paths based on minimum end-to-end latency. After initially evaluating the path's value, it uses reinforcement learning to evaluate each path's value, combining characteristics such as round-trip latency, packet loss rate, congestion window size, and send window size. Based on the current path status, the path with the highest value is prioritized for sending the packet, maximizing network throughput while ensuring that packets arrive in order.

[0041] In order to effectively reflect the priorities of different streams during multi-stream parallel transmission, the stream manager adopts a dependency tree structure similar to that widely used in current scheduling methods, such as Figure 1 As shown in Figure 2. In this structure, each flow generated by a connection is defined as a node on the tree. Before being scheduled, each flow is assigned a priority based on different requirements. Dependencies can be defined between this flow and other flows, similar to a parent-child hierarchy. The transmission rule is that the transmission of a child node can only begin after the parent node completes. Furthermore, the flow priority determines how available resources are allocated to different sibling nodes within the same parent node, ensuring that higher-priority flows receive priority transmission opportunities.

[0042] The weighted round-robin method stipulates that each flow sends a number of packets equal to its priority level during its scheduling period. However, this scheduling strategy is not always optimal. If a flow completely fills the sending window, critical flows with higher priority may be blocked. Figure 2 An example illustrates this situation.

[0043] Assume that there are three flows with different priorities in the system. Set the priority of flow A to 2, the priority of flow B to 6, and the priority of flow C to 4. Set the size of the sending window to accommodate a maximum of 6 data packets. Although critical flows with higher priorities should be given priority, traditional scheduling methods do not strictly transmit according to the priority of the flows. In this scheduling process, although flow B has a higher priority and flow C has a higher priority than flow A, flow A is transmitted first. This will cause the lower-priority flows to block the more critical flows B and C. This blocking phenomenon will affect the transmission efficiency of the network, cause inter-flow interference problems, and may increase transmission delays. This paper proposes an improved flow scheduler to alleviate the inter-flow blocking problem in multi-flow services. The scheduler has the following key functions:

[0044] (1) Guaranteeing priority: The transmission of flows is strictly ordered according to priority, ensuring that flows with higher priorities are scheduled first to ensure the transmission efficiency of key flows.

[0045] (2) Ensure fairness: avoid blocking between multiple flows, especially reduce the mutual interference between flows of different priorities and reduce the possibility of a single flow occupying the entire sending window.

[0046] (3) Ensure the continuity of key flows: Ensure that the data transmission of high-priority flows maintains continuity first to avoid interference from the transmission of low-priority flows in a multi-flow environment.

[0047] After the flow manager selects a data flow to send, the path selector will decide which path the data packets of the flow will be transmitted through based on the current real-time status of the network.

[0048] The path selector primarily leverages the autonomous learning capabilities of reinforcement learning to update the Q-table in real time. During the exploration phase, the scheduler tracks parameters such as the current round-trip delay and packet loss for each network path. The path selector estimates the end-to-end delay of a path based on these parameters and prioritizes the path with the lowest end-to-end delay. After initially evaluating the path's value during the exploration phase, the Q-learning method further evaluates the value of each path, combining characteristics such as round-trip delay, packet loss rate, congestion window size, and send window size.

[0049] Based on the current path status, the path with the highest value is selected to send data packets, maximizing network throughput while ensuring that data packets are delivered as sequentially as possible.

[0050] The following first gives the learning model of the reinforcement learning method involved in this implementation. The model mainly consists of the environment, the agent, the state space, the action space and the reward function. Figure 3 shown.

[0051] Environment: The research environment for this method design is a heterogeneous wireless mesh network.

[0052] Agent: Responsible for determining the optimal path to transmit data packets. Its core goal is to improve system performance by optimizing the reward function.

[0053] State space: The state of the agent contains all relevant decision-making information and reflects the real-time state of the network. In view of the high dynamics of paths in heterogeneous wireless mesh networks, this model considers the round-trip delay, congestion window, packet loss rate, etc. of the path in the state space. At time t, the state of n paths can be expressed as: S t =(s t,1 ,s t,2 ,...,s t,n ).

[0054] where s t,i (1≤i≤n) represents the characteristics of the i-th path, which can be expressed as: s t,i =(r t,i ,w t,i ,n t,i ,l t,i ).

[0055] The meaning of each parameter in the expression is as follows: t,i is the average round trip delay of the i-th path at time t; w i,t It refers to the congestion window size of the i-th path at time t; n i,t is the sending window size of the i-th path at time t. i,t is the number of packet losses on the ith path at time t;

[0056] Action space: The agent selects the path to transmit data based on the current status of different paths. This method considers defining the action space as a one-dimensional vector. Each action defined in the vector corresponds to selecting a path from the optimal path set to transmit data, where a i (1≤i≤n), A=(a1,a2,...,a n ).

[0057] Reward function: At time t, the agent observes the state s of the current path t , take corresponding action a according to the state t When the agent selects a path for data transmission, the state of the path will change and enter the next stage s t+1 And receive a corresponding reward value.

[0058] The reward function must be designed to reflect the target requirement. The selected path should increase the throughput H t, reduce packet loss rate loss t with RTT t , define the reward function including the above items as: R(s t ,a t )=(1-α-β-χ)·H t -α·RTT t -β·loss t -χ·cost

[0059] By introducing the first term of the reward function to encourage maximizing throughput, we introduce -αRTT t Minimize packet transmission delay by introducing -βloss t To reduce packet loss and network congestion, and considering that different paths in heterogeneous network environments may have significantly different costs, the reward function penalizes high-cost paths through the negative impact of cost, thereby guiding the scheduling method to select paths with good performance and low cost. 0 < α, β, χ < 1 are adjustable weights.

[0060] In heterogeneous wireless mesh networks, multipath parameters may vary significantly. Selecting some of the optimal transmission paths for transmission can achieve better overall performance than using all paths for transmission. Therefore, we first propose a scheme for selecting the optimal path set for parallel transmission and determine the period for path set selection.

[0061] In multipath transmission over wireless networks, path set selection is a critical step in ensuring transmission quality. First, the paths with the best performance and that meet transmission requirements are selected. These paths form the so-called "valid path set." Considering the highly dynamic nature of wireless networks, the path set is periodically updated to adapt to changes in the network environment.

[0062] In this implementation, the path quality assessment is first performed based on factors such as network delay, bandwidth, and packet loss rate between the sender and receiver, combined with real-time transmission data, to comprehensively evaluate the performance of each path. Suppose the path set in the heterogeneous wireless mesh network is U = {P1, P2, ..., P N}, select m paths from them and add them to the valid path set, path P i Path quality Q i Using the formula Q i =psize / B i +FT i Indicates. Among them, psize represents the packet size, B i represents the bandwidth of path i, and the transmission delay of the data packet is calculated by these two items. i To express the forward transmission delay of available path i, that is, the delay of the receiver receiving the data packet, in this scheme, the formula FTi =RTT i / 2 estimation. The above calculation is used to estimate the transmission quality Q of a certain path. i , when Q i When it is smaller, the end-to-end transmission delay is shorter.

[0063] After obtaining the required path quality through the above calculation, sort them from high quality to low quality. If the sorted result is Q1≤Q2≤...≤Q n , which means that the performance of path P1 is better.

[0064] In addition to latency performance, the selected path must also support the transmission rate requirements of the upper-layer application. Assume that the required data transmission rate is v and the number of valid paths is m (1≤m≤N). The following calculation should be met: CWND in formulas i / RTT i Represents path P i The instantaneous transmission rate of m paths at this time is the minimum number of paths that can meet the transmission requirements. Figure 4 As shown, P={P1,P2,...P m The paths represented by solid lines in the figure represent the minimum set of paths that can meet the transmission requirements of the sender, and the remaining available path sets are represented by dotted lines.

[0065] In real-world networks, selecting an effective path set isn't a one-time task; it's a dynamic process. Due to changes in the network environment and link status, the path set needs to be regularly updated. The length of the update cycle depends on the rate of change in path quality. If path quality changes rapidly, the update cycle should be shortened to allow for timely adjustment of the effective path set. Conversely, if path quality changes slowly, the update cycle can be appropriately extended to reduce the frequency of path set updates, thereby reducing network overhead.

[0066] Similar to the above, the path set of available paths is set to U = {P1, P2, ..., P N In order to avoid the path selection set update cycle being too short, the minimum update cycle is set to be no less than the data packet transmission cycle. Let the round trip time of the data packet be RTT, and min(RTT i ) is defined as the end period of the updated path set. For the convenience of analysis, it is considered as the initial period. The initial period is defined as UT0 = min(RTT i ).

[0067] As analyzed above, the length of the update cycle needs to be determined according to the speed of change of the path quality. Suppose path P iThe path quality change of the kth time compared with the last time, that is, the (k-1)th time is represented by , and its calculation formula is For the path set U={P1, P2,..., P N}, it is defined that is the average path quality change, and its formula is

[0068] Therefore, the effective path set update period of the kth time can be represented as: UT k =max(UT0, aUT k-1 ),

[0069] α represents the period change coefficient of the kth time compared with the (k-1)th time of path quality. When , the path quality variable is very stable, and at this time, α=2, the update period of the path set is doubled to reduce the overhead brought by the update. When becomes larger, it means that the degree of path quality change is increasing, α becomes smaller, and the update period becomes shorter, so as to reflect the change of path quality more timely.

[0070] The following is an example to illustrate the process of periodic update. When , that is, the current path quality changes by 10% compared with the last path quality, α=1, and the previous update period is maintained; when is very large, α→0, and the update period tends to 0 at this time. In order to avoid frequent updating of the path set, the period is set to the minimum period UT0.

[0071] After obtaining the effective path set, the optimal path for priority transmission is selected in combination with the reinforcement learning method. The reinforcement method designed this time mainly includes two processes, learning process and scheduling process. The two processes are executed in parallel and do not need to wait for the learning of the scheduler before scheduling. In the learning process, the method will continuously update the Q value. The reward function is used to estimate the optimal action of the next step. Whenever the receiver receives an acknowledgement packet, it will calculate the reward value and the Q value.

[0072] Q(s t ,a t )←(1-δ)Q(s t ,a t )+δ[R t+1 +γmaxQ(s t+1 ,a t+1 )] is used to update the data of the Q table.

[0073] Where s t and a t represent the state and action at the current time, s t+1 ​and a t+1 Represents the state and action at the next moment, R represents the reward function, the learning rate δ and the discount factor γ determine the learning speed of the method. According to this update rule, the intelligent experience will be based on the action a t Get the corresponding reward R, and then transition to the next state s t+1 And update the data of Q table.

[0074] Before sending each data packet, the scheduling process is activated. Due to the changes in dynamic network path conditions, a method that combines exploration and utilization is adopted. The method applies an exploration mechanism based on minimum end-to-end delay. The exploration process allows the agent to explore new paths with minimum end-to-end delay in an unused state. This paper divides the end-to-end delay into two parts. The first is the forward transmission delay, which mainly covers the entire process delay from the data packet being sent at the sender to the data packet being received at the receiver, including various delays outside the sending process. The other part is the queuing delay at the sender, that is, the time waiting for other data packets to be transmitted.

[0075] During the utilization process, the method will search for the action with the highest Q value in the Q table according to the current state, and select the corresponding path to transmit the data packet.

[0076] The following is a detailed introduction to the considerations and calculation methods of the end-to-end delay in this implementation. First, the calculation formula is given: T i =FT i +WT i +DT i +PT i .

[0077] Among them, FT i WT is the forward transmission delay of the data packet, which is mainly the time it takes for the data to be converted into a signal and propagate in the wireless channel; i DT is the sending queue delay, which is mainly the queue time that the data packet waits for other data to be sent during transmission; i It is the processing delay, which is mainly the time for header analysis, error detection and other operations on the received data; PT i The propagation delay is mainly the time it takes for data to propagate in the physical medium after being converted into a signal, which is related to the distance and speed of signal propagation. Figure 5 As shown above, this paper considers the end-to-end delay into two parts: the queuing delay at the sender and the forward transmission delay.

[0078] The following is a calculation method for the forward transmission delay of a data packet. When the network condition is relatively stable and there is little packet loss, the data packet is sent to the path P. i FT i =RTTi / 2 estimation, where RTT represents the round-trip delay of data.

[0079] During the transmission process, packet loss may occur due to network changes or other reasons. Figure 5 As shown in the figure, packets with crosses represent packets lost during transmission. To improve the reliability of the overall transmission, the sender will try to recover the data by retransmitting the packet after timeout when the packet is lost. Therefore, the total delay from the time the packet is sent to the time it is successfully received includes the time it takes to send, retransmit, and receive the packet, and the calculation formula is: FT i '=RTT i / 2+RTO.

[0080] RTO is the timeout retransmission time specified in advance by the sender before transmission.

[0081] Use loss i To represent the path P i Packet loss rate, sFT i It represents the average forward transmission delay of this path, and its calculation formula can be expressed as: sFT i =(1-loss i )*FT i +loss i *FT i '.

[0082] After sorting out the above calculation formula, the data packet on path P i The average forward transmission delay can be calculated using sFT i =RTT i / 2+loss i *Indicated by RTO.

[0083] Next, consider the data queuing delay. When the sender continues to send data, the data will wait in the buffer. The time WT to queue the data is i Defined as: WT i =CW i / B i .

[0084] CW in the formula i Represents path P i The amount of data in the buffer area that is ready to be sent but has not been sent and the amount of data that has been sent but has not received confirmation messages are composed of two parts, which can be expressed as path P i Congestion window size CWND i Through the above analysis, the path P i The end-to-end delay T i Indicated as T i=sFT i +WT i =RTT i / 2+loss i *RTO+CW i / B i .

[0085] Packet loss rate is one of the key factors affecting path reliability. i The calculation method is as follows:

[0086] It is stipulated that a total of l packets are sent between the two packet losses before time t i The amount of data sent between time t and the last packet loss is l i (t)'. If there is no packet loss before t, let l i (t) = 0, and stipulate that l i (t)' is the total amount of data transmission. Define L i (t) = max{l i (t),l i (t)'}, then 1 / L i (t) is expressed as path P i The estimated packet loss rate at time t.

[0087] So far, the minimum end-to-end delay minT based on the reinforcement learning exploration process is given. i The calculation method of , the update method of Q table in this learning process will be introduced in detail below.

[0088] First, initialize the state and action space, initialize the Q table, and set the Q value of all state-action pairs to zero. Set the learning rate σ, discount factor γ, and exploration rate ε. Among them, ε is used to balance the exploration process and the utilization process. During the exploration process, the path with the best delay performance at the current moment is preferentially selected according to the minimum end-to-end delay prediction calculation function given above. t , send a data packet, and after the transmission is completed, observe the immediate reward r after receiving the confirmation packet t and the next state s t+1 , calculate r according to the reward function t In each round of update iteration, the Q value and Q table will be updated, where the calculation formula of the Q value is: Q next =r t +γmaxQ(s t+1 ,a t+1 )]. According to the current state s t and reward r t , calculate the optimal Q value of the next state. The update formula of the Q table is: Q(s t ,a t)←(1-δ)Q(s t ,a t )+δQ next

[0089] Use current and future rewards to adjust the values ​​in the Q table. If the performance of a path changes, the scheduler will dynamically adjust its decision based on the current state and the Q table.

[0090] In order to reduce the impact of unreasonable resource allocation between multiple flows caused by inter-flow blocking, it is necessary to reasonably consider how to handle the priorities and dependencies between different flows in the scheduling method. To this end, a multi-flow manager is introduced in the design of this multi-path scheduling. Each time a connection is established, this method generates a dependency tree structure to more conveniently handle priority relationships. The root node of the dependency tree corresponds to a virtual flow numbered 0. When a data flow is ready to be sent, it will first be added to the dependency tree node. If flow i depends on flow j, then the node corresponding to flow i is set to a child node of flow j. If the flow does not depend on any other flow, it is set as a child node of the root node.

[0091] The flow manager first assigns a token value to each flow. Each token represents an opportunity for that flow to send a packet. During a scheduling cycle, the flow manager determines how many packets each flow can send. It then visits flow nodes in a specific order based on the flow's priority and dependencies. At the beginning of the scheduling process, the flow manager first visits the child nodes of the root node and allows them to send packets according to the number of tokens they were assigned.

[0092] In this method, we use the priority weight of each flow to determine how many packets the flow can send in a single scheduling cycle, that is, the token size. This number is determined by the flow priority ratio and the sending window. The maximum number of packets that a flow can send in a sending window is

[0093] Use priority in the formula i represents the priority of stream i, and W represents the sending window size. In this scheme, the sending window size determines the amount of data sent by each stream, and streams with higher priorities are sent first. Each stream will first send a specified number of data packets based on its priority ratio within a scheduling cycle. Given that the data volume of each stream is not fixed, it is sometimes impossible to align the maximum payload value of the packets. To avoid packet loss caused by bundling the remaining data of multiple streams together, this paper adopts a solution to independently package the data of each stream. Even if the last data packet of a stream is small, it will not interfere with the transmission of other streams.

[0094] Regarding the multi-stream blocking problem mentioned above, the effect after improvement is as shown in the figure. Figure 6 For the convenience of analysis, the above Figure 2 The same model is used in

[15] . Assume that the priorities of flows A, B, and C are 2, 6, and 4, respectively, and the sending window size is 6. In the first scheduling cycle, the method will first send 3 data packets from flow B, followed by 2 data packets from flow C, and finally 1 data packet from flow A. In subsequent scheduling cycles, if there is remaining data in the flow, it will still be sent in this order. The improved method ensures that high-priority flows are sent first while maintaining the flow priority. At the same time, this method effectively prevents a single flow from occupying the entire sending window, thereby further improving the transmission efficiency and fairness of the network.

[0095] In addition, the method has been optimized to solve the problem that the original re-injection method may cause the higher priority flow to be blocked by other secondary flows. Figure 7 As shown in the figure, suppose there are two flows, where flow B has a higher priority than flow C. During transmission, suppose that a packet B2 of flow B is lost. After the data transmission of flow B is completed, the system will continue to send the data of flow C. However, packet B2 of flow B is not retransmitted immediately. Instead, it must wait until the data transmission of flow C is completed before it can be retransmitted from the unacknowledged queue.

[0096] During this process, the transmission of low-priority stream C blocked the retransmission of packet B2 of high-priority stream B, affecting the continuity of stream B. To avoid this problem, the improved solution proposed in this paper optimizes the re-injection method. Even in the presence of low-priority streams, it ensures that the data transmission of high-priority streams is not unnecessarily blocked, thereby improving the overall efficiency and smoothness of the system.

[0097] like Figure 8 As shown, after critical stream B has sent all its data, it immediately checks the unack_queue for data with the same priority as the previous packet. If such a packet is found in the queue, the method immediately retransmits it, preventing lower-priority streams from blocking the transmission of higher-priority streams. After the retransmission, the system resumes sending data from stream C. If streams B and C have the same priority, the method still prioritizes data packet B2 from stream B.

[0098] The goal is to prioritize uninterrupted data flow transmission and ensure that packet B2 can be quickly retransmitted to the receiving end, thereby expediting the reordering of stream B and timely delivery to upper-layer applications. Stream C transmission proceeds only after the retransmission of stream B is complete and confirmed. If multiple streams of the same priority exist and have not previously sent data packets, the method can select a single stream to send data based on the scheduling policy.

[0099] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solutions disclosed in the present invention without the need for creative work should be included in the scope of protection of the present invention.

Claims

1. A method and device for adaptive traffic scheduling in heterogeneous dynamic networks, characterized in that: The device is applicable to heterogeneous and dynamic network environments, including: The adaptive path selector module is used to select a set of valid paths from the currently available paths. It uses reinforcement learning to dynamically adjust the path selection strategy based on network status parameters such as round-trip delay, packet loss rate, and congestion window of each path. To avoid local optimality, it further combines a greedy strategy to explore new paths with minimal delay. The flow priority manager module is used to dynamically allocate the data transmission volume of each flow in each scheduling cycle according to the priority weight and sending window size of each flow; and after the high-priority flow is sent, it gives priority to processing unconfirmed data packets of the same priority level to improve the continuity and real-time performance of key business flows.

2. The multipath scheduling device according to claim 1, wherein: The adaptive path selector module further includes a path evaluation submodule for constructing a path evaluation function based on multiple dimensions such as round-trip delay, packet loss rate, congestion window, and path bandwidth of the path, and updating the path score in real time during the scheduling process to support dynamic path adjustment.

3. The multipath scheduling device according to claim 2, characterized in that: The flow priority manager module includes a flow status monitoring unit and a window scheduling unit. The flow status monitoring unit is used to monitor the congestion window and sending rate of each business flow. The window scheduling unit dynamically adjusts the allocation rate according to the priority ratio of the flow, and supports immediate scheduling of pending packets with the same priority after the high-priority flow is completed to reduce interruption delay.

4. The multipath scheduling device according to claim 3, characterized in that: The device is adapted to the multi-stream multiplexing mechanism, the path selection and priority scheduling strategy can be executed independently among multiple sub-connections, and the scheduling result can be bound to the transmission path based on the connection identifier, thereby realizing the decoupling of multi-path scheduling and multi-stream priority control.

5. The multipath scheduling device according to claim 1, characterized in that: The adaptive path selector module further includes a path exploration submodule and a path score dynamic update submodule; the path exploration submodule is used to comprehensively evaluate the path quality based on the delay, congestion window, and packet loss rate of the greedy strategy and reinforcement learning feedback, and explore and activate new paths with better performance; The path score dynamic update submodule is used to update the evaluation value of each path based on the reinforcement learning strategy. The evaluation value changes dynamically with the path status to respond to the dynamic changes of the heterogeneous network environment in real time, thereby improving the dynamic adaptability of path selection and bandwidth utilization.

6. The multipath scheduling device according to claim 1, characterized in that: The flow priority manager module further includes a data flow mapping unit and a flow granularity adjustment unit; the data flow mapping unit is used to parse data packets and map them to corresponding priority queues according to the priority information marked on the data packets; the flow granularity adjustment unit dynamically adjusts the data transmission granularity of each priority queue based on the real-time monitoring of the network round-trip delay and the transmission status of each flow, and when the length of the high priority queue is less than a given threshold, the data transmission frequency of the high priority queue is increased, thereby reducing the waiting delay of key business data packets; The data flow mapping unit and the flow granularity adjustment unit jointly implement differentiated scheduling of data packets of different priorities, thereby meeting the real-time transmission requirements of delay-sensitive services.

7. A computer program product, characterized in that The invention comprises a computer program, which is used to execute the multipath scheduling method according to claim 7 when the computer program is run on one or more processors.

8. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the computer program is stored in the memory and executed by the processor to execute the operating instructions of the multipath scheduling method according to claim 7.

Images