Network traffic scheduling method and device based on reinforcement learning, equipment and medium

Abstract

The present application relates to a kind of network traffic scheduling method, device, equipment and medium based on reinforcement learning, method is through dynamic priority attenuation mechanism, real-time adjustment service weight, solve the background flow starvation deadlock problem caused by static priority scheduling;Introduce the entropy constraint fairness evaluation index to quantify resource allocation balance degree, overcome the defect that traditional method ignores the fairness between streams;Strategy network decision-making-data plane execution-state feedback update closed-loop system is built, the following technical processes are realized: network flow characteristics and link state are fused across modal to generate joint state vector;Based on flow life cycle and system load, calculate dynamic attenuation factor;Bandwidth allocation decision is generated by strategy network and attenuation constraint is applied;According to the execution result, detect deadlock risk and re-distribute guarantee bandwidth;Strategy network is updated in combination with entropy constraint reward;Closed loop refreshes network state and drives next cycle decision.

Description

Technical Field

[0001] This invention relates to the field of network resource management technology, and in particular to a network traffic scheduling method, apparatus, device, and medium based on reinforcement learning. Background Technology

[0002] In modern communication networks, network traffic scheduling methods play a crucial role. These technologies directly impact network throughput, transmission latency, and quality of service by intelligently prioritizing data streams and allocating resources. Especially in scenarios such as data centers, 5G core networks, and the Industrial Internet, efficient traffic scheduling ensures the timely transmission of critical services such as remote surgical commands and autonomous driving control signals, while simultaneously allocating bandwidth resources rationally to non-real-time services such as data backup and software updates. With the explosive growth of cloud computing and IoT devices, network traffic exhibits dynamic, variable, and heterogeneous concurrency characteristics, making traditional scheduling mechanisms inadequate for handling increasingly complex resource coordination needs.

[0003] However, existing traffic scheduling methods still have significant drawbacks. While static priority scheduling strategies are simple and efficient, they can lead to latency-sensitive flows continuously preempting resources, causing long-term starvation of background flows and even triggering system-level deadlocks. More problematic is that current reinforcement learning-based scheduling schemes excessively pursue overall throughput optimization, completely neglecting the crucial indicator of inter-flow fairness. This imbalance in resource allocation not only reduces network resource utilization but also triggers a chain reaction of congestion—when background flows time out and retransmit due to long waiting times, a large number of redundant data packets are generated, further exacerbating network congestion. Existing methods cannot dynamically adapt to changes in service priorities, nor can they effectively quantify the fairness of resource allocation, becoming a bottleneck restricting the development of high-reliability networks. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide a network traffic scheduling method, apparatus, device and medium based on reinforcement learning that can dynamically coordinate service priorities, intelligently prevent system deadlock and ensure fairness between flows.

[0005] The objective of this invention is achieved through the following solution: In a first aspect, the present invention provides a network traffic scheduling method based on reinforcement learning, comprising the following steps: The network flow feature data and link load data collected in real time are processed, and the flow attributes of flow type and latency sensitivity are combined with the link status of link utilization and transmission latency to generate a joint state vector. The joint state vector is subjected to dynamic priority decay processing to generate a dynamic decay factor matrix. The joint state vector is processed based on the preset policy network, the action vector of the original action prediction is calculated and the attenuation constraint is applied in combination with the dynamic attenuation factor matrix, the bandwidth allocation ratio vector is generated and sent to the data plane switch. The bandwidth allocation execution results fed back by the data plane switch are processed, the background flow waiting time is detected and the fairness entropy value is calculated. The minimum guaranteed bandwidth of the background flow is adjusted based on the deadlock risk threshold, and the updated bandwidth allocation results are generated. The updated bandwidth allocation results and joint state vector are processed, and the real-time reward signal of the policy network is calculated based on the bandwidth allocation ratio and fairness entropy value, and the policy network is updated. The network status is refreshed based on the updated bandwidth allocation results, updating the link load characteristics of switch ports and the flow data characteristics of network packets, and generating a joint state vector for the next decision cycle.

[0006] In one embodiment, S1 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: Extract flow attributes from real-time collected network flow feature data, parse the flow type identifier, latency requirement value and the amount of data to be transmitted, and generate a standard flow feature matrix; At the port status acquisition point of the real-time collected link load data, the link utilization value and transmission delay value of the switch port are measured to generate a real-time link load vector. Attention weights are assigned to the standard flow feature matrix and the real-time link load vector. Cross-modal correlation scores are calculated using the trainable parameter matrix to generate a joint state vector.

[0007] In one embodiment, S2 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: The time decay calculation is performed on the delay requirement value in the joint state vector. The lifetime decay coefficient is calculated based on the flow arrival timestamp and the current time difference to generate the time decay component. Active flow statistics are performed on the flow feature data in the joint state vector. The number of flow records with a data volume greater than zero is counted to obtain the number of active flows in the system. The load adjustment coefficient is calculated by logarithmic function transformation to generate the load adjustment component. The time decay component and the load adjustment component are matrix-fused, and the component values ​​are combined by the flow identifier index to generate a dynamic decay factor matrix.

[0008] In one embodiment, S3 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: The joint state vector is processed by policy network inference, and the original action probability distribution is calculated by multilayer perceptron to generate unconstrained action vectors. The unconstrained action vector and dynamic attenuation factor matrix are subjected to attenuation constraint processing. The corrected action value of the flow level action value is calculated by element-wise product, and the corrected action vector is generated. The correction action vector is processed by bandwidth ratio conversion. Based on the normalized allocation ratio of the total system bandwidth, a bandwidth allocation ratio vector is generated and sent to the data plane switch. The bandwidth allocation ratio vector is used to indicate the specific bandwidth allocation parameters for the data plane switch to perform flow table entry updates.

[0009] In one embodiment, S4 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: The bandwidth allocation execution results data fed back by the data plane switch are processed for background flow status monitoring, the average waiting time of all background flows is calculated, and background flow waiting indicators are generated. The actual allocation ratio in the execution result of the bandwidth allocation ratio vector is evaluated for fairness. The Shannon entropy formula is called to calculate the bandwidth distribution dispersion and generate a fairness entropy value. The fairness entropy value is used to indicate the degree of fairness in the allocation of system resources. The background flow waiting index and fairness entropy value are subjected to deadlock risk assessment. When the background flow waiting index exceeds the preset waiting time threshold, the guaranteed bandwidth is reallocated and an updated bandwidth allocation result is generated.

[0010] In one embodiment, S5 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: The updated bandwidth allocation results are subjected to basic performance evaluation processing to calculate the total system throughput and average transmission latency, and to generate basic reward components. The fairness entropy value of the updated bandwidth allocation result is processed by target deviation calculation, the absolute difference between the actual entropy value and the target entropy value is measured, and an entropy constraint reward component is generated. The basic reward component and the entropy constraint reward component are weighted and fused, and the final reward value is calculated by combining the preset weight coefficients to generate a real-time reward signal. The real-time reward signal is processed, the policy gradient is calculated using a near-end policy optimization algorithm, and the network weight parameters are updated by backpropagation, thus updating the policy network.

[0011] In one embodiment, step S6 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: The updated bandwidth allocation results are recalculated for link load, and the port utilization value is updated based on the actual allocated bandwidth to generate a new link load vector. Perform flow feature update processing on network data packets related to the updated bandwidth allocation results, deduct the amount of data already transmitted and maintain the status of live flows, and generate a new flow feature matrix; Perform periodic state initialization processing on the new link load vector and the new flow feature matrix, clear non-live flow records and reset the time window to generate the joint state vector for the next period.

[0012] Secondly, the present invention provides a network traffic scheduling device based on reinforcement learning, which is configured with the following modules: The joint state generation module is used to process real-time collected network flow feature data and link load data, and generate a joint state vector by combining flow attributes with flow type and latency sensitivity with link utilization and transmission latency. The attenuation factor generation module is used to perform dynamic priority attenuation processing on the joint state vector and generate a dynamic attenuation factor matrix. The bandwidth ratio generation module is used to process the joint state vector based on the preset policy network, calculate the action vector of the original action prediction and apply attenuation constraints in combination with the dynamic attenuation factor matrix, generate the bandwidth allocation ratio vector and send it to the data plane switch. The update allocation generation module is used to process the bandwidth allocation execution results fed back by the data plane switch, detect the background flow waiting time and calculate the fairness entropy value, combine the deadlock risk threshold to determine and adjust the minimum guaranteed bandwidth of the background flow, and generate updated bandwidth allocation results. The policy network update module is used to process the updated bandwidth allocation results and joint state vector, calculate the real-time reward signal of the policy network based on the bandwidth allocation ratio and fairness entropy value, and update the policy network. The network status refresh module is used to refresh the network status of the updated bandwidth allocation results, update the link load characteristics of the switch ports and the flow data characteristics of network packets, and generate the joint state vector for the next decision cycle.

[0013] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the above-mentioned network traffic scheduling methods based on reinforcement learning.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the above-mentioned reinforcement learning-based network traffic scheduling methods.

[0015] In summary, the network traffic scheduling method based on reinforcement learning and the fairness evaluation method based on entropy constraints provided in this application can achieve accurate quantification of resource allocation balance, overcoming the fundamental defect of existing technologies that neglect inter-flow fairness. A closed-loop feedback system is designed to achieve continuous collaborative optimization of policy decisions and network states. Specifically, cross-modal fusion processing can deeply correlate flow attributes and link state characteristics, enabling the joint state vector to comprehensively represent network dynamics; the dynamic decay factor matrix generation mechanism can intelligently adjust priorities based on flow lifecycle and system load changes, ensuring the quality of service for latency-sensitive flows while avoiding resource depletion of background flows; the bandwidth allocation decision based on policy networks and decay constraints can achieve efficient exploration and utilization balance of the action space; the minimum guaranteed bandwidth reallocation mechanism triggered by the deadlock risk threshold can proactively prevent system-level blocking; the entropy-constrained reward-driven policy network update can continuously optimize resource allocation strategies; and the network state refresh processing achieves efficient connection of decision cycles through synchronous updates of link load and flow characteristics.

[0016] This method enables real-time adaptive adjustment of service weights through a dynamic priority decay mechanism, effectively solving the background flow starvation deadlock problem caused by traditional static priority scheduling. While ensuring low-latency transmission of high-priority services, it significantly improves background flow reachability, eliminates the chain congestion effect caused by resource allocation imbalance, and ultimately achieves a globally optimal balance among network throughput, latency control, and fairness. Attached Figure Description

[0017] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a network traffic scheduling method based on reinforcement learning, provided for an embodiment of this application; Figure 2 A schematic diagram illustrating the process of generating a joint state vector provided in an embodiment of this application; Figure 3 A schematic diagram of the process for generating a dynamic decay factor matrix provided in an embodiment of this application; Figure 4 A schematic diagram of the process for generating a bandwidth allocation ratio vector provided in an embodiment of this application; Figure 5 A schematic diagram of the process for generating updated bandwidth allocation results provided in an embodiment of this application; Figure 6 A flowchart illustrating the update strategy network provided in this application embodiment; Figure 7A schematic diagram of the process for generating the joint state vector for the next period, provided in an embodiment of this application; Figure 8 This is a schematic diagram of a network traffic scheduling device based on reinforcement learning, provided as another embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0021] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0022] In one embodiment, such as Figure 1 As shown, a network traffic scheduling method based on reinforcement learning is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps: S1: Process the real-time collected network flow feature data and link load data, combine the flow attributes of flow type and latency sensitivity with the link status of link utilization and transmission latency, and generate a joint state vector.

[0023] Specifically, the system collects network flow characteristic data and link load data, employing a common network traffic acquisition protocol that supports the extraction of characteristics from various data flows in the network. The flow identifier in the network flow characteristic data is composed of key information from the network and transport layers, uniquely distinguishing different network flows; packet length distribution is obtained by statistically analyzing the lengths of a certain number of packets; message arrival intervals are calculated by recording the time difference between consecutive message arrivals; the current queue length is obtained by querying the buffer queue status of the switch port; and the number of retransmissions is statistically analyzed based on the retransmission mechanism of the transport protocol. The total link bandwidth in the link load data is determined by the physical characteristics of the link; real-time utilization is calculated as the ratio of the actual amount of data transmitted per unit time to the total bandwidth; bidirectional transmission delay is obtained by sending probe messages at both ends of the link and recording the round-trip time difference; packet loss rate is calculated as the difference between the total number of sent messages and the total number of received messages; and the congestion window size is obtained in real-time based on the congestion control mechanism of the transport protocol.

[0024] The system parses the five-tuple information of network flows and combines it with the physical link connections obtained through the network topology discovery protocol to associate each network flow with the physical link currently used for transmission, constructing a flow-link mapping table. During feature normalization, latency sensitivity attributes are categorized into different levels and mapped to corresponding intervals based on the real-time requirements of the service type; link utilization is mapped to a specified interval based on the ratio of actual bandwidth used to total bandwidth; and transmission latency is proportionally converted and mapped to the corresponding interval based on the maximum allowable latency range of the link.

[0025] For example, the system forms a joint state vector by sequentially concatenating flow attribute features and link state features. The number of rows in the joint state vector corresponds to the number of network flows currently existing in the network, and the number of columns is the sum of the number of flow attribute features and link state features associated with each flow. During data cleaning, the system can employ statistical outlier detection criteria to mark collected data that exceeds the normal range. For marked outliers, the system uses a sliding window approach, taking the median of the normal data within the window to fill the gap, thereby maintaining the stability and continuity of the data in the joint state vector.

[0026] S2: Perform dynamic priority decay processing on the joint state vector to generate a dynamic decay factor matrix.

[0027] Considering that maintaining an initial high-priority schedule for an extended period under high load can easily lead to persistent starvation of non-critical flows, and that existing methods struggle to quantify the dynamic changes in business priorities, this application constructs a dynamic priority decay model based on multiple dimensions, including latency sensitivity, current link utilization, and historical scheduling duration.

[0028] Specifically, the system calculates a dynamic attenuation coefficient for each flow or service priority based on indicators such as its latency sensitivity level in the current joint state vector, the remaining bandwidth percentage of its link, and the percentage of bandwidth occupied in the past several decision cycles, using a pre-trained nonlinear mapping function. For example, a relatively large attenuation coefficient is assigned to a flow that is highly latency sensitive but has occupied too much bandwidth resources for a long time; while a smaller attenuation coefficient is assigned to a background flow that is low latency sensitive but has been suppressed for a long time, so as to dynamically adjust its probability of acquiring resources in the future. The dynamic attenuation coefficients corresponding to all flows or services are organized into a matrix form, that is, a dynamic attenuation factor matrix is ​​generated, whose row and column dimensions correspond to the flow or service classification in the network, providing a quantitative basis for subsequent resource allocation constraints, ensuring that resource allocation can respond to real-time needs while taking into account long-term fairness, and effectively avoiding the resource allocation imbalance caused by priority solidification in traditional methods.

[0029] S3: Based on the preset policy network, the joint state vector is processed to calculate the action vector of the original action prediction and apply attenuation constraints in combination with the dynamic attenuation factor matrix to generate the bandwidth allocation ratio vector and send it to the data plane switch.

[0030] Specifically, the system in this application is pre-configured with a deep reinforcement learning policy network to perform feature extraction and transformation processing on the input joint state vector. This policy network is based on a deep neural network architecture, contains multiple hidden layers, and learns complex feature representations of the state vector through a non-linear activation function. The output of the policy network is the original action prediction vector corresponding to each data stream in the network, and its elements represent the initial bandwidth allocation ratio proposal for each data stream in the current state.

[0031] For example, the system can use matrix multiplication to multiply the original action prediction vector with the dynamic attenuation factor matrix to achieve attenuation constraint processing on the initial bandwidth allocation ratio, resulting in a corrected bandwidth allocation ratio vector. The bandwidth allocation ratio is then normalized to ensure that the sum of the bandwidth ratios allocated to each data stream does not exceed the available link bandwidth resource limit. The corrected bandwidth allocation ratio vector is sent to the data plane switch via a specific communication protocol and data format conversion. Upon receiving the vector, the data plane switch allocates and schedules bandwidth resources for each data stream according to the indicated bandwidth allocation ratio.

[0032] S4: Process the bandwidth allocation execution results fed back by the data plane switch, detect the background flow waiting time and calculate the fairness entropy value, combine the deadlock risk threshold to determine and adjust the minimum guaranteed bandwidth of the background flow, and generate updated bandwidth allocation results.

[0033] Specifically, after executing the bandwidth allocation strategy, the data plane switch feeds back the execution results to the system. This feedback includes the actual bandwidth value obtained by each data flow, changes in packet transmission delay, queue length updates, and possible congestion indications. The system processes this feedback data to evaluate the effectiveness of the current bandwidth allocation strategy. The system calculates the waiting time for each data flow based on the feedback data. By comparing the time a data packet enters the switch queue with the actual start time of transmission, the system obtains the waiting time of each data flow within the switch.

[0034] Preferably, the system can use Shannon entropy theory to calculate the fairness entropy value, treating the actual bandwidth ratio obtained by each data flow as a probability distribution and substituting it into the entropy formula for calculation, thereby quantifying the fairness of the current bandwidth allocation. The system compares the calculated fairness entropy value with a preset deadlock risk threshold, which is determined based on historical network operation data and theoretical analysis, and is used to determine whether the current resource allocation may lead to system deadlock. If the fairness entropy value is lower than the deadlock risk threshold, it indicates that there is unfairness in resource allocation, which may cause some data flows to not receive sufficient bandwidth for a long time, thereby triggering timeout retransmissions and network congestion. The system adjusts the minimum guaranteed bandwidth for background flows. Based on the current network status and fairness assessment results, the system increases the minimum bandwidth guarantee for background flows according to predetermined rules and recalculates the bandwidth allocation ratio of each data flow. The system generates an updated bandwidth allocation result and sends the result back to the data plane switch. The data plane switch readjusts its queue scheduling mechanism based on the updated bandwidth allocation result to ensure that the new bandwidth allocation strategy is implemented. This feedback and adjustment process is ongoing, with the system continuously optimizing bandwidth allocation strategies to maintain stable network operation and efficient resource utilization, and to avoid deadlocks and other performance issues.

[0035] S5: Process the updated bandwidth allocation results and joint state vector, calculate the real-time reward signal of the policy network based on the bandwidth allocation ratio and fairness entropy value, and update the policy network.

[0036] Specifically, in the multi-dimensional reward function constructed by the system, the throughput metric is calculated as the ratio of the actual data transmitted by the link to the maximum data transmission capacity of the link, reflecting the efficiency of link resource utilization. The fairness entropy value is processed through a specific transformation method before being incorporated into the reward calculation; the better the fairness, the higher the reward value. Deadlock risk is incorporated into the reward calculation after a reverse transformation; the lower the deadlock risk, the higher the reward value. Execution deviation is directly incorporated into the reward calculation as a penalty; the larger the execution deviation, the heavier the penalty. The weights of the parameters in each dimension of the reward function are dynamically adjusted according to the real-time network load. When the network is under high load, the system increases the weights of parameters related to fairness and deadlock risk to prioritize the stable operation of the network and the fairness of resource allocation. When the network is under low load, the system increases the weights of parameters related to throughput to fully utilize network resources and improve transmission efficiency. The reward value is constrained to a specific range through normalization to prevent gradient explosion during policy network training due to excessively large or small reward values. For example, when the system employs an experience replay mechanism, the joint state vector, action vector, reward value, and next state vector generated in each decision cycle are stored in an experience pool. The capacity of the experience pool is set according to training requirements. During training, the system randomly selects a certain number of samples from the experience pool for batch training each time to improve training stability and efficiency. The target network and policy network set by the system have the same network structure. The parameters of the target network are updated using a soft update method, that is, the parameters of the policy network are slowly updated to the target network according to a fixed soft update coefficient to ensure the smoothness of the training process. The optimizer can use a specific type of adaptive optimization algorithm, which calculates the time difference error and backpropagates it to each layer of neurons in the policy network to adjust the network weight parameters, enabling the policy network to gradually learn a better decision policy. The parameters of the policy network are saved at fixed intervals to prevent model loss due to training interruptions and to facilitate subsequent model analysis and optimization.

[0037] S6: Perform network status refresh processing on the updated bandwidth allocation results, update the link load characteristics of switch ports and the flow data characteristics of network packets, and generate the joint state vector for the next decision cycle.

[0038] Specifically, the system parses the updated bandwidth allocation results, extracting the final transmission state parameters of each data stream in the current decision cycle, including the updated bandwidth allocation ratio and actual transmission latency. Based on these parameters, the system updates the link load characteristic parameter library of the switch ports and the flow data characteristic parameter library of network packets. For updating the link load characteristic parameter library, the system recalculates the utilization rate and available bandwidth of each link; for updating the flow data characteristic parameter library, the system corrects the transmission rate and remaining data volume of each data stream. After completing the parameter library update, the system uses the updated parameter library to generate a joint state vector for the next decision cycle, following the same processing flow as step S1 above. This joint state vector will serve as the input for subsequent reinforcement learning loops, initiating a new round of network traffic scheduling decision-making, thereby achieving dynamic optimization and continuous improvement of the network traffic scheduling strategy.

[0039] In summary, the network traffic scheduling method based on reinforcement learning and the fairness evaluation method based on entropy constraints provided in this application can achieve accurate quantification of resource allocation balance, overcoming the fundamental defect of existing technologies that neglect inter-flow fairness. A closed-loop feedback system is designed to achieve continuous collaborative optimization of policy decisions and network states. Specifically, cross-modal fusion processing can deeply correlate flow attributes and link state characteristics, enabling the joint state vector to comprehensively represent network dynamics; the dynamic decay factor matrix generation mechanism can intelligently adjust priorities based on flow lifecycle and system load changes, ensuring the quality of service for latency-sensitive flows while avoiding resource depletion of background flows; the bandwidth allocation decision based on policy networks and decay constraints can achieve efficient exploration and utilization balance of the action space; the minimum guaranteed bandwidth reallocation mechanism triggered by the deadlock risk threshold can proactively prevent system-level blocking; the entropy-constrained reward-driven policy network update can continuously optimize resource allocation strategies; and the network state refresh processing achieves efficient connection of decision cycles through synchronous updates of link load and flow characteristics.

[0040] This method enables real-time adaptive adjustment of service weights through a dynamic priority decay mechanism, effectively solving the background flow starvation deadlock problem caused by traditional static priority scheduling. While ensuring low-latency transmission of high-priority services, it significantly improves background flow reachability, eliminates the chain congestion effect caused by resource allocation imbalance, and ultimately achieves a globally optimal balance among network throughput, latency control, and fairness.

[0041] In one embodiment, such as Figure 2 As shown, S1 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: S11: Extract flow attributes from the real-time collected network flow feature data, parse the flow type identifier, latency requirement value and the amount of data to be transmitted, and generate a standard flow feature matrix.

[0042] Specifically, network flow feature data includes flow header information and transmission parameters. The system obtains the flow type identifier by parsing the protocol field in the flow header. The flow type identifier exists in a preset encoded form, corresponding to the classification identifier of different service types. The system extracts the latency requirement value from the flow's transmission control information. The latency requirement value is the maximum allowable delay during flow transmission, and this value is embedded in the flow feature data through negotiation information during the flow establishment phase.

[0043] For example, the system calculates the amount of data to be transmitted by the total number of data packets contained in the stream and the length of each data packet. The amount of data to be transmitted is the total data scale that the stream needs to transmit. The system arranges the extracted stream type identifier, latency requirement value, and amount of data to be transmitted in the order of the streams, with each stream corresponding to one row of data. Each row contains the encoded value of the stream type identifier, the quantization result of the latency requirement value, and the calculated result of the amount of data to be transmitted. The system standardizes the arranged data to eliminate the dimensional differences between different attributes, forming a standard stream feature matrix. The number of rows in the matrix is ​​consistent with the number of currently active streams, and the number of columns corresponds to the number of extracted stream attributes.

[0044] S12: At the port status acquisition point of the real-time collected link load data, measure the link utilization value and transmission delay value of the switch port, and generate a real-time link load vector.

[0045] Specifically, the system performs port status acquisition and processing on the real-time collected link load data. The link load data covers the transmission status information of each port of the switch. For each switch port, the system separates the transmission records related to that port from the link load data. The system calculates the link utilization value based on the port's transmission records. The link utilization value is the ratio of the amount of data actually transmitted by the port per unit time to the maximum amount of data that the port can transmit. The calculation process needs to exclude interference during the port's idle periods.

[0046] The system obtains the transmission delay value by recording the time difference between data packets entering and leaving the port. The transmission delay value is the time spent processing and forwarding data packets at the port. The calculation requires removing the time spent propagating the data packets in the transmission medium. The system then arranges the link utilization value and transmission delay value of each port according to their port numbers, with each port corresponding to two consecutive values: the link utilization value and the transmission delay value. The system integrates the arranged numerical sequence into a one-dimensional array, forming a real-time link load vector. The length of the vector is twice the number of switch ports, and the position of each element in the vector corresponds one-to-one with the port number.

[0047] S13: Assign attention weights to the standard flow feature matrix and the real-time link load vector, calculate the cross-modal correlation score through the trainable parameter matrix, and generate a joint state vector.

[0048] Specifically, the system introduces a trainable parameter matrix, which contains two sub-matrices that match the dimensions of the standard flow feature matrix and the real-time link load vector, respectively. The system multiplies the standard flow feature matrix with its corresponding sub-matrices to obtain a weighted matrix of flow features; it also multiplies the real-time link load vector with its corresponding sub-matrices to obtain a weighted vector of link load. The weighting process is implemented through element-wise operations between matrices and vectors.

[0049] The system obtains cross-modal association scores by calculating the similarity between the weighted matrix of flow features and the weighted vector of link load. These scores reflect the degree of correlation between flow features and link load, and are calculated using vector dot product or cosine similarity algorithms. The system assigns attention weights based on these scores, with higher scores receiving higher weights and lower scores receiving lower weights. The system adjusts the standard flow feature matrix according to these attention weights, and simultaneously adjusts the real-time link load vector with the same weights. The adjusted matrix and vector are then concatenated by rows and columns to form a joint state vector. This joint state vector contains the association information between flow features and link load, and its dimension is the sum of the number of columns in the standard flow feature matrix and the length of the real-time link load vector.

[0050] In one embodiment, such as Figure 3 As shown, S2 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: S21: Perform time decay calculation on the delay requirement value in the joint state vector, calculate the lifetime decay coefficient based on the flow arrival timestamp and the current time difference, and generate the time decay component.

[0051] Specifically, the system performs time decay calculations on the delay requirement values ​​in the joint state vector, extracts the delay requirement values ​​corresponding to each flow from the joint state vector, and simultaneously obtains the arrival timestamp of each flow. The arrival timestamp is the time information recorded when the flow enters the network, and the current time is the real-time time when the system performs this step. The system calculates the difference between the arrival timestamp of each flow and the current time; this time difference reflects the duration the flow has existed in the network.

[0052] For example, the system calculates the attenuation of the latency requirement based on the time difference. The calculation process follows a preset attenuation rule, meaning that the attenuation coefficient changes as the time difference increases. The system obtains the lifetime attenuation coefficient for each flow through this calculation. The lifetime attenuation coefficient is related to the time difference; changes in the time difference will cause the coefficient value to adjust accordingly. The system arranges the lifetime attenuation coefficients of all flows in order of flow identification, forming a time attenuation component. Each element in the time attenuation component corresponds to the attenuation coefficient of a flow, and the position of the element corresponds one-to-one with the flow identification, ensuring accurate matching of the corresponding flow in subsequent processing.

[0053] S22: Perform active flow statistics processing on the flow feature data in the joint state vector, count the number of flow records with a data volume greater than zero to obtain the number of active flows in the system, and calculate the load adjustment coefficient through logarithmic function transformation to generate load adjustment components.

[0054] Specifically, the system performs active flow statistics processing on the flow feature data in the joint state vector, separating the flow feature data from the joint state vector. This flow feature data contains information about the amount of data to be transmitted for each flow. The system determines the amount of data to be transmitted for each flow, filtering out flow records with a data volume greater than zero; these records correspond to flows that are in an active state. The system counts the filtered active flow records to obtain the number of active flows in the system, which reflects the total number of flows in the current network that still need to transmit data.

[0055] For example, the system inputs the number of active flows into a logarithmic function for transformation, obtaining a load adjustment coefficient. This load adjustment coefficient reflects the impact of the number of active flows on the system load. The system assigns the load adjustment coefficient according to the flow identification order, with each flow corresponding to a load adjustment coefficient value, forming a load adjustment component. The structure of the load adjustment component is consistent with the flow identification order to ensure accurate correlation with other components.

[0056] S23: Perform matrix-based fusion processing on the time decay component and the load adjustment component, combine the component values ​​according to the flow identifier index, and generate a dynamic decay factor matrix.

[0057] Specifically, the system performs matrix-based fusion processing on the time decay component and the load adjustment component to obtain the flow identifier corresponding to each element in both components. The flow identifier is unique information that distinguishes different flows. The system matches elements in the two components according to the flow identifier index, combining time decay component elements and load adjustment component elements belonging to the same flow. Each flow corresponds to a set of combined component values. The system constructs a matrix framework with the number of flows as the dimension, where the number of rows and columns of the matrix corresponds to the number of flows.

[0058] For example, the system fills the matrix frame with the combined component values ​​of each group in the order of the flow identifiers. The rows and columns of the matrix correspond to the flow identifiers, and the element value at each position in the matrix is ​​the result of combining the components of the corresponding row and column flow identifiers. When the row and column flow identifiers are the same, the element value is the combined component value of that flow; when they are different, the element value is set according to a preset rule. Through this process, the system generates a dynamic attenuation factor matrix, which contains the time attenuation and load adjustment information of all flows, providing data support for subsequent priority adjustments.

[0059] In one embodiment, such as Figure 4 As shown, S3 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: S31: Perform policy network inference processing on the joint state vector, calculate the original action probability distribution through a multilayer perceptron, and generate an unconstrained action vector.

[0060] Specifically, when the system performs policy network inference processing on the joint state vector, it first reorganizes the features of each dimension of the joint state vector according to the flow identifier order to form an input matrix of dimension m×d (where m is the number of active flows and d is the feature dimension of a single flow). This matrix is ​​then processed by a linear transformation layer, and the transformation formula is as follows: ,in For the input matrix, This is the weight matrix for this layer. As the bias vector, the output X1 dimension is adjusted to m× ( (This refers to the transformed dimension). The processing result is fed into the hidden layer of the multilayer perceptron, where a linear rectified function is used for activation, calculated as follows: ; Where l is the hidden layer number, , These represent the weights and biases of this layer, respectively, and higher-order feature associations are extracted through nonlinear transformation. The output layer receives the output of the last hidden layer, undergoes a linear transformation, and then calculates the original action probability distribution using a normalized exponential function, as shown in the formula: ; in, This represents the original output of the i-th neuron in the output layer. Let be the action probability of the i-th flow. All Arrange the flows in order of their identifiers to generate an unconstrained action vector. Each element in the vector reflects the resource allocation tendency of the corresponding flow under unconstrained conditions.

[0061] S32: Perform attenuation constraint processing on the unconstrained action vector and the dynamic attenuation factor matrix, calculate the corrected action value of the flow level action value by element-wise product, and generate the corrected action vector.

[0062] Specifically, when the system applies attenuation constraints to the unconstrained action vector and the dynamic attenuation factor matrix, it first verifies the dimensionality consistency of the two. The unconstrained action vector is a 1×m row vector, and the dynamic attenuation factor matrix is ​​an m×1 column matrix. The system uses a transpose operation to convert the dynamic attenuation factor matrix into a 1×m row matrix, ensuring dimensionality matching. Attenuation constraints are implemented using element-wise multiplication. For the i-th flow, the calculation formula is: ; in, For the i-th element of the unconstrained action vector, The i-th element of the dynamic decay factor matrix These are the corrected action values. This operation scales the original action values ​​proportionally to the decay factor, suppressing action values ​​that occupy resources for extended periods or are under high load. All corrected action values ​​are combined in their original order to generate a corrected action vector. Each element in the vector takes values ​​in the range [0,1], preserving the relative priority of the original actions while incorporating constraints related to time decay and load adjustment. The system validates the corrected action vector; if any element is negative, it is automatically set to 0, ensuring that all values ​​in the vector comply with the non-negativity constraint.

[0063] S33: Perform bandwidth ratio conversion processing on the correction action vector, generate a bandwidth allocation ratio vector based on the normalized allocation ratio of the total system bandwidth, and send it to the data plane switch. The bandwidth allocation ratio vector is used to indicate the specific bandwidth allocation parameters for the data plane switch to perform flow table entry updates.

[0064] Specifically, when the system performs bandwidth ratio conversion on the corrected action vector, it first reads the total available bandwidth B of the data plane switch through the network control interface. This value is the sum of the aggregated bandwidth of all ports of the switch, in bits per second. The system then calculates the cumulative sum of all elements of the corrected action vector: ; When S=0, each element is automatically reset to 1 / m to avoid allocation failure. The bandwidth allocation ratio is obtained through normalization calculation, and the allocation ratio formula for the i-th flow is: Ensure that the sum of all proportions is 1, that is: The system multiplies each proportional value by the total bandwidth to obtain the actual allocated bandwidth value for each flow, but only arranges the proportional values ​​in order of flow identifier to generate a bandwidth allocation proportional vector. This vector is sent to the data plane switch through the protocol interface. After receiving it, the switch parses each element in the vector and maps it to the weight parameters in the flow table entry, which are used to adjust the priority of the weighted round-robin scheduler. At the same time, the switch updates the rate limiting parameters of the flow controller according to the proportional vector to ensure that the actual forwarding rate is consistent with the allocation proportion, thus completing the execution of the resource allocation command.

[0065] In one embodiment, such as Figure 5 As shown, S4 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: S41: Perform background flow status monitoring on the bandwidth allocation execution result data fed back by the data plane switch, calculate the average waiting time of all background flows, and generate background flow waiting indicators.

[0066] Specifically, when the system performs background flow status monitoring on the bandwidth allocation execution results data fed back by the data plane switch, it first extracts the type identifier and timestamp information of all flows from the feedback data. Background flows are identified by their type identifiers, which match the type codes in the flow feature data. The system filters out all background flow records by matching them against a preset list of background flow type codes. For each background flow, the system extracts the enqueue timestamp and dequeue timestamp of its first data packet. The enqueue timestamp is the time the data packet enters the switch's buffer queue, and the dequeue timestamp is the time the data packet leaves the queue. The difference between the two is the single-time waiting time for that background flow. The system counts the single-transaction wait time of all background streams and calculates the average wait time using the following formula: ; Where k is the total number of background streams. Let i be the single wait time for the i-th background stream. This is the average waiting time. This average waiting time serves as a background flow waiting metric, stored as a time series, and reflects the resource acquisition latency of background flows within the current decision-making cycle. The system also records the waiting time distribution for each background flow. If a single background flow's waiting time significantly exceeds the average, that flow is marked separately for subsequent targeted processing.

[0067] S42: Perform fairness evaluation on the actual allocation ratio in the execution result of the bandwidth allocation ratio vector, call the Shannon entropy formula to calculate the bandwidth distribution dispersion, and generate a fairness entropy value. The fairness entropy value is used to indicate the degree of fairness in the allocation of system resources.

[0068] Specifically, when the system performs a fairness assessment on the actual allocation ratio in the execution result of the bandwidth allocation ratio vector, it extracts the actual bandwidth allocation value for each flow from the execution result data and calculates the actual allocation ratio. , that is, the ratio of the actual allocated bandwidth of the i-th flow to the total system bandwidth, expressed by the formula: ; in, Let B be the actual bandwidth allocated to the i-th flow, and let B be the total system bandwidth, satisfying the following conditions: (where n is the total number of all streams). The system calls the Shannon entropy formula to calculate the bandwidth distribution dispersion. The formula is: ; Where H is the fairness entropy value, H represents the actual allocation ratio for the i-th flow. H reaches its maximum value when all flows have the same allocation ratio. When resources are concentrated in a minority of flows, the value of H decreases, indicating a reduction in allocation fairness. This fairness entropy value serves as a quantitative indicator to indicate the degree of fairness in system resource allocation. It is compared with a preset fairness threshold to provide a basis for subsequent adjustments.

[0069] S43: Perform deadlock risk assessment on background flow waiting indicators and fairness entropy values. When the background flow waiting indicators exceed the preset waiting time threshold, reallocate the guaranteed bandwidth and generate updated bandwidth allocation results.

[0070] Specifically, when the system performs deadlock risk assessment on the background flow waiting index and the fairness entropy value, it first sets the background flow waiting index... Compared with the preset waiting time threshold A comparison is performed. The preset waiting time threshold is determined based on the service level agreement of the background flow and stored in the system configuration file. When When the system determines that the background flow is at risk of resource starvation, it further assesses the possibility of deadlock based on the fairness entropy value H. If H is lower than a preset fairness threshold, the system initiates a background flow guaranteed bandwidth reallocation process. During reallocation, the system calculates the minimum guaranteed bandwidth for each background flow. The calculation formula is: ; in, To ensure basic bandwidth for background streams, To adjust the coefficient, Let B be the proportion of the background flow in the correction action vector, and let B be the total system bandwidth. The system accumulates the minimum guaranteed bandwidth of all background flows. If the sum exceeds a preset proportion of the total system bandwidth, the guaranteed bandwidth of each background flow is reduced proportionally to ensure the feasibility of resource allocation. The reallocated bandwidth proportion vector constitutes the updated bandwidth allocation result, which replaces the original allocation result and is sent to the data plane switch for update.

[0071] In one embodiment, such as Figure 6 As shown, S5 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: S51: Perform basic performance evaluation on the updated bandwidth allocation results, calculate the total system throughput and average transmission latency, and generate basic reward components.

[0072] Specifically, the system performs basic performance evaluation on the updated bandwidth allocation results, extracting the actual transmitted data volume for each flow. The actual transmitted data volume is the total number of data packets successfully transmitted by each flow per unit time, calculated by summing the products of the number of data packets and the length of each data packet for each flow. The system sums the actual transmitted data volumes of all flows to obtain the total system throughput, which reflects the overall data transmission capacity of the system per unit time. Simultaneously, the system extracts the transmission delay for each flow, which is the time interval between a data packet being sent from the source and received at the destination. This delay is calculated by recording the transmission time difference between the first and last data packets of each flow. The system sums the transmission delays of all flows and divides by the number of flows to obtain the average transmission delay, which reflects the overall transmission speed level of the system. The system generates a basic reward component based on the total throughput and average transmission delay. The value of the basic reward component is positively correlated with the total throughput and negatively correlated with the average transmission delay.

[0073] S52: Perform target deviation calculation on the fairness entropy value of the updated bandwidth allocation result, measure the absolute difference between the actual entropy value and the target entropy value, and generate an entropy constraint reward component.

[0074] Specifically, the system performs target deviation calculation on the fairness entropy value of the updated bandwidth allocation results, obtaining the actual fairness entropy value from the updated bandwidth allocation results. The actual fairness entropy value is calculated based on the current bandwidth allocation situation. The system calls a preset target entropy value, which is a pre-set entropy value reflecting the ideal fairness state, determined according to network scenario requirements. The system calculates the absolute difference between the actual fairness entropy value and the target entropy value by taking the absolute value of the difference. This absolute difference reflects the degree of deviation between actual fairness and target fairness. The system generates an entropy constraint reward component based on the absolute difference. The value of the entropy constraint reward component is negatively correlated with the absolute difference; that is, the smaller the absolute difference, the higher the component value, and vice versa. This guides the system to adjust towards the target fairness state, and this component reflects the system's performance in resource allocation fairness.

[0075] S53: Perform weighted fusion processing on the basic reward component and the entropy constraint reward component, calculate the final reward value by combining the preset weight coefficients, and generate a real-time reward signal.

[0076] Specifically, the system performs weighted fusion processing on the basic reward component and the entropy constraint reward component to obtain a preset weight coefficient. This preset weight coefficient represents the proportion of the basic reward component and the entropy constraint reward component in the final reward value, and is pre-set in the system according to the network optimization objective. The system multiplies the basic reward component by its corresponding weight coefficient to obtain the basic reward weighted value; it also multiplies the entropy constraint reward component by its corresponding weight coefficient to obtain the entropy constraint reward weighted value. The system adds the basic reward weighted value and the entropy constraint reward weighted value to obtain the final reward value. The final reward value comprehensively reflects the system's overall performance in terms of basic performance and fairness. The system converts the final reward value into a real-time reward signal. This real-time reward signal conforms to the input format of the policy network. Numerical conversion ensures that the signal can be recognized and processed by the policy network, and this signal is used to drive the parameter updates of the policy network.

[0077] S54: Process the real-time reward signal, calculate the policy gradient through the near-end policy optimization algorithm, and backpropagate to update the network weight parameters and update the policy network.

[0078] Specifically, the system processes the real-time reward signal, calculates the policy gradient using a proximal policy optimization algorithm, and updates the network weight parameters through backpropagation. The system inputs the real-time reward signal into the proximal policy optimization algorithm framework, which consists of two parts: an old policy network (the currently used policy network) and a new policy network (the policy network to be updated).

[0079] The system generates an action probability distribution using the old policy network and calculates the advantage value for each action based on real-time reward signals. The advantage value is the difference between the reward generated by that action and the average reward. The system calculates the policy loss by comparing the advantage value with the action probabilities of the new and old policy networks. The policy loss measures the improvement of the new policy network compared to the old one. The system calculates the policy gradient based on the policy loss, which reflects the influence of network weight parameters on the loss value. The system backpropagates the policy gradient to each layer of neurons in the new policy network, adjusting the connection weight parameters between neurons to make the output of the new policy network closer to the optimal action selection. After the parameter update, the system replaces the old policy network with the new one, completing the policy network update. The updated policy network is used for action generation in the next decision cycle.

[0080] In one embodiment, such as Figure 7 As shown, S6 of the network traffic scheduling method based on reinforcement learning provided by the present invention specifically includes the following steps: S61: Perform link load recalculation on the updated bandwidth allocation results, update the port utilization value based on the actual allocated bandwidth, and generate a new link load vector.

[0081] Specifically, the system recalculates the link load of the updated bandwidth allocation results, extracting the actual allocated bandwidth for each flow from the updated bandwidth allocation results. The actual allocated bandwidth is the amount of bandwidth resource allocated to each flow in the current period. Based on the flow's transmission path information, the system associates the actual allocated bandwidth of each flow with its corresponding switch port; each flow's transmission corresponds to a specific switch port. The system sums the actual allocated bandwidth of all flows associated with each port to obtain the actual occupied bandwidth of that port. The system then calculates the port utilization value using the following formula, combined with the port's maximum supported bandwidth: ; in, Let p be the utilization value of the p-th port. This represents the actual bandwidth used by the port. This represents the maximum supported bandwidth of the port. The system arranges the utilization values ​​of each port sequentially according to their port numbers, forming a new link load vector. The structure of the new link load vector is consistent with the real-time link load vector, containing the current utilization information of all ports. It is used to reflect the link load status after bandwidth allocation adjustment, providing link-level data support for the status assessment in the next cycle.

[0082] S62: Perform flow feature update processing on network data packets related to the updated bandwidth allocation results, deduct the amount of data already transmitted and maintain the status of live flows, and generate a new flow feature matrix.

[0083] Specifically, the system performs flow feature update processing on network data packets related to the updated bandwidth allocation results, extracts the flow identifiers corresponding to the updated bandwidth allocation results from the network data packets, and associates these flow identifiers with specific flow records. The system also counts the number of data packets transmitted by each flow within the current period and the length of each data packet, calculating the amount of data transmitted. : ; in, Let n be the amount of data already transmitted, and n be the number of data packets transmitted in the current period. Let be the length of the k-th data packet. The system subtracts the amount of data already transmitted from the initial amount of data to be transmitted in the stream to calculate the remaining amount of data to be transmitted: ; in, The remaining amount of data to be transmitted. This represents the initial amount of data to be transmitted when the stream is established. The system updates the "Awaiting Data Volume" field in the stream characteristics to reflect the remaining amount of data to be transmitted. The system checks the remaining amount of data to be transmitted for each stream. If the remaining amount is zero, the stream is marked as non-live; if the remaining amount is greater than zero, the stream remains live. Based on the latest characteristics of the live streams, including the updated amount of data to be transmitted, latency requirements, and stream type identifier, the system reconstructs the matrix to form a new stream characteristic matrix. The structure of the new stream characteristic matrix is ​​consistent with the standard stream characteristic matrix, containing only the characteristic information of live streams, ensuring the timeliness and accuracy of the stream characteristic data.

[0084] S63: Perform periodic state initialization processing on the new link load vector and the new flow feature matrix, clear non-live flow records and reset the time window, and generate the joint state vector for the next period.

[0085] Specifically, the system performs periodic state initialization processing on the new link load vector and the new flow feature matrix. The system filters out non-living flow records from the new flow feature matrix; these records are the row data corresponding to flows with zero remaining data to be transmitted. The system removes these records from the matrix, retaining only the feature information of the living flows. The system resets the time window, which is the time range of the current period. The reset process includes updating the timestamp and setting the period start time to the current processing time for time-related calculations in the next period. The system checks the new link load vector, removing invalid portions of the port load information related to non-living flows to ensure that the vector only contains load data from currently active ports.

[0086] The system performs cross-modal correlation processing on the processed new link load vector and new flow feature matrix according to the attention weight allocation method, and calculates the correlation score using the following formula: ; in, The correlation score between the i-th flow feature and the j-th link load is given. Let i be the feature vector of the i-th flow. Let j be the load vector of the j-th link. This represents the vector dot product operation. The system assigns weights based on the association scores and concatenates them to form the joint state vector for the next cycle. This joint state vector contains the association information of flow features and link loads required for the next decision cycle, providing input data for the new round of inference in the policy network and completing the iterative update of the cycle state.

[0087] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0088] Based on the same inventive concept, this application also provides a reinforcement learning-based network traffic scheduling device for implementing the reinforcement learning-based network traffic scheduling method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more reinforcement learning-based network traffic scheduling device embodiments provided below can be found in the limitations of the reinforcement learning-based network traffic scheduling method described above, and will not be repeated here.

[0089] Preferably, such as Figure 8 As shown, the present invention provides a network traffic scheduling device 700 based on reinforcement learning, which is configured with the following modules: The joint state generation module 710 is used to process the real-time collected network flow feature data and link load data, and generate a joint state vector by combining the flow attributes of flow type and latency sensitivity with the link status of link utilization and transmission latency. The attenuation factor generation module 720 is used to perform dynamic priority attenuation processing on the joint state vector and generate a dynamic attenuation factor matrix. The bandwidth ratio generation module 730 is used to process the joint state vector based on the preset policy network, calculate the action vector of the original action prediction and apply attenuation constraint in combination with the dynamic attenuation factor matrix, generate the bandwidth allocation ratio vector and send it to the data plane switch. The update allocation generation module 740 is used to process the bandwidth allocation execution results fed back by the data plane switch, detect the background flow waiting time and calculate the fairness entropy value, combine the deadlock risk threshold to determine the minimum guaranteed bandwidth of the background flow, and generate the updated bandwidth allocation result. The policy network update module 750 is used to process the updated bandwidth allocation results and joint state vector, calculate the real-time reward signal of the policy network based on the bandwidth allocation ratio and fairness entropy value, and update the policy network. The network status refresh module 760 is used to refresh the network status of the updated bandwidth allocation results, update the link load characteristics of the switch ports and the flow data characteristics of network packets, and generate the joint state vector for the next decision cycle.

[0090] Preferably, the joint state generation module 710 provided in this application is configured with the following modules: The flow attribute extraction unit is used to extract flow attributes from real-time acquired network flow feature data, parse the flow type identifier, latency requirement value and the amount of data to be transmitted, and generate a standard flow feature matrix. The port status acquisition unit is used to process the real-time collected link load data, measure the link utilization value and transmission delay value of the switch port, and generate a real-time link load vector. The attention allocation unit is used to allocate attention weights to the standard flow feature matrix and the real-time link load vector, calculate the cross-modal correlation score through the trainable parameter matrix, and generate a joint state vector.

[0091] Preferably, the attenuation factor generation module 720 provided in this application is configured with the following modules: The time decay calculation unit is used to perform time decay calculation on the delay requirement value in the joint state vector, calculate the lifetime decay coefficient based on the flow arrival timestamp and the current time difference, and generate time decay components. The load adjustment calculation unit is used to perform active flow statistical processing on the flow characteristic data in the joint state vector, count the number of flow records with a data volume greater than zero to obtain the number of active flows in the system, and calculate the load adjustment coefficient through logarithmic function transformation to generate load adjustment components. The matrix fusion unit is used to perform matrix fusion processing on the time decay component and the load adjustment component, combining the component values ​​by the flow identifier index to generate a dynamic decay factor matrix.

[0092] Preferably, the bandwidth ratio generation module 730 provided in this application is configured with the following modules: The policy reasoning unit is used to perform policy network reasoning on the joint state vector, calculate the original action probability distribution through a multilayer perceptron, and generate an unconstrained action vector. The attenuation constraint unit is used to perform attenuation constraint processing on the unconstrained action vector and the dynamic attenuation factor matrix, calculate the corrected action value of the flow level action value by element-wise multiplication, and generate the corrected action vector. The scaling conversion unit is used to perform bandwidth scaling conversion on the correction action vector. Based on the normalized allocation ratio of the total system bandwidth, it generates a bandwidth allocation ratio vector and sends it to the data plane switch. The bandwidth allocation ratio vector is used to instruct the data plane switch on the specific bandwidth allocation parameters for updating flow table entries.

[0093] Preferably, the update allocation generation module 740 provided in this application is configured with the following modules: The background flow monitoring unit is used to monitor the background flow status of the bandwidth allocation execution result data fed back by the data plane switch, calculate the average waiting time of all background flows, and generate background flow waiting indicators. The fairness assessment unit is used to perform fairness assessment on the actual allocation ratio in the execution result of the bandwidth allocation ratio vector, call the Shannon entropy formula to calculate the bandwidth distribution dispersion, and generate a fairness entropy value to indicate the degree of fairness in the allocation of system resources. The deadlock risk assessment unit is used to assess the deadlock risk of background flow waiting indicators and fairness entropy values. When the background flow waiting indicators exceed the preset waiting time threshold, the guaranteed bandwidth is reallocated and an updated bandwidth allocation result is generated.

[0094] Preferably, the policy network update module 750 provided in this application is configured with the following modules: The basic evaluation unit is used to perform basic performance evaluation processing on the updated bandwidth allocation results, calculate the total system throughput and average transmission latency, and generate basic reward components. The entropy deviation calculation unit is used to perform target deviation calculation on the fairness entropy value of the updated bandwidth allocation result, measure the absolute difference between the actual entropy value and the target entropy value, and generate entropy constraint reward components. The reward fusion unit is used to perform weighted fusion processing on the basic reward component and the entropy constraint reward component, calculate the final reward value by combining the preset weight coefficients, and generate a real-time reward signal. The policy update unit processes the real-time reward signal, calculates the policy gradient using a near-end policy optimization algorithm, and backpropagates to update the network weight parameters, thereby updating the policy network.

[0095] Preferably, the network status refresh module 760 provided in this application is configured with the following modules: The load recalculation unit is used to recalculate the link load of the updated bandwidth allocation results, update the port utilization value according to the actual allocated bandwidth, and generate a new link load vector. The flow feature update unit is used to perform flow feature update processing on network data packets related to the updated bandwidth allocation results, deduct the amount of transmitted data and maintain the status of live flows, and generate a new flow feature matrix. The periodic initialization unit is used to perform periodic state initialization processing on the new link load vector and the new flow feature matrix, clear non-live flow records and reset the time window, and generate the joint state vector for the next period.

[0096] In one embodiment, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described reinforcement learning-based network traffic scheduling method.

[0097] In one embodiment, this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described reinforcement learning-based network traffic scheduling method.

[0098] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0099] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A network traffic scheduling method based on reinforcement learning, characterized in that, The method includes: The network flow feature data and link load data collected in real time are processed, and the flow attributes of flow type and latency sensitivity are combined with the link status of link utilization and transmission latency to generate a joint state vector. The joint state vector is subjected to dynamic priority decay processing to generate a dynamic decay factor matrix; The joint state vector is processed based on a preset policy network to calculate the action vector of the original action prediction and apply attenuation constraints in combination with the dynamic attenuation factor matrix to generate a bandwidth allocation ratio vector and send it to the data plane switch. The bandwidth allocation execution results fed back by the data plane switch are processed, the background flow waiting time is detected and the fairness entropy value is calculated. The minimum guaranteed bandwidth of the background flow is adjusted based on the deadlock risk threshold, and the updated bandwidth allocation results are generated. The updated bandwidth allocation result and the joint state vector are processed to calculate the real-time reward signal of the policy network based on the bandwidth allocation ratio and the fairness entropy value, and the policy network is updated. The updated bandwidth allocation results are processed to refresh the network status, update the link load characteristics of the switch ports and the flow data characteristics of network packets, and generate a joint state vector for the next decision cycle.

2. The network traffic scheduling method based on reinforcement learning according to claim 1, characterized in that, The steps of processing the real-time collected network flow characteristic data and link load data, and combining the flow attributes of flow type and latency sensitivity with the link status of link utilization and transmission latency, include: Extract flow attributes from real-time collected network flow feature data, parse the flow type identifier, latency requirement value and the amount of data to be transmitted, and generate a standard flow feature matrix; At the port status acquisition point of the real-time collected link load data, the link utilization value and transmission delay value of the switch port are measured to generate a real-time link load vector. Attention weights are assigned to the standard flow feature matrix and the real-time link load vector, and cross-modal correlation scores are calculated using a trainable parameter matrix to generate a joint state vector.

3. The network traffic scheduling method based on reinforcement learning according to claim 1, characterized in that, The step of performing dynamic priority decay processing on the joint state vector includes: The time decay calculation is performed on the delay requirement value in the joint state vector, and the lifetime decay coefficient is calculated based on the flow arrival timestamp and the current time difference to generate the time decay component; The active flow statistics are processed on the flow feature data in the joint state vector. The number of flow records with a data volume greater than zero is counted to obtain the number of active flows in the system. The load adjustment coefficient is calculated by logarithmic function transformation to generate the load adjustment component. The time decay component and the load adjustment component are matrix-fused, and the component values ​​are combined according to the flow identifier index to generate a dynamic decay factor matrix.

4. The network traffic scheduling method based on reinforcement learning according to claim 1, characterized in that, The process of processing the joint state vector based on the preset policy network to calculate the action vector predicted from the original action and applying attenuation constraints in combination with the dynamic attenuation factor matrix includes the following steps: The joint state vector is processed by policy network inference, and the original action probability distribution is calculated by multilayer perceptron to generate unconstrained action vectors. The unconstrained action vector and the dynamic attenuation factor matrix are subjected to attenuation constraint processing. The corrected action value of the flow level action value is calculated by element-wise multiplication, and the corrected action vector is generated. The modified action vector is processed by bandwidth ratio conversion. Based on the normalized allocation ratio of the total system bandwidth, a bandwidth allocation ratio vector is generated and sent to the data plane switch. The bandwidth allocation ratio vector is used to indicate the specific bandwidth allocation parameters for the data plane switch to perform flow table entry updates.

5. The network traffic scheduling method based on reinforcement learning according to claim 1, characterized in that, The steps include processing the bandwidth allocation execution results fed back by the data plane switch, detecting the background flow waiting time and calculating the fairness entropy value, and determining and adjusting the minimum guaranteed bandwidth of the background flow based on the deadlock risk threshold. The bandwidth allocation execution results data fed back by the data plane switch are processed for background flow status monitoring, the average waiting time of all background flows is calculated, and background flow waiting indicators are generated. The actual allocation ratio in the execution result of the bandwidth allocation ratio vector is subjected to fairness evaluation processing. The Shannon entropy formula is called to calculate the bandwidth distribution dispersion and generate a fairness entropy value. The fairness entropy value is used to indicate the degree of fairness in the allocation of system resources. The background flow waiting index and the fairness entropy value are subjected to deadlock risk assessment. When the background flow waiting index exceeds the preset waiting time threshold, the guaranteed bandwidth is reallocated, and an updated bandwidth allocation result is generated.

6. The network traffic scheduling method based on reinforcement learning according to claim 1, characterized in that, The steps of processing the updated bandwidth allocation result and the joint state vector, calculating the real-time reward signal of the policy network based on the bandwidth allocation ratio and the fairness entropy value, and updating the policy network include: The updated bandwidth allocation results are subjected to basic performance evaluation processing to calculate the total system throughput and average transmission latency, and a basic reward component is generated. The fairness entropy value of the updated bandwidth allocation result is processed by target deviation calculation, the absolute difference between the actual entropy value and the target entropy value is measured, and an entropy constraint reward component is generated. The basic reward component and the entropy constraint reward component are weighted and fused, and the final reward value is calculated by combining the preset weight coefficients to generate a real-time reward signal. The real-time reward signal is processed, and the policy gradient is calculated using a near-end policy optimization algorithm and backpropagated to update the network weight parameters, thereby updating the policy network.

7. The network traffic scheduling method based on reinforcement learning according to claim 1, characterized in that, The steps of performing network state refresh processing on the updated bandwidth allocation results, updating the link load characteristics of switch ports and the flow data characteristics of network packets, and generating a joint state vector for the next decision cycle include: The updated bandwidth allocation result is recalculated for link load, and a new link load vector is generated based on the updated port utilization value according to the actual allocated bandwidth. The network data packets related to the updated bandwidth allocation results are processed for flow feature updates, the amount of data already transmitted is deducted and the status of live flows is maintained, and a new flow feature matrix is ​​generated. The new link load vector and the new flow feature matrix are subjected to periodic state initialization processing, non-live flow records are cleared and the time window is reset to generate the joint state vector for the next period.

8. A network traffic scheduling device based on reinforcement learning, the device being applicable to the network traffic scheduling method based on reinforcement learning as described in any one of claims 1 to 7, characterized in that, The device includes: The joint state generation module is used to process real-time collected network flow feature data and link load data, and generate a joint state vector by combining flow attributes with flow type and latency sensitivity with link utilization and transmission latency. The attenuation factor generation module is used to perform dynamic priority attenuation processing on the joint state vector to generate a dynamic attenuation factor matrix. The bandwidth ratio generation module is used to process the joint state vector based on a preset policy network, calculate the action vector of the original action prediction and apply attenuation constraints in combination with the dynamic attenuation factor matrix, generate a bandwidth allocation ratio vector and send it to the data plane switch. The update allocation generation module is used to process the bandwidth allocation execution results fed back by the data plane switch, detect the background flow waiting time and calculate the fairness entropy value, combine the deadlock risk threshold to determine the minimum guaranteed bandwidth of the background flow, and generate the updated bandwidth allocation result. The policy network update module is used to process the updated bandwidth allocation result and the joint state vector, calculate the real-time reward signal of the policy network based on the bandwidth allocation ratio and the fairness entropy value, and update the policy network. The network status refresh module is used to refresh the network status of the updated bandwidth allocation results, update the link load characteristics of the switch ports and the flow data characteristics of network packets, and generate a joint state vector for the next decision cycle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images