Traffic scheduling method and electronic device

Abstract

The application discloses a flow scheduling method and electronic equipment, and relates to the technical field of flow scheduling. The method comprises the following steps: determining the priority weight of a microservice, predicting target flow according to historical flow information of a network device; constructing a graph model according to the topology information of the network device and the dependency relationship of the microservice, and performing embedding learning on the nodes in the graph model to generate a state vector representing the network state; taking the priority weight of the microservice, the state vector and the target flow as inputs of a reinforcement learning model, obtaining a flow scheduling strategy of the network device; performing iterative search on iterative particles formed by coding the strategy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model to determine the parameters of the reinforcement learning model; and issuing the flow scheduling strategy to the network device and executing the flow scheduling strategy, so as to solve the technical problem that the flow scheduling method in the prior art is difficult to adapt to the dynamic and complex network environment and business demand under the microservice architecture, and improve the flow scheduling reliability.

Description

Technical Field

[0001] This application relates to the field of traffic scheduling technology, and in particular to a traffic scheduling method and electronic device. Background Technology

[0002] The rapid development of network technology and the widespread deployment of network infrastructure have promoted the digital transformation of society. The computation, transmission and storage of massive amounts of data have also driven the construction and intelligent upgrading of large data centers. Network traffic is the amount of data transmitted over the network.

[0003] In related technologies, the main approach is to query the real-time traffic components carried in the direction of congestion and the congestion status, evaluate the impact of scheduling different traffic components on different network paths, and generate and issue scheduling policies. However, this method only makes decisions from the single dimension of network congestion status and path performance, and cannot proactively perceive the complex call dependency topology between microservices. Its scheduling policy is lagging and lacks a global perspective. Therefore, it has the technical problem of being unable to adapt to the dynamic and complex network environment and business needs under the microservice architecture. Summary of the Invention

[0004] This application provides a traffic scheduling method and electronic device to at least solve the technical problem that traffic scheduling methods in the related art are difficult to adapt to the dynamic and complex network environment and business needs under the microservice architecture.

[0005] This application provides a traffic scheduling method, which includes:

[0006] The priority weights of microservices are calculated based on microservice information, including traffic information, dependencies, and interaction frequency. The target traffic of network devices in future periods is predicted based on historical traffic information. A graph model is constructed based on the network device topology and microservice dependencies. Embedding learning is performed on the nodes in the graph model to generate state vectors representing the network state. The priority weights of microservices, state vectors, and target traffic are used as inputs to a reinforcement learning model, which outputs a traffic scheduling strategy for the network devices. The parameters of the reinforcement learning model used to generate the traffic scheduling strategy are determined through iterative search based on the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model, forming iterative particles. The traffic scheduling strategy is then distributed to the network devices, and the traffic of the network devices is adjusted based on the traffic scheduling strategy.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the traffic scheduling method described in the following embodiments.

[0008] The priority weights of microservices are calculated based on microservice information, including traffic information, dependencies, and interaction frequency. The target traffic of network devices in future periods is predicted based on historical traffic information. A graph model is constructed based on the network device topology and microservice dependencies. Embedding learning is performed on the nodes in the graph model to generate state vectors representing the network state. The priority weights of microservices, state vectors, and target traffic are used as inputs to a reinforcement learning model, which outputs a traffic scheduling strategy for the network devices. The parameters of the reinforcement learning model used to generate the traffic scheduling strategy are determined through iterative search based on the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model, forming iterative particles. The traffic scheduling strategy is then distributed to the network devices, and the traffic of the network devices is adjusted based on the traffic scheduling strategy.

[0009] The traffic scheduling method provided in this application calculates the priority weights of microservices based on microservice information, including traffic information, dependencies, and interaction frequency of microservices. It predicts the target traffic of network devices in future periods based on historical traffic information of network devices within historical time periods. A graph model is constructed based on the topology information of network devices and the dependencies of microservices. Embedding learning is performed on the nodes in the graph model to generate state vectors representing the network state. The priority weights of microservices, the state vectors, and the target traffic are used as inputs to a reinforcement learning model, which outputs a traffic scheduling strategy for the network devices. The parameters of the reinforcement learning model used to generate the traffic scheduling strategy are determined through iterative search based on the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model, forming iterative particles. The traffic scheduling strategy is then distributed to the network devices, and the traffic of the network devices is adjusted based on the traffic scheduling strategy.

[0010] Thus, by calculating the priority weights of microservices based on microservice information, these weights participate in generating traffic scheduling strategies, linking network scheduling with business value and achieving precise alignment between the two. By predicting target traffic, the system can predict and avoid potential congestion. By constructing a graph model based on the topology information of network devices and the dependencies of microservices, embedding learning is performed on the nodes in the graph model to generate state vectors representing the network state. This achieves the fusion of discrete device states and service relationships into a single state vector, which deeply encodes the overall health status and structural characteristics of the network, providing comprehensive situational awareness for decision generation. Simultaneously, by using the priority weights of microservices, the state vector, and the target traffic as inputs to the reinforcement learning model, a near-optimal traffic scheduling strategy can be generated in a high-dimensional, continuous policy space by comprehensively balancing business fairness, real-time performance, and future load. Furthermore, the reinforcement learning model parameters in this application are determined through iterative search based on the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model, forming iterative particles that enable faster and more stable learning of high-performance scheduling strategies. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a traffic scheduling method provided in an embodiment of this application;

[0013] Figure 2 A flowchart illustrating a traffic scheduling method provided in another embodiment of this application;

[0014] Figure 3 A flowchart illustrating a traffic scheduling method provided in yet another embodiment of this application;

[0015] Figure 4 This is an internal structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0017] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or network device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or network device. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0018] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] The rapid development of network technology and the widespread deployment of network infrastructure have promoted the digital transformation of society. The computation, transmission, and storage of massive amounts of data have also driven the construction and intelligent upgrading of large data centers. Network traffic is the amount of data transmitted over a network. Traffic scheduling is essentially the process of adjusting network parameters such as BGP routing policies and IGP metrics to regulate network traffic. Traffic scheduling is divided into simulated scheduling and actual dispatch scheduling.

[0020] BGP routing policies are strategies that assign preference to a route, comparing it to other routes leading to the same destination; higher values ​​indicate higher preference. IGP metric values ​​refer to data related to network traffic scheduling and can include: bandwidth, which is the maximum data transmission rate of a network transmission channel; latency, which is the time required for data to travel from one end of a network or link to the other; and hop count, which is the number of routers a data packet passes through from its source to its destination, and the cost of reaching the target network.

[0021] Since abnormal traffic may occur during production operations, arbitrary modifications to network parameters are not advisable. Incorrect parameter changes could disrupt other services and cause network paralysis. Therefore, before issuing network parameters to network devices, a simulated scheduling process is necessary. The purpose of this simulation is to ensure that modifications to network device parameters achieve the desired traffic composition on relevant links without affecting the normal operation of the actual network. The simulation incorporates various network parameters of the analyzed and processed routing paths into the system's network model, simulating the adjusted routing and traffic flow of various services, and demonstrating whether the adjusted network traffic flow achieves the expected results. The simulation does not actually change the network traffic flow. Actual scheduling is implemented after the simulated network parameters meet the expected network adjustment results, and then the network parameter configuration is issued to the real network devices. This configuration is issued via BGP routing policy commands. After issuance, the network device parameters are changed, and the traffic flow changes accordingly. This achieves the purpose of traffic scheduling. Traffic scheduling, as a technology for effectively utilizing network resources, can optimize network performance and help the network quickly adapt to business changes.

[0022] In related technologies, after traffic monitoring detects network congestion, it queries the real-time traffic components carried in that direction based on the congestion direction and status. It then assesses the impact of scheduling different traffic components on different network paths and generates and issues a scheduling policy. Once traffic monitoring indicates the network is in a normal state, it evaluates the traffic impact after revoking the current scheduling policy; if revocation is possible, the policy is revoked. The results of policy issuance or revocation are then verified. This approach enables automatic scheduling and handling to ensure stable network operation when traffic congestion occurs.

[0023] However, it has the following drawbacks: it relies on BGP route modification and offline simulation, resulting in poor real-time performance; it lacks AI adaptive capabilities and cannot handle dynamic network changes; its rate limiting strategy is based on historical predictions, resulting in insufficient real-time performance; and it does not consider microservice priorities, among other shortcomings.

[0024] In response to the above technical problems, such as Figure 1 As shown, an embodiment of this application provides a traffic scheduling method, which specifically includes the following steps:

[0025] Step 101: Calculate the priority weight of microservices based on microservice information, which includes microservice traffic information, microservice dependencies, and microservice interaction frequency.

[0026] This system can collect real-time traffic, topology, and microservice information from network devices. Network devices can be switches (hubs), gateways, routers, and other devices that connect, forward, and manage data flows within a network. Network devices are crucial components of computer networks, belonging to network nodes. Traffic information from network devices can be collected in parallel using the NetFlow protocol (providing precise flow statistics) and the sFlow protocol (providing high-speed interface statistics based on sampling), with a sampling interval of ≤1 second. The NetFlow protocol is primarily used to collect traffic statistics in IP networks, including source and destination IP addresses, port numbers, protocol types, input and output interfaces, and key indicators of precise flow-level metadata such as input byte count, output byte count, packet loss rate, and latency. sFlow is a sampling-based traffic monitoring protocol that periodically samples network packets to obtain real-time network traffic data, such as input byte count, output byte count, and packet loss rate—interface-level statistical data based on sampling. Combining the NetFlow and sFlow protocols allows for more comprehensive and accurate acquisition of network device traffic information, with the sampling interval set to ≤1 second to ensure data real-time performance and validity.

[0027] The topology information of network devices can be obtained through LLDP (Link Layer Discovery Protocol) or the automatic topology discovery function of the SDN controller. It mainly includes network device connection relationships, link bandwidth status, and network device health status.

[0028] Microservices are an architectural style that uses a set of small services to create an independent application system. Microservices break down a large, complex software application into a set of small, independent services, each focused on performing a specific business function. These services communicate with each other through lightweight communication mechanisms (typically HTTP / RESTful APIs). Each service has its own database and autonomous deployment capabilities, and can be developed using different programming languages ​​and technology stacks. Each microservice is built around a specific business function and can be independently deployed to production, pre-production, etc. Microservice information can be microservice metadata, which is obtained in real-time from a service registry, such as the Kubernetes service mesh or Spring Cloud service registry. Specifically, in a Kubernetes service mesh environment, detailed information for each microservice, including service name, service address, port number, and service dependencies, can be obtained from its built-in service registry. Similarly, the Spring Cloud service registry provides similar service information and data on the frequency of interactions between microservices, providing a basis for subsequent service weight and priority calculations.

[0029] This section describes how to obtain microservice traffic information, microservice dependencies, and microservice interaction frequency. Microservice dependencies refer to how one microservice depends on the functionality or data provided by another microservice. This dependency can be direct, indirect, or even circular. Specifically: if microservice A directly calls microservice B's API to complete its business logic (direct dependency), and microservice A calls microservice C, which in turn calls microservice B, then microservice A indirectly depends on microservice B (indirect dependency), and microservice A depends on microservice B, while microservice B also depends on microservice A, forming a cycle (circular dependency). Microservice interaction frequency refers to the frequency of communication between microservices. This is usually closely related to the dependencies between services but can also be influenced by business requirements, system design, and other factors. Interaction frequency can be used to measure system load and assess potential performance bottlenecks.

[0030] The service weight of a microservice is calculated based on its traffic information, dependencies, interaction frequency, and service weight calculation formula. The total number of microservices is obtained, and the priority weight of each microservice is calculated based on the total number of microservices, their service weights, and the priority weight calculation formula.

[0031] The service weight calculation formula is as follows:

[0032] ;

[0033] Where X1 represents the traffic information weight of the microservice, X2 represents the dependency weight of the microservice, X3 represents the interaction frequency weight of the microservice, and X1+X2+X3=1, T represents the preset time window length, and V i (t) represents the traffic information of microservice i collected at time t, D i Y represents the number of microservices that depend on microservice i, C represents the set of interacting services, which is the set of other microservices that interact with microservice i, and Y represents the number of microservices that depend on microservice i. （i,k） This represents the interaction frequency between microservice i and microservice k within a preset time window T.

[0034] here, F i This represents the total historical traffic of microservice i within the time window T. For example, data from the past hour can be used as a time window. For each microservice, the total number of bytes counted at all traffic sampling points within that time window is the historical traffic summation for that microservice. E i This represents the total interaction frequency of microservice i within a time window T. It collects the number of interactions between microservice i and other microservices within a preset time window and sums them to obtain the total interaction frequency.

[0035] Given that X1+X2+X3=1, the specific values ​​of X1, X2, and X3 can be set according to actual needs. For example, in a business system that is more sensitive to historical traffic, the weight of X1 can be increased appropriately; if the service dependency relationship has a greater impact on network traffic scheduling, the weight of X2 should be increased; conversely, if the interaction frequency has a more significant impact on business performance, the weight of X3 should be increased accordingly.

[0036] W i The larger the value, the more important the microservice. Based on the historical traffic sum, the number of dependent services, the total interaction frequency, and the determined weight coefficients, the service weight of each microservice is calculated to reflect the importance of each microservice in network traffic and its demand for network resources.

[0037] The priority weight calculation formula is as follows:

[0038] ;

[0039] Among them, w i This indicates the priority weight of microservice i. This represents the sum of the service weights of the microservices, where M is the total number of microservices.

[0040] Service weights for microservices are used to comprehensively consider the importance and priority of services during traffic scheduling, providing a reference for the generation of subsequent scheduling strategies. This application calculates the service weights of microservices based on their traffic, dependencies, and interaction frequency; calculates their priority weights based on these service weights; and generates traffic scheduling strategies based on these priority weights. This allows the traffic scheduling strategies to dynamically reflect the actual needs and importance differences on the business side, thereby ensuring that high-priority services receive the necessary network resource guarantees and avoiding a disconnect between network resource allocation and business value.

[0041] Step 102: Predict the target traffic of network devices in future periods based on the historical traffic information of network devices in historical periods.

[0042] Specifically, a short-term network traffic prediction model is constructed, and historical traffic information of network devices within a historical period is input into the short-term network traffic prediction model so as to output the target traffic of network devices in the future period through the short-term network traffic prediction model.

[0043] Here, constructing a short-term network traffic prediction model can be achieved by building a lightweight LSTM prediction model (Lite-LSTM), such as... Figure 2 As shown, the Lite-LSTM model reduces the number of parameters through weight pruning. The basic structure of the Lite-LSTM model is similar to that of the standard LSTM, including components such as input gates, forget gates, output gates, and cell states. However, to improve the model's lightweight nature, weight pruning is used to reduce the number of parameters. Weight pruning employs amplitude pruning, setting a threshold and removing weights with absolute values ​​less than that threshold, thereby compressing the model. This method reduces the number of model parameters while preserving the model's predictive performance as much as possible.

[0044] This application defines the input to the short-term network traffic prediction model as historical traffic time series data formed by historical traffic information of network devices. Typically, traffic information from several past time steps (e.g., the past 10 sampling intervals) is selected as the input sequence, and the output is the predicted traffic for future periods, such as predicting the traffic value for one or more future sampling intervals, in order to provide a short-term traffic prediction basis for network traffic scheduling. For example, in this application, the input to the Lite-LSTM model is a historical traffic time series of n time steps. Output for the future Predicted flow after the time period ;generally Set to 1 to predict the flow rate at the next moment.

[0045] This application also utilizes historical traffic information collected from network devices to train the Lite-LSTM model, employing mean squared error (MSE) as the loss function. The parameters of the short-term network traffic prediction model are updated using the backpropagation algorithm, continuously optimizing the prediction accuracy. During training, common optimization algorithms such as the Adam optimizer can be used, with appropriate hyperparameters such as the learning rate and number of iterations set to ensure the short-term network traffic prediction model can effectively learn the time-series features and patterns in the traffic information, thereby achieving accurate prediction of network traffic.

[0046] Specifically, the short-term network traffic prediction model is trained based on a loss function, which is shown below:

[0047] ;

[0048] in, The mean squared error (MSE) represents the loss value used to train a short-term network traffic prediction model. The smaller the MSE, the more accurate the prediction. N represents the number of training samples, and V(t) represents the actual traffic value at time t. This represents the predicted flow rate of the model at time t.

[0049] This application uses historical traffic prediction to obtain target traffic for future periods, enabling the system to understand traffic change trends and thus transform scheduling decisions from passive response to proactive optimization, effectively avoiding potential congestion.

[0050] Step 103: Construct a graph model based on the topology information of network devices and the dependencies of microservices, and perform embedding learning on the nodes in the graph model to generate state vectors representing the network state.

[0051] The graph model defined in this application embodies network topology and microservice dependencies. Specifically, network devices and microservices in the system are abstracted as nodes in the graph model, and the topological information of network devices and the dependencies of microservices are abstracted as edges in the graph model to construct the graph model. The graph model can be a GAT model, which stands for Graph Attention Network, a neural network model for processing graph-structured data. Graph Attention Network (GAT) introduces an attention mechanism, providing a more refined way to learn node features in graph neural networks, enabling the model to better understand the complex relationships in graph-structured data and improving its performance on graph-related tasks. In a graph, the effects of a node's neighbors are not equal or symmetrical. GAT allows each node to assign weights based on the importance of its neighboring nodes, allowing the model to more flexibly capture the relationships between nodes in the graph.

[0052] In this application, the initial feature vector of a node is used as input to the graph model, and the initial feature vector of each node i consists of its type and its real-time data. When node i is a network device node, the real-time data of its initial feature vector comes from the collected network device traffic information and network device topology information. When node i is a microservice node, the real-time data of its initial feature vector comes from microservice information, the service weight of the microservice calculated based on the microservice information, and the priority weight of the microservice. That is, the initial features of a node are directly composed of network device traffic information and microservice metadata.

[0053] After obtaining the initial feature vectors of the nodes, these vectors are used as input to the graph model. The graph model aggregates the initial feature vectors of the node and its neighboring nodes. An attention mechanism is used to assign different aggregation weights to different neighboring nodes. After multi-layer propagation using the feature propagation formula, the final embedded representation of the node is generated. Pooling is then performed on the final embedded representation of the node to generate a state vector representing the network state. The state vector representing the network state is expressed as:

[0054] ;

[0055] s t The state vector represents the network state and is also the input DDPG graph-level state representation vector. `Readout()` represents the graph readout function, such as global average pooling, used to aggregate the node embeddings into a global graph representation. i (L) This represents the final embedded representation of node i after processing at the L-th layer of the graph model, which is the final and richest representation after processing the entire GAT model. L represents the total number of layers in the GAT model; here, h... i (L) The initial feature vectors containing node i and all its neighboring nodes are the final outputs of the entire GAT model and are used as the state inputs s of the DDPG. t In the middle, C represents the set of nodes.

[0056] In this application, the state vector representing the network state generated by the GAT model is used as the higher-level specific content and digital representation of the state space of the DDPG algorithm. The state space refers to the set of all information on which the agent (DDPG) makes decisions. In this application, the set of all information on which the agent (DDPG) makes decisions specifically includes the real-time traffic information vector (the initial feature vector of network device nodes), the network topology state vector (the connection relationship vector of network device nodes, how control information is propagated in GAT), the priority weight vector (the initial feature vector of microservice nodes), and the predicted traffic vector (the predicted target traffic vector). These information together constitute the complete state space describing the network traffic scheduling environment.

[0057] The feature propagation formula is expressed as:

[0058] ;

[0059] h i (l+1) The embedding representation of node i at layer l+1 of the graph model is obtained by aggregating the embedding representations of node i and its neighboring nodes j at layer l of the graph model. σ represents a nonlinear function, and α represents the embedding representation of node i at layer l. ij W represents the attention weight, which measures the importance of neighbor node j to node i. （l） Let h represent the trainable weight matrix. j (l) This represents the embedding representation of node i's neighbor node j in the l-th layer of the graph model.

[0060] Attention weights are represented as follows:

[0061] ;

[0062] Where exp() represents the exponential function and LeakfyReLU() represents the activation function. Let n(i) represent the attention vector, and n(i) represent the set of neighboring nodes j of node i. Let W(l) be the transpose of the attention vector, and let h be the trainable weight matrix. i (l) h represents the embedding representation of node i in the l-th layer of the graph model. j (l) This represents the embedding representation of node i's neighbor node j in the l-th layer of the graph model.

[0063] In this application, the set n(i) of neighbor nodes j is determined by the graph model (topology information of network devices and microservice dependencies), h i (l) and h j (l)In the first iteration (l=0), the initial features consist of traffic information and metadata. In subsequent iterations, these are fused representations of this data after nonlinear transformation. For example, the representation h of input node i in the current layer (layer l) i (l) The representation of h of h and all its neighboring nodes in the current layer. j (l) Trainable weight matrix W （l） and used for attention vectors Calculate the embedding vector h of node i in the next layer (l+1) i (l+1) .

[0064] Therefore, the attention mechanism's dynamic information fusion process relies on the original traffic and metadata, by transforming the features W of all neighbors. （l） h j （l） According to the calculated attention weight α ij A weighted summation process is performed to fuse multi-source data, including traffic, status, and priority information, from different neighboring nodes (which can be network devices or microservices). Through the GAT message passing mechanism, network device traffic information (as node characteristics), network device topology information, and microservice metadata (collectively defining graph connectivity) undergo deep, attention-based interaction and fusion at each layer. The final result... It is a powerful representation that condenses local and global contextual information.

[0065] For example, the GAT model may include three layers. The first layer is the input layer, which receives the initial feature vector h of the nodes. i (0) As input to GAT, it generates the individual original features h of each node. i （l） The second layer is the intermediate layer, located in the middle of information propagation. It is used to perform nonlinear transformations and feature extraction within the GAT model to obtain h. i (l+1) The third layer is the final layer, and it serves as the output h of the graph readout function. i (L) Aggregate to generate a global state vector s t The graph model in this application is used to learn the structural features of network topology to understand the relationships between nodes (such as distance, bandwidth, and congestion level), and encodes this complex, non-Euclidean structural information into meaningful numerical vectors (i.e., embedding representations).

[0066] Here, we first consider the current feature h of each neighbor node j. j(l) A linear transformation is performed and multiplied by a trainable weight matrix to map it to a new feature space, enhancing the model's expressive power. Then, attention weights are calculated between node i and each of its neighboring nodes j. These attention weights represent the importance of neighboring node j to node i and are calculated using a small neural network (typically a single-layer feedforward network) whose input is the concatenation of the transformed features of nodes i and j.

[0067] The feature W after transforming all neighbors （l） h j （l） The calculated attention weights are then weighted and summed. In this way, node i selectively absorbs information from its neighbors, with more important neighbors contributing more. Finally, the weighted summed information is processed by a nonlinear function. (For example, ReLU) output, to obtain the embedding vector of node i in the next layer (l+1).

[0068] By first performing linear transformations on the features of node i and its neighbor node j respectively, and then concatenating the transformed features, we obtain the concatenated features. Utilizing learnable attention vectors The concatenated features are scored, and a relevance score is calculated. The correlation score reflects the importance of a node's neighboring nodes to node i.

[0069] To facilitate stable training, relevance scores are converted into weights. Specifically, the Softmax function is used to normalize the scores of all neighbors. The Softmax function is calculated as: the exponential score of a given neighbor / the sum of the exponential scores of all neighbors. The resulting normalized attention weights are then used to determine the attention weight assigned to each neighbor node i based on the relevance score of its neighbor node j.

[0070] By constructing a graph model by combining the topology information of network devices with the dependencies of microservices, and using the graph model for embedding learning to generate a state vector representing the overall state of the network for each node, this vector deeply encodes the structural features and real-time operating status of the network, providing unprecedented global situational awareness capabilities for decision-making.

[0071] Step 104: Use the priority weights, state vectors, and target traffic of the microservices as input to the reinforcement learning model, and output the traffic scheduling policy of the network device through the reinforcement learning model; wherein, the reinforcement learning model parameters used to generate the traffic scheduling policy are determined by iterative search based on the iterative particles formed by encoding the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model.

[0072] Specifically, the state vector is fused with the target traffic to form an enhanced state representation; the enhanced state representation is input into the policy network of the reinforcement learning model to obtain the initial scheduling action; the initial scheduling action is weighted and corrected using the priority weights of microservices to output the final traffic scheduling policy, which is represented as the bandwidth allocation policy of network devices or the data flow forwarding path policy.

[0073] For example, the state space formed by the state vectors representing the network state generated by GAT is a list of numbers, such as 64 numbers [x1, x2, x3, ..., x64]. It condenses the current state of the network, including which links are busy, which devices are critical nodes, and how microservices depend on each other. hat This represents a vector of target flows predicted by the LSTM model. The target flow is typically a scalar (a numerical value) or a low-dimensional vector (such as a predicted flow at several future time points). For concatenation, the target flow needs to be converted into a vector. If the target flow is a single value, creating a one-dimensional vector can be represented as [V]. hat If multiple values ​​are predicted, it can be represented as [V]. hat(t+1) V hat(t+2) The target traffic vector is itself a low-dimensional vector. The state vector representing the network state generated by GAT is concatenated with the target traffic vector to form an enhanced state representation. For example, the enhanced state representation se = [x1, x2, x3, ..., x64, V...]. hat If the state vector is 64-dimensional and the target traffic vector is 1-dimensional, then the augmented state representation is a 64+1=65-dimensional vector. This results in an augmented state representation that integrates the current network state and future traffic predictions. This augmented state representation is then input into the policy network of the reinforcement learning model. The policy network is a deep neural network that, after training, can map complex state information to optimal control actions. Upon receiving the augmented state representation, the policy network performs calculations and outputs an initial scheduling action. For example, if the output is a bandwidth allocation policy: the initial scheduling action might be a vector [0.3, 0.4, 0.3], suggesting that 30%, 40%, and 30% of the total bandwidth be allocated to the three microservice groups or data flows, respectively.

[0074] If the output is a path selection strategy: the initial scheduling action might be a probability distribution [0.7, 0.2, 0.1], meaning that for a new data flow, there is a 70% probability of choosing path A, a 20% probability of choosing path B, and a 10% probability of choosing path C. However, the initial decision in the first stage may not fully consider the business importance of microservices. Therefore, it is necessary to introduce microservice priority weights to correct this, which can be achieved by making the final resource allocation to each microservice as close as possible to the proportion of its priority weight. That is, to give higher-priority services more resources.

[0075] The initial scheduling action is compared with the priority weight vector of the microservice. The initial scheduling action can be modified by taking a weighted average of the initial scheduling action vector and the priority weight vector, or by scaling the initial scheduling action proportionally based on the priority weight. The modified initial scheduling action is the final traffic scheduling strategy for network devices output by the reinforcement learning model.

[0076] For example, suppose there are three microservices with priority weights w. i1 = [0.5, 0.3, 0.2] (Microservice A is the most important). The initial scheduling action output by the policy network is a. tc = [0.3, 0.4, 0.3]. After weighted adjustments, the final scheduling strategy may be adjusted to a. tz = [0.45, 0.35, 0.2]. In this example, the bandwidth allocation for the most important microservice A is increased from 30% to 45%, closer to its 50% weight; while the allocation for the less important microservices is correspondingly reduced. This ensures network efficiency while greatly reflecting the fairness of the business.

[0077] The revised initial scheduling action can be a higher-level specific content and digital representation of the action space, which is a continuous value space mapped to a path selection probability distribution or bandwidth allocation ratio. For example, bandwidth allocation ratio: a tz1 =[bw_1,bw_2,...,bw_m] (the proportion allocated to m services or links, summing to 1). Path selection probability: a tz2 =[p_1,p_2,...,p_n] (probabilities assigned to n possible paths, summing to 1). Regarding path selection, for each data stream or microservice request, the optimal transmission path is selected based on the probability distribution output by the policy network. Regarding bandwidth allocation, network bandwidth resources are allocated to different services or traffic based on the bandwidth allocation ratio in the action space, thereby achieving effective scheduling of network traffic.

[0078] In one embodiment, the reinforcement learning model parameters for generating the traffic scheduling strategy include: encoding the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model to form iterative particles; and guiding the particle swarm to perform iterative search in the joint parameter space of the policy network parameters and the feature learning network parameters according to the particle swarm optimization algorithm and the fitness function to query the optimal reinforcement learning model parameters.

[0079] The reinforcement learning model in this application is essentially based on DDPG as its algorithm core, but GAT replaces the first fully connected layer (or the entire perception module) at the state input end of the original DDPG. GAT acts as a more powerful state encoder here. GAT encodes the original, structured network topology information (nodes, links and their features) into a fixed-dimensional state vector that can profoundly reflect the global and local relationships of the network, and passes it to the subsequent layers of the DDPG policy network and value network for decision-making. Therefore, the model here is specifically a hybrid model based on GAT-DDPG.

[0080] The feature learning network parameters for a graph model can be the trainable weights of the graph neural network itself. Specifically, these might include: graph convolutional layer weights used to aggregate neighbor node information, attention mechanism parameters that determine the importance of different neighbors when aggregating information, and fully connected layer weights used to further transform the learned node embeddings. Different network topologies (e.g., data center networks, wide area networks) or different traffic patterns require different feature extraction methods. Optimizing these parameters means allowing the model to automatically learn the feature representation best suited to the current traffic scheduling task.

[0081] The policy network is the "brain" of the reinforcement learning agent. It receives states, such as topological features processed by the graph model, real-time link utilization, and traffic demand, and then outputs an action, i.e., a scheduling decision, such as routing data packets from path A to path B, or allocating traffic to server C. The policy network parameters of the reinforcement learning model can be the trainable weights of the policy neural network itself, specifically including the weights and biases of the multilayer perceptron. The policy network parameters of the reinforcement learning model directly determine how the model makes decisions based on the current state. Optimizing the policy network parameters of the reinforcement learning model helps find an optimal policy that accumulates rewards over the long term. By concatenating the graph feature network parameters and the policy network parameters, an iterative particle is formed. Assuming the graph network has 3 parameters and the policy network has 4 parameters, the position of each particle is a (3+4) dimensional vector.

[0082] The particle swarm optimization algorithm is expressed as:

[0083] ;

[0084] ;

[0085] Among them, v r k+1 Let ω represent the velocity vector of particle r at the (k+1)th iteration, ω represent the adaptive inertia weight, k represent the current iteration number, c1 and c2 represent acceleration constants, controlling the influence of individual and social experience respectively, r1 and r2 represent uniformly distributed random numbers in the interval [0,1], used to increase the randomness of the search and avoid getting trapped in local optima, pbest represents the best position that particle r has reached historically, and gbest represents the global best position that the particle swarm has reached in all iterations. , ρ r k This represents the position of particle r in the k-th iteration, which directly encodes the policy network parameters θ of the reinforcement learning model and the feature learning network parameters of the graphical model. .

[0086] here, The adaptive inertia weights are represented by c1=c2=1.5, the particle swarm size can be 20, and the maximum number of iterations can be 50; this is achieved by guiding the particle swarm in the joint parameter space ρ. r k We perform an efficient search to find the optimal combination of parameters that enables the reinforcement learning model to obtain the highest cumulative reward.

[0087] The fitness function is expressed as:

[0088] ;

[0089] Fitness (ρ) r () indicates that particle r represents parameter ρ r The fitness value represents the optimization objective of the IPSO algorithm, πρ r Indicated by parameter ρ r Defined traffic scheduling policy (traffic scheduling policy generated by GAT-DDPG). A complete interaction trajectory is represented by a sequence of states, actions, and rewards. This represents a discount factor, giving greater importance to recent rewards, r t This represents the reward value obtained by the reinforcement learning model parameters at time t. This reward value is calculated based on the reward function.

[0090] The fitness function can be the IPSO function, the purpose of which is to quantitatively evaluate the parameters ρ for each set. r The goal of IPSO is to maximize this fitness value, taking into account the strengths and weaknesses of each fitness level.

[0091] First, a particle swarm (i.e., multiple sets of random joint parameters) is randomly generated. The position of each particle is randomly initialized (including the weights of GAT and DDPG). For each particle, its parameter ρ is used. r k Instantiate the GAT-DDPG model, run it for multiple rounds in a simulated environment or historical data, and calculate its fitness value (ρ). r The current fitness value is compared with the particle's own historical best (pbest) and the global historical best (gbest), and pbest and gbest are updated. According to the particle swarm optimization algorithm, the search direction (velocity) and position (parameter set) of each particle are updated using the information from pbest and gbest. The fitness value, pbest and gbest, and the search direction (velocity) and position (parameter set) of each particle are iteratively calculated. Particles will move towards their own historical best position pbest and the global best position gbest. This movement process involves searching in the joint parameter space until the maximum number of iterations is reached. Finally, the parameter set ρ corresponding to the global best position gbest is... r These parameters are set for the final GAT-DDPG model and are used for actual online traffic scheduling.

[0092] For example, the particle swarm size can be set to 20, and the maximum number of iterations can be set to 50. The choice of particle swarm size requires a trade-off between search efficiency and solution quality; too small a size may lead to insufficient searching, while too large a size will increase computational costs. The maximum number of iterations is set according to the complexity of the actual problem and the required convergence speed to ensure that the algorithm can converge to a relatively optimized solution within a reasonable time. It is understood that the specific value of the particle swarm size is not limited to 20, and the specific value of the maximum number of iterations is not limited to 50; these values ​​can be set according to actual needs.

[0093] In the process of updating particle velocity based on particle swarm optimization algorithm, the adaptive inertia weight is dynamically adjusted according to factors such as the number of iterations during the algorithm operation. The initial value can be set to about 0.9, and gradually decreased as the number of iterations increases, so as to ensure the global search capability of particles in the early stage and enhance the local search capability in the later stage. c1, c2=1.5, r1, r2 are random numbers uniformly distributed in the interval [0,1] to increase the randomness and diversity of particle velocity update and prevent particles from getting trapped in local optima.

[0094] This application establishes a collaborative optimization framework integrating GAT, DDPG, and IPSO. By deeply integrating graph attention networks with the DDPG algorithm, it enables the model to fully understand the complex relationships in network topology, thereby providing a state representation rich in structural information for decision-making. The parameters of the DDPG policy network and the feature extraction parameters of GAT are jointly encoded into the positions of particles in the improved particle swarm optimization algorithm, so that each particle represents a complete set of GAT-DDPG joint model parameter configurations. During optimization, IPSO acts as a meta-optimizer, guiding the particle swarm to iteratively search throughout the joint parameter space. Each generation of particles is evaluated based on its performance (i.e., fitness function) in the network simulation environment to guide the population to evolve towards better regions. Ultimately, the globally optimal particle position output by IPSO represents the fully optimized combination of model parameters. Thus, through the deep integration of reinforcement learning and meta-heuristic search, the problem of high-quality policy generation and stable training in complex environments is solved. The aforementioned microservice priority weights, global state vectors, and predicted target traffic are jointly input into the reinforcement learning model, enabling it to generate near-optimal traffic scheduling policies in a high-dimensional, continuous policy space, comprehensively balancing business fairness, real-time performance, and future load. The particle swarm optimization algorithm approved in this application automatically searches for the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model as joint optimization objectives. This design effectively overcomes the problems of unstable training and easy getting trapped in local optima in deep reinforcement learning models, and significantly improves the convergence speed and final performance of the policy generation model.

[0095] Step S105: Distribute the traffic scheduling policy to the network devices and adjust the traffic of the network devices based on the traffic scheduling policy.

[0096] After the traffic scheduling policy is distributed to the network devices and before the traffic of the network devices is adjusted based on the traffic scheduling policy, the process includes: obtaining the actual network performance data after the network devices execute the traffic scheduling policy; calculating the actual reward value based on the actual network performance data and the priority weight of the microservices; calculating the deviation between the actual reward value and the preset reward value; in response to the deviation value being greater than the preset threshold, updating the decision experience of the reinforcement learning model based on the actual network performance data and the actual reward value; and training the reinforcement learning model based on the updated decision experience to adjust the policy network parameters of the reinforcement learning model.

[0097] Please see Figure 3The Software-Defined Networking (SDN) controller translates the final output actions of DDPG into specific OpenFlow flow table entries (such as setting queue bandwidth and modifying forwarding paths). These flow table entries are then distributed to the corresponding network devices via the OpenFlow protocol. The controller distributes traffic scheduling policies to the network devices and collects real-time data on the actual average network latency and average link utilization after the traffic scheduling policies are implemented.

[0098] The `traceroute` command is used on the server to verify whether the actual forwarding path of the data packets is consistent with the path specified by the policy, ensuring that the scheduling actions output by the reinforcement learning model are correctly executed by the network devices. The deviation between the actual reward value and the preset reward value triggers a closed-loop feedback mechanism. If the deviation value is greater than the preset threshold, it indicates that the model's prediction is seriously inconsistent with the actual situation, and that the current scheduling policy deviates significantly from the actual network conditions, failing to meet the expected performance requirements. At this time, the closed-loop feedback mechanism is triggered to update the DDPG experience pool, storing the current network state (topology information processed by GAT, real-time link utilization, device queue length, traffic demand, etc.), scheduling actions, the actual reward value calculated from the network performance data measured after executing the policy corresponding to the scheduling action, and the new state of the entire network system after executing the policy corresponding to the scheduling action, into the experience replay pool.

[0099] Sample from the updated experience pool and retrain the Actor and Critic networks (the policy network parameters of the reinforcement learning model) of DDPG. Continue until the bias value is less than or equal to a preset threshold, then encode it to indicate that the current traffic scheduling strategy is effective. Maintain the current strategy and continue monitoring.

[0100] The network performance data includes average network latency and average link utilization. The actual reward value is calculated based on the actual network performance data and the priority weight of the microservices. The calculation of the actual reward value is based on the actual average network latency, the actual average link utilization, the priority weight of the microservices, and the reward value calculation formula.

[0101] The formula for calculating the reward value is as follows:

[0102] ;

[0103] Where, r t λ1 represents the reward value, λ2 represents the delay weight, λ3 represents the utilization weight, and λ4 represents the allocation fairness weight. t This represents the average network latency at time t; the smaller the better. (utilization) t This represents the average network link utilization at time t; the higher the better (1-utilization).t ) represents the penalty term, alloc i w represents the actual bandwidth allocation ratio for microservice i. i This indicates the priority weight of microservice i. This represents the sum of the absolute errors between the actual allocation and the theoretical allocation of a microservice. The smaller this value is, the more the scheduling strategy matches the priority weight of the microservice.

[0104] The traffic scheduling strategy generated by the reinforcement learning model guided by the reward function can simultaneously meet the requirements of minimizing latency and maximizing utilization, thereby improving the overall performance and resource utilization of the network.

[0105] By leveraging a software-defined network architecture, precise policy delivery and closed-loop feedback are achieved. The software-defined SDN controller, based on the OpenFlow protocol, enables flexible management and control of network devices, ensuring that scheduling policies can be executed quickly and accurately in the actual network. Furthermore, the closed-loop feedback mechanism can update the DDPG experience pool and retrain the model in a timely manner based on verification errors, guaranteeing continuous optimization and effectiveness of the scheduling policies. This ensures that the server network maintains a highly efficient and stable operating state when facing constantly changing traffic demands.

[0106] In one implementation, overall network performance metrics can be monitored in real time, including the percentile of the network-wide average latency, the number of times the critical link utilization threshold is exceeded, and the sliding window statistics of microservice priority fulfillment rate. If the percentile of the network-wide average latency continuously exceeds the preset average latency threshold, the latency weight is automatically increased, favoring latency optimization. If the critical link utilization frequently exceeds the safety threshold (e.g., 90%), the utilization weight is automatically increased, focusing more on load balancing. If the resource allocation fulfillment rate of high-priority microservices decreases, the fairness weight is automatically increased, reinforcing the principle of priority-based scheduling. The latency weight, utilization weight, and allocation fairness weight are adjusted periodically or triggered, and the adjusted weights are applied to the reward calculation, thereby guiding the reinforcement learning model to dynamically adapt to the network optimization priorities at different times. In this way, the reward function is transformed from a static optimization objective into a dynamic adaptive optimization. At different stages of network operation, such as congestion avoidance during peak business periods, energy efficiency optimization during daily operations, and service level agreement stages to ensure critical business operations, the system can automatically adjust the focus of optimization, thereby possessing stronger environmental adaptability and long-term effectiveness in complex multi-objective trade-offs.

[0107] Embodiments of this application also provide an electronic device, such as... Figure 4 As shown, it includes a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above-described traffic scheduling method embodiments.

[0108] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described traffic scheduling method embodiments at runtime.

[0109] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), mobile network device, magnetic disk, or optical disk.

[0110] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] The above provides a detailed description of a traffic scheduling method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A traffic scheduling method, characterized in that, The traffic scheduling method includes: The priority weight of a microservice is calculated based on microservice information, which includes microservice traffic information, microservice dependencies, and microservice interaction frequency. Predict the target traffic of network devices in future periods based on historical traffic information of network devices in historical periods; A graph model is constructed in the reinforcement learning model based on the topology information of network devices and the dependencies of microservices. The graph model is used to replace the perception module in the original reinforcement learning model. Embedding learning is performed on the nodes in the graph model to generate state vectors that represent the network state. The state vector is fused with the target traffic to form an enhanced state representation; the enhanced state representation is input into the policy network of the reinforcement learning model to obtain an initial scheduling action; the initial scheduling action is weighted and corrected using the priority weights of the microservices to output the final traffic scheduling policy. The reinforcement learning model parameters used to generate the traffic scheduling strategy are the optimal reinforcement learning model parameters obtained by iteratively searching the particle swarm formed by the particle swarm guided by the particle swarm optimization algorithm and fitness function in the joint parameter space of the policy network parameters and the feature learning network parameters. The iterative particles are formed by encoding the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model. The traffic scheduling policy is distributed to network devices, and the traffic of the network devices is adjusted based on the traffic scheduling policy.

2. The traffic scheduling method according to claim 1, characterized in that, The calculation of microservice priority weights based on microservice information includes: Obtain microservice traffic information, microservice dependencies, and microservice interaction frequency; The service weight of a microservice is calculated based on the traffic information of the microservice, the dependencies of the microservice, the interaction frequency of the microservice, and the service weight calculation formula. Obtain the total number of microservices, and calculate the priority weight of each microservice based on the total number of microservices, the service weight of each microservice, and the priority weight calculation formula. The service weight calculation formula is as follows: ； Where X1 represents the traffic information weight of the microservice, X2 represents the dependency weight of the microservice, X3 represents the interaction frequency weight of the microservice, T represents the preset time window length, and V i (t) represents the traffic information of microservice i collected at time t, D i Y represents the number of microservices that depend on microservice i, C represents the set of interacting services, which is the set of other microservices that interact with microservice i, and Y represents the number of microservices that depend on microservice i. （i,k） This represents the interaction frequency between microservice i and microservice k within a preset time window T; The priority weight calculation formula is as follows: ； Among them, w i This indicates the priority weight of microservice i. This represents the sum of the service weights of the microservices, where M is the total number of microservices.

3. The traffic scheduling method according to claim 1, characterized in that, The method of predicting the target traffic of network devices in future periods based on historical traffic information of network devices in historical periods includes: A short-term network traffic prediction model is constructed by inputting the historical traffic information of network devices within the historical period into the short-term network traffic prediction model, so as to output the target traffic of network devices in the future period through the short-term network traffic prediction model. The short-term network traffic prediction model is trained based on a loss function, which is shown below: ； Among them, L MSE The loss value is used to train a short-term network traffic prediction model, where N represents the number of training samples and V(t) represents the actual traffic value at time t. This represents the predicted flow rate of the model at time t.

4. The traffic scheduling method according to claim 1, characterized in that, The step of constructing a graph model based on the topology information of network devices and the dependencies of microservices, and performing embedding learning on the nodes in the graph model to generate state vectors representing the network state includes: The network devices and microservices in the system are abstracted as nodes in the graph model, and the topology information of the network devices and the dependencies of the microservices are abstracted as edges in the graph model to construct the graph model. The initial feature vector of the node is used as the input of the graph model, and the initial feature vectors of the node and its neighboring nodes are aggregated through the graph model. By using an attention mechanism to assign different aggregation weights to different neighbor nodes, and then generating the final embedded representation of the node through multi-layer propagation using a feature propagation formula; The final embedded representation of the node is pooled to generate a state vector representing the network state. The state vector representing the network state is expressed as: ； s t The state vector represents the network state, and Readout() is the graph readout function used to aggregate the node embeddings into a global graph representation. i (L) h represents the final embedded representation of node i after processing at the Lth layer of the graph model. i (L) The initial feature vectors of node i and all its neighboring nodes are included, and C represents the set of nodes; The feature propagation formula is expressed as follows: ； h i (l+1) The embedding representation of node i at layer l+1 of the graph model is obtained by aggregating the embedding representations of node i and its neighboring nodes j at layer l of the graph model. σ represents a nonlinear function, and α represents the embedding representation of node i at layer l. ij W represents the attention weight, which measures the importance of neighbor node j to node i. （l） Let h represent the trainable weight matrix. j (l) This represents the embedding representation of node i's neighbor node j in the l-th layer of the graph model.

5. The traffic scheduling method according to claim 4, characterized in that, The attention weights are represented as follows: ； Where exp() represents the exponential function and LeakfyReLU() represents the activation function. Let n(i) represent the attention vector, and n(i) represent the set of neighboring nodes j of node i. Let W(l) be the transpose of the attention vector, and let h be the trainable weight matrix. i (l) h represents the embedding representation of node i in the l-th layer of the graph model. j (l) This represents the embedding representation of node i's neighbor node j in the l-th layer of the graph model.

6. The traffic scheduling method according to claim 1, characterized in that, The traffic scheduling strategy is expressed as a bandwidth allocation strategy or a data flow forwarding path strategy for network devices.

7. The traffic scheduling method according to claim 1, characterized in that, The particle swarm optimization algorithm is expressed as follows: ； ； Among them, v r k+1 Let ω represent the velocity vector of particle r at the (k+1)th iteration, ω represent the adaptive inertia weight, k represent the current iteration number, c1 and c2 represent acceleration constants, controlling the influence of individual and social experience respectively, r1 and r2 represent uniformly distributed random numbers in the interval [0,1], used to increase the randomness of the search and avoid getting trapped in local optima, pbest represents the best position that particle r has reached historically, and gbest represents the global best position that the particle swarm has reached in all iterations. , ρ r k This represents the position of particle r in the k-th iteration, which directly encodes the policy network parameters θ of the reinforcement learning model and the feature learning network parameters of the graphical model. ; The fitness function is expressed as follows: ； Among them, Fitness (ρ r () indicates that particle r represents parameter ρ r The fitness value represents the optimization objective of the IPSO algorithm, πρ r Indicated by parameter ρ r Defined traffic scheduling strategy, A complete interaction trajectory is represented by a sequence of states, actions, and rewards. This represents a discount factor, giving greater importance to recent rewards. t This represents the reward value obtained by the reinforcement learning model parameters at time t.

8. The traffic scheduling method according to claim 1, characterized in that, After the traffic scheduling policy is distributed to the network device, but before adjusting the traffic of the network device based on the traffic scheduling policy, the following steps are included: Obtain actual network performance data after the network device executes the traffic scheduling strategy; The actual reward value is calculated based on the actual network performance data and the priority weight of the microservice. The deviation between the actual reward value and the preset reward value is calculated. In response to the deviation value being greater than the preset threshold, the decision experience of the reinforcement learning model is updated based on the actual network performance data and the actual reward value. The reinforcement learning model is then trained based on the updated decision experience to adjust the policy network parameters of the reinforcement learning model.

9. The traffic scheduling method according to claim 8, characterized in that, Network performance data includes average network latency and average link utilization. The calculation of the actual reward value based on the actual network performance data and the priority weights of the microservices includes: The actual reward value is calculated based on the actual average network latency, actual average link utilization, microservice priority weights, and the reward value calculation formula. The formula for calculating the reward value is expressed as follows: ； Where, r t λ1 represents the reward value, λ2 represents the delay weight, λ3 represents the utilization weight, and λ4 represents the allocation fairness weight. t The average network latency at time t is represented by the utilization. t This represents the average network link utilization at time t, (1-utilization) t ) represents the penalty term, alloc i w represents the actual bandwidth allocation ratio for microservice i. i This indicates the priority weight of microservice i. This represents the sum of the absolute errors between the actual allocation and the theoretical allocation of microservices.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the traffic scheduling method as described in any one of claims 1 to 9 when executing the computer program.

Images