Traffic scheduling method and electronic equipment

Abstract

The invention discloses a traffic scheduling method and an electronic device, and relates to the technical field of traffic scheduling, and the method comprises the steps: determining the priority weight of a micro-service, and predicting a target traffic according to the historical traffic information of a network device; constructing a graph model according to the topological information of the network equipment and the dependency relationship of the micro-service, and performing embedded learning on nodes in the graph model to generate a state vector representing a network state; the priority weight, the state vector and the target traffic of the micro-service serve as input of a reinforcement learning model, and a traffic scheduling strategy of the network equipment is obtained; performing iterative search according to iterative particles formed by encoding the strategy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model to determine reinforcement learning model parameters; and issuing the traffic scheduling strategy to the network equipment and executing the traffic scheduling strategy so as to solve the technical problem that a traffic scheduling method in related technologies is difficult to adapt to a dynamic and complex network environment and service requirements under a micro-service architecture, and the reliability of traffic scheduling is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traffic scheduling, and in particular to a traffic scheduling method and an electronic device. BACKGROUND

[0002] The rapid development of network technology and the extensive deployment of network infrastructure have promoted the digital transformation of society, and the calculation, transmission and storage of massive data have also promoted the construction and intelligent upgrading of large data centers. Network traffic is the amount of data transmitted on the network.

[0003] In the related art, the real-time traffic components in the direction are queried according to the congestion direction and the congestion condition, the influence caused by scheduling different traffic components on different network paths is evaluated, a scheduling strategy is generated and issued, however, this method only makes decisions from a single dimension of the congestion condition and the path performance of the network itself, and cannot actively perceive the complex calling and depending topology between microservices, the scheduling strategy is lagging and lacks a global view, therefore, there is a technical problem that the traffic scheduling method in the related art is difficult to adapt to the dynamic and complex network environment and business requirements under the microservice architecture. SUMMARY

[0004] The present application provides a traffic scheduling method and an electronic device to at least solve the technical problem that the traffic scheduling method in the related art is difficult to adapt to the dynamic and complex network environment and business requirements under the microservice architecture.

[0005] The present application provides a traffic scheduling method, which comprises: calculating the priority weight of the microservice based on the microservice information, the microservice information comprising the traffic information of the microservice, the dependency relationship of the microservice and the interaction frequency of the microservice; predicting the target traffic of the network device in a future period based on the historical traffic information of the network device in a historical period; constructing a graph model according to the topology information of the network device and the dependency relationship of the microservice, and performing embedding learning on the nodes in the graph model to generate a state vector representing the network state; inputting the priority weight of the microservice, the state vector and the target traffic into a reinforcement learning model as inputs, and outputting a traffic scheduling strategy of the network device through the reinforcement learning model; wherein the reinforcement learning model parameters used to generate the traffic scheduling strategy are determined by iterative searching of iterative particles formed by encoding the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model; and issuing the traffic scheduling strategy to the network device and adjusting the traffic of the network device based on the traffic scheduling strategy.

[0006] The present application also provides an electronic device comprising a memory for storing a computer program and a processor for executing the computer program to implement the steps of the traffic scheduling method in the embodiments.

[0007] The priority weight of the microservice is calculated based on microservice information, the microservice information including traffic information of the microservice, a dependency relationship of the microservice, and an interaction frequency of the microservice; target traffic of the network device in a future period is predicted according to historical traffic information of the network device in a historical period; a graph model is constructed according to topology information of the network device and the dependency relationship of the microservice, and embedding learning is performed on nodes in the graph model to generate a state vector representing a network state; the priority weight of the microservice, the state vector, and the target traffic are taken as inputs of a reinforcement learning model, and a traffic scheduling strategy of the network device is output by the reinforcement learning model; wherein the reinforcement learning model parameter used to generate the traffic scheduling strategy is determined by iterative search of an iterative particle formed according to policy network parameters of the reinforcement learning model and feature learning network parameters of the graph model; and the traffic scheduling strategy is sent to the network device, and traffic of the network device is adjusted based on the traffic scheduling strategy.

[0008] The traffic scheduling method provided in the application calculates the priority weight of the microservice based on microservice information, the microservice information including traffic information of the microservice, a dependency relationship of the microservice, and an interaction frequency of the microservice; target traffic of the network device in a future period is predicted according to historical traffic information of the network device in a historical period; a graph model is constructed according to topology information of the network device and the dependency relationship of the microservice, and embedding learning is performed on nodes in the graph model to generate a state vector representing a network state; the priority weight of the microservice, the state vector, and the target traffic are taken as inputs of a reinforcement learning model, and a traffic scheduling strategy of the network device is output by the reinforcement learning model; wherein the reinforcement learning model parameter used to generate the traffic scheduling strategy is determined by iterative search of an iterative particle formed according to policy network parameters of the reinforcement learning model and feature learning network parameters of the graph model; and the traffic scheduling strategy is sent to the network device, and traffic of the network device is adjusted based on the traffic scheduling strategy.

[0009] Thus, by calculating the priority weight of the microservice according to the microservice information, the priority weight of the microservice is involved in generating the traffic scheduling strategy, the network scheduling is associated with the business value, the accurate alignment of the network scheduling and the business value is realized, the system can predict and avoid potential congestion through the predicted target traffic, by constructing a graph model according to the topology information of the network device and the dependency relationship of the microservice, the node in the graph model is embedded to generate a state vector representing the network state, the discrete device state and service relationship are fused into a state vector, which deeply encodes the overall health status and structural features of the network, providing comprehensive situation awareness capability for decision making, at the same time, the priority weight of the microservice, the state vector and the target traffic are input into the reinforcement learning model, which can comprehensively weigh the business fairness, real-time performance and future load in the high-dimensional and continuous policy space, and generate a traffic scheduling strategy close to the optimal, and the reinforcement learning model parameters in the present application are determined by iterative search of the iterative particles coded according to the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model, which can learn a high-performance scheduling strategy faster and more stably. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0011] Figure 1 Flowchart of the traffic scheduling method provided by an embodiment of the present application; Figure 2 Flowchart of the traffic scheduling method provided by another embodiment of the present application; Figure 3 Flowchart of the traffic scheduling method provided by another embodiment of the present application; Figure 4 Internal structure diagram of the electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0012] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0013] It should be noted that in the description of the present application, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or network device including a series of elements includes not only those elements, but also other elements not explicitly listed, or further includes elements inherent to such a process, method, article or network device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, not to describe a specific order or sequence.

[0014] In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below in conjunction with the drawings and specific embodiments.

[0015] The rapid development of network technology and the extensive deployment of network infrastructure promote the development of social digital transformation. The calculation, transmission and storage of massive data also promote the construction and intelligent upgrading of large data centers. Network traffic is the amount of data transmitted on the network. Traffic scheduling is actually to adjust the network traffic by adjusting the network parameters such as BGP routing strategy and IGP Metric value. Traffic scheduling is divided into simulation scheduling and actual scheduling.

[0016] BGP routing strategy is a strategy that gives a route a degree of preference for comparing it with other routes to the same destination. The higher the value, the higher the degree of preference. IGP metric value refers to data related to network traffic scheduling, which can include: bandwidth, that is, the maximum data transmission rate of the network transmission channel. Delay refers to the time required for data to be transmitted from one end of the network or link to the other end. The number of hops refers to the number of routers that a data packet passes through from the source to the destination, as well as the cost of reaching the target network.

[0017] Since when the traffic anomaly occurs, it is possible that the enterprise is in production, at this time it is not possible to modify the network parameters at will, because if the parameter modification is wrong, it may affect other services and cause the entire network to be paralyzed, so before the network parameters of the network device are issued, the network parameters of the current network device need to be simulated and scheduled. The purpose of simulation scheduling is to ensure that the traffic composition on the relevant link after modifying the network device parameters achieves the expected effect and does not affect the normal operation of the actual network. The simulation scheduling brings the various network parameters of the analyzed and processed routing path into the network model of the system, simulates the adjusted routing of the network and the flow direction of various business traffic, and presents the flow direction of the adjusted network traffic, whether it can achieve the expected effect. The simulation scheduling does not actually change the flow direction of the network, and the actual scheduling is to configure and issue the network parameters to the real network device after the simulated network parameters meet the expected effect of network adjustment. The configuration and issuance can be issued to the device through the BGP routing strategy related command. After the issuance, the network parameters of the network device are changed, and the flow direction is also changed. In this way, the purpose of traffic scheduling is achieved. As an effective network resource utilization technology, traffic scheduling can optimize network performance and help the network quickly adapt to business changes.

[0018] In the related art, after the traffic monitoring finds that the current network is in a congested state, according to the congestion direction and the congestion condition, the real-time bearing traffic composition in the direction is queried, the influence caused by scheduling different traffic compositions on different network paths is evaluated, and a scheduling strategy is generated and issued; after the traffic monitoring finds that the current network is in a normal state, the traffic evaluation after the strategy revocation of the currently running scheduling strategy is performed, and if the scheduling strategy can be revoked, the scheduling strategy is revoked; and the result verification is performed after the scheduling strategy is issued or revoked. In this way, when the network appears traffic congestion, automatic scheduling disposal is realized to ensure the stable operation of the network.

[0019] However, the following defects exist: dependence on BGP route modification and offline simulation, poor real-time performance; lack of AI adaptive ability, unable to handle dynamic network changes, insufficient real-time performance of the flow limiting strategy, and failure to consider the priority of microservices and other defects.

[0020] In view of the above technical problems, as shown in Figure 1 The embodiment of the present application provides a traffic scheduling method, which specifically comprises the following steps: Step 101: calculating the priority weight of the microservice based on the microservice information, wherein the microservice information comprises traffic information of the microservice, a dependency relationship of the microservice, and an interaction frequency of the microservice.

[0021] The traffic information of the network device, the topology information of the network device, and the micro-service information can be collected in real time. The network device can be a switch (hub), gateway, router, and the like, which plays a role in connecting, forwarding, and managing data flow in the network. The network device is an important component of a computer network, and it belongs to a network node. The traffic information of the network device can be collected in parallel through a NetFlow protocol (providing accurate flow statistical information) and an sFlow protocol (providing sampled high-speed interface statistical information) with a sampling interval ≤ 1 second. The NetFlow protocol is mainly used to collect traffic statistical information in an IP network, including the source IP address and destination IP address, port number, protocol type, input and output interface, and other information of a data packet, as well as key indicators of accurate flow-level metadata such as the number of input bytes, the number of output bytes, the packet loss rate, and the time delay. The sFlow is a kind of sampled traffic monitoring protocol, which obtains the real-time situation of network traffic, such as the number of input bytes, the number of output bytes, the packet loss rate, and other sampled interface-level statistical data, by periodically sampling network data packets. The combination of the NetFlow protocol and the sFlow protocol can more comprehensively and accurately obtain the traffic information of the network device, and the sampling interval is set to be less than or equal to 1 second to ensure the real-time and effectiveness of the data.

[0022] The topology information of the network device can be obtained through LLDP (Link Layer Discovery Protocol) or the automatic topology discovery function of the SDN controller, mainly including the connection relationship of the network device, the link bandwidth state, the health state of the network device, and the like.

[0023] Microservices is an architectural style that involves developing a set of small services as an independent application system. Microservices is an architectural style that involves breaking down a large, complex software application into a set of small, independent services, each focused on completing a specific business function. These services communicate with each other through lightweight communication mechanisms, typically HTTP / RESTful APIs. Each service has its own database and autonomous deployment capabilities, and can be developed using different programming languages and technology stacks. Each service in a microservices architecture is built around a specific business function and can be independently deployed to production, staging, or other environments. Microservices information can be microservices metadata, which is obtained in real-time from a service registry, such as a Kubernetes service mesh or a Spring Cloud service registry. For example, in a Kubernetes service mesh environment, detailed information about each microservice can be obtained from the built-in service registry, including service name, service address, port number, service dependency relationships, and other information. In a Spring Cloud service registry, similar service information and interaction frequency data between microservices can also be obtained, providing a basis for subsequent service weight and priority weight calculations.

[0024] Obtain traffic information of microservices, dependency relationships of microservices, and interaction frequency of microservices. Dependency relationships of microservices refer to how one microservice depends on the functionality or data provided by another microservice. This dependency can be direct or indirect, and even may occur in a circular dependency situation. Specifically: if microservice A directly calls the API of microservice B to complete its own business logic (direct dependency). If microservice A calls microservice C, and microservice C calls microservice B, then microservice A indirectly depends on microservice B (indirect dependency). If microservice A depends on microservice B, and microservice B also depends on microservice A, forming a loop (circular dependency). Interaction frequency of microservices refers to the frequency of communication between microservices. This is usually closely related to the dependency relationships between services, but can also be influenced by business requirements, system design, and other factors. Interaction frequency can be used to measure system load and assess potential performance bottlenecks.

[0025] Based on the traffic information of microservices, the dependency relationships of microservices, the interaction frequency of microservices, and the service weight calculation formula, calculate the service weight of microservices. Obtain the total number of microservices, and calculate the priority weight of microservices according to the total number of microservices, the service weight of microservices, and the priority weight calculation formula.

[0026] The service weight calculation formula is as follows: ; wherein X1 represents the traffic information weight of the microservice, X2 represents the dependency relationship weight of the microservice, X3 represents the interaction frequency weight of the microservice, and X1+X2+X3=1 is satisfied, T represents a preset time window length, V i (t) represents the traffic information of the microservice i collected at time t, D i represents the number of microservices having a dependency relationship with the microservice i, C represents an interaction service set, which is a set of other microservices having an interaction with the microservice i, Y （i,k） represents the interaction frequency of the microservice i with the microservice k within the preset time window T.

[0027] Here, , F i represents the historical traffic sum of the microservice i within the time window T, for example, taking the data of the past 1 hour as the time window. For each microservice, the sum of the byte numbers of all traffic sampling points within the time window is counted, that is, the historical traffic sum of the microservice. , E i represents the interaction frequency sum of the microservice i within the time window T, the interaction times of the microservice with other microservices within the preset time window are collected, and the sum is obtained to obtain the interaction frequency sum.

[0028] In the case of satisfying X1+X2+X3=1, the specific values of X1, X2 and X3 can be set according to actual needs, for example, in a business system which is more sensitive to historical traffic, the weight of X1 can be appropriately increased; if the service dependency relationship has a greater impact on network traffic scheduling, the weight of X2 is increased; otherwise, if the interaction frequency has a more significant impact on business performance, the weight of X3 is increased accordingly.

[0029] W i is greater, it indicates that the microservice is more important. Based on the historical traffic sum, the number of dependent services, the interaction frequency sum and the determined weight coefficient obtained by the above calculation, the service weight of each microservice is calculated to reflect the importance of each microservice in the network traffic and the demand degree of network resources.

[0030] wherein the priority weight calculation formula is as follows: ; wherein w i represents the priority weight of the microservice i, represents the sum of the service weights of the microservices, and M is the total number of microservices.

[0031] The service weight of the microservice is used to comprehensively consider the importance and priority of the service in the traffic scheduling process, and provides a reference basis for subsequent scheduling strategy generation. In the present application, the service weight of the microservice is calculated based on the traffic, dependency relationship and interaction frequency of the microservice, the priority weight of the microservice is calculated according to the service weight of the microservice, and the traffic scheduling strategy is generated according to the priority weight of the microservice, so that the traffic scheduling strategy can dynamically reflect the real demand and importance difference of the business side, thereby ensuring that the high-priority service obtains the necessary network resource guarantee, thereby avoiding the disconnection between network resource allocation and business value.

[0032] Step 102: predicting the target traffic of the network device in the future period according to the historical traffic information of the network device in the historical period.

[0033] Specifically, a short-term network traffic prediction model is constructed, and the historical traffic information of the network device in the historical period is input into the short-term network traffic prediction model to output the target traffic of the network device in the future period through the short-term network traffic prediction model.

[0034] Here, the short-term network traffic prediction model can be a lightweight LSTM prediction model (Lite-LSTM) for short-term network traffic prediction, as shown in Figure 2 As shown, the Lite-LSTM model reduces the parameter amount through weight pruning. The Lite-LSTM model has a basic structure similar to that of a standard LSTM, including input gates, forget gates, output gates, and cell states, etc. However, in order to improve the lightweight degree of the model, the weight pruning technique is used to reduce the parameter amount of the model. The weight pruning adopts the amplitude pruning method, sets a threshold, removes the weight parameters with an absolute value less than the threshold in the model, thereby realizing the compression of the model. This method can reduce the parameter amount of the model while retaining the prediction performance of the model as much as possible.

[0035] In the present application, the input of the short-term network traffic prediction model is defined as the historical traffic time series data formed by the historical traffic information of the network device, and the traffic information of the past several time steps (for example, the traffic information of the past 10 sampling intervals) is usually selected as the input sequence, and the predicted traffic in the future period is output, for example, the traffic value in the future 1 or more sampling intervals is predicted, so as to provide a short-term traffic prediction basis for network traffic scheduling. For example, in the present application, the input of the Lite-LSTM model is the historical traffic time series of n time steps , and the output is the predicted traffic after the future period ; usually 1, the predicted traffic value of the next moment is predicted.

[0036] The historical traffic information of the collected network device can also be used to train the Lite-LSTM model in the present application, mean square error (MSE) is used as the loss function, and the short-term network traffic prediction model parameters are updated through the back propagation algorithm to continuously optimize the prediction accuracy of the short-term network traffic prediction model. In the training process, common optimization algorithms such as Adam optimizer can be used, and appropriate learning rate, iteration number and other hyperparameters are set to ensure that the short-term network traffic prediction model can effectively learn the time sequence characteristics and rules in the traffic information, thereby realizing accurate prediction of network traffic.

[0037] Specifically, the short-term network traffic prediction model is trained according to the loss function, and the loss function is as follows: ; Among them, represents the loss value, which is used to train the short-term network traffic prediction model, the smaller the mean square error loss value, the more accurate the prediction, N represents the number of training samples, V(t) represents the true traffic value at time t, represents the predicted traffic value of the model at time t.

[0038] In the present application, the target traffic of the future period is obtained through the prediction of the historical traffic, so that the system can understand the traffic change trend, thereby changing the scheduling decision from passive response to active optimization, and effectively avoiding potential congestion.

[0039] Step 103: constructing a graph model according to the topology information of the network device and the dependency relationship of the microservice, and performing embedding learning on the nodes in the graph model to generate a state vector representing the network state.

[0040] The graph model defined in the present application embodies the network topology and microservice dependency. Specifically, the network devices and microservices in the system are abstracted as nodes in the graph model, and the topology information of the network devices and the dependency relationship of the microservices are abstracted as edges in the graph model to construct the graph model. The graph model can be a GAT model, GAT refers to a graph attention network, which is a neural network model for processing graph structure data. The graph attention network (GAT) provides a more fine-grained node feature learning method for graph neural networks by introducing an attention mechanism, so that the model can better understand the complex relationships in the graph structure data and improve the performance of the model in graph-related tasks. In the graph, the neighbors of a node do not have equal and symmetric effects on it. GAT allows each node to assign weights according to the importance of its neighbor nodes, which makes the model more flexible to capture the relationships between nodes in the graph.

[0041] In the present application, the initial feature vector of the node is set as the input of the graph model, and the initial feature vector of each node i is composed of its type and real-time data. When node i is a network device node, the real-time data of its initial feature vector comes from the collected traffic information of the network device and the topology information of the network device. When node i is a microservice node, the real-time data of its initial feature vector comes from the microservice information, the service weight of the microservice calculated based on the microservice information, and the priority weight of the microservice. That is, the initial feature of the node is directly composed of the traffic information of the network device and the microservice metadata.

[0042] After obtaining the initial feature vector of the node, the initial feature vector of the node is set as the input of the graph model, and the initial feature vectors of the node and its neighbor nodes are aggregated by the graph model; different aggregation weights are assigned to different neighbor nodes by using the attention mechanism, and the final embedding representation of the node is generated after multi-layer propagation by the feature propagation formula; the final embedding representation of the node is pooled to generate a state vector representing the network state; wherein the state vector representing the network state is represented as: ; s t The state vector representing the network state is also the input DDPG graph level state representation vector, and Readout() represents a graph readout function, for example, global average pooling, which is used to aggregate the embedding representation of the node into a global graph representation. i (L) The final embedding representation of the node i obtained after the Lth layer processing of the graph model is represented as h i (L) The initial feature vector of node i and all neighbor nodes of node i is the final output of the entire GAT model, which is used as the state input s t of DDPG in the subsequent process. C represents a set of nodes.

[0043] In the present application, the state vector representing the network state generated by the GAT model is used as the specific content and digital embodiment of the state space of the DDPG algorithm. The state space refers to the set of all information on which the agent (DDPG) makes decisions. In the present application, the set of all information on which the agent (DDPG) makes decisions specifically includes the real-time traffic information vector (the initial feature vector of the network device node), the network topology state vector (the connection relationship vector of the network device node, which controls how the control information propagates in the GAT), the priority weight vector (the initial feature vector of the microservice node), and the predicted traffic vector (the target traffic vector predicted). These information comprehensively constitute the complete state space describing the network traffic scheduling environment.

[0044] The feature propagation formula is expressed as: ; h i (l+1) represents the embedding representation of node i at the (l+1)th layer of the graph model, which is updated by aggregating the embedding representation of node i and its neighbor node j at the lth layer of the graph model, σ represents a nonlinear function, and α ij represents the attention weight, which is used to measure the importance of neighbor node j to node i, W （l） represents a trainable weight matrix, h j (l) represents the embedding representation of neighbor node j of node i at the lth layer of the graph model.

[0045] The attention weight is expressed as: ; wherein exp() represents an exponential function, LeakyReLU() represents an activation function, represents an attention vector, n(i) represents a set of neighbor nodes j of node i, represents a transposed vector of the attention vector, W(l) represents a trainable weight matrix, and h i (l) represents the embedding representation of node i at the lth layer of the graph model, h j (l) represents the embedding representation of neighbor node j of node i at the lth layer of the graph model.

[0046] In the present application, the set of neighbor nodes j n(i) is determined by the graph model (topological information of the network device and microservice dependency), h i (l) and h j (l) At the first iteration (l=0), they are the initial features composed of traffic information and metadata. In subsequent iterations, they are the fused representations of these data after nonlinear transformation. For example, by inputting the representation h i (l) of node i at the current layer (lth layer) and the representation h j (l) of all neighbor nodes of node i at the current layer, a trainable weight matrix W （l） and an attention vector are calculated to obtain the embedding vector h i (l+1) of node i at the next layer (l+1th layer).

[0047] Therefore, the attention mechanism's dynamic information fusion process relies on the original traffic and metadata, by transforming the features W of all neighbors. （l） h j （l） According to the calculated attention weight α ij A weighted summation process is performed to fuse multi-source data, including traffic, status, and priority information, from different neighboring nodes (which can be network devices or microservices). Through the GAT message passing mechanism, network device traffic information (as node characteristics), network device topology information, and microservice metadata (collectively defining graph connectivity) undergo deep, attention-based interaction and fusion at each layer. The final result... It is a powerful representation that condenses local and global contextual information.

[0048] For example, the GAT model may include three layers. The first layer is the input layer, which receives the initial feature vector h of the nodes. i (0) As input to GAT, it generates the individual original features h of each node. i （l） The second layer is the intermediate layer, located in the middle of information propagation. It is used to perform nonlinear transformations and feature extraction within the GAT model to obtain h. i (l+1) The third layer is the final layer, and it serves as the output h of the graph readout function. i (L) Aggregate to generate a global state vector s t The graph model in this application is used to learn the structural features of network topology to understand the relationships between nodes (such as distance, bandwidth, and congestion level), and encodes this complex, non-Euclidean structural information into meaningful numerical vectors (i.e., embedding representations).

[0049] Here, we first consider the current feature h of each neighbor node j. j (l) A linear transformation is performed and multiplied by a trainable weight matrix to map it to a new feature space, enhancing the model's expressive power. Then, attention weights are calculated between node i and each of its neighboring nodes j. These attention weights represent the importance of neighboring node j to node i and are calculated using a small neural network (typically a single-layer feedforward network) whose input is the concatenation of the transformed features of nodes i and j.

[0050] The feature W after transforming all neighbors （l） h j （l）The weighted sum is performed according to the calculated attention weight, so that the node i selectively absorbs information from its neighbor nodes, the contribution of important neighbor nodes is greater, and finally the information after the weighted sum is output through a nonlinear function (for example, ReLU) to obtain the embedding vector of node i in the next layer (the l+1 layer).

[0051] The transformed features of the node i and the neighbor node j are spliced to obtain spliced features The spliced features are scored using a learnable attention vector to calculate a correlation score The correlation score reflects the importance of the neighbor node to the node i.

[0052] In order to facilitate stable training, the correlation score is converted into a weight. Specifically, the scores of all neighbors are normalized using a Softmax function. The Softmax function: the exponential score of a neighbor / the sum of the exponential scores of all neighbors. The final result is the normalized attention weight. In this way, the node i finally determines the attention weight value allocated to each neighbor node according to the correlation score of the neighbor node j.

[0053] By constructing a graph model using the topology information of the network device and the dependency relationship of the microservice, and using the graph model for embedding learning to generate a state vector representing the overall state of each node, the vector deeply encodes the structural features and real-time running status of the network, providing an unprecedented global situational awareness capability for decision-making.

[0054] Step 104: input the priority weight, state vector, and target traffic of the microservice into the reinforcement learning model, and output the traffic scheduling strategy of the network device through the reinforcement learning model; wherein the reinforcement learning model parameter used to generate the traffic scheduling strategy is determined by iterative search of the iterative particles coded from the policy network parameter of the reinforcement learning model and the feature learning network parameter of the graph model.

[0055] Specifically, the state vector and the target traffic are fused to form an enhanced state representation; the enhanced state representation is input into the policy network of the reinforcement learning model to obtain an initial scheduling action; the initial scheduling action is weighted and corrected using the priority weight of the microservice to output a final traffic scheduling strategy, which represents a bandwidth allocation strategy or a data flow forwarding path strategy of the network device.

[0056] ​Exemplarily, the state space formed by the state vector generated by GAT to represent the network state is a list of numbers, such as 64 numbers [x1, x2, x3,..., x64]. It condenses the current situation of the network, which links are busy, which devices are key nodes, how microservices depend on each other, etc. hat The vector representing the target traffic predicted by the LSTM model. The target traffic here is usually a scalar (a numerical value) or a low-dimensional vector (such as predicting the traffic at several future time points). In order to splice, the target traffic needs to be converted into a vector. If the target traffic is a value, create a one-dimensional vector, which can be represented as [V hat ]. If multiple values are predicted, it can be represented as [V hat(t+1) ,V hat(t+2) ], and the vector of target traffic itself is a low-dimensional vector. The state vector generated by GAT to represent the network state is spliced with the vector of target traffic to form an enhanced state representation. Exemplarily, the enhanced state representation se = [x1, x2, x3,..., x64, V hat ]. If the state vector is 64-dimensional and the target traffic vector is 1-dimensional, then the enhanced state representation is a 64+1=65-dimensional vector. In this way, the enhanced state representation that integrates the current network state and future traffic prediction is obtained, and the enhanced state representation that integrates the current network state and future traffic prediction is input into the policy network of the reinforcement learning model. The policy network is a deep neural network that is trained to map complex state information to the optimal control action. After the policy network receives the enhanced state representation, it will perform calculations and output an initial scheduling action. Exemplarily, if the output is a bandwidth allocation policy: the initial scheduling action may be a vector [0.3, 0.4, 0.3], indicating that it is recommended to allocate 30%, 40%, and 30% of the total bandwidth to the three microservice groups or data flows, respectively.

[0057] If the output is a path selection policy: the initial scheduling action may be a probability distribution [0.7, 0.2, 0.1], indicating that for a new data flow, there is a 70% probability of selecting path A, a 20% probability of selecting path B, and a 10% probability of selecting path C. However, the initial decision in the first stage may not fully consider the business importance of microservices. Therefore, the priority weight of microservices needs to be introduced for correction, which can be achieved by making the proportion of resources finally allocated to each microservice as close as possible to the proportion of its priority weight. That is, let high-priority services obtain more resources.

[0058] The initial scheduling action is compared with the priority weight vector of the microservice. The initial scheduling action can be modified by weighted averaging the initial scheduling action vector with the priority weight vector, or by scaling the initial scheduling action proportionally based on the priority weight. The modified initial scheduling action is the final traffic scheduling strategy of the network device output by the reinforcement learning model.

[0059] For example, assume that there are three microservices with priority weights w i1 = [0.5, 0.3, 0.2] (microservice A is the most important). The initial scheduling action output by the policy network is a tc = [0.3, 0.4, 0.3]. After weighted modification, the final scheduling strategy can be adjusted to a tz = [0.45, 0.35, 0.2]. In this example, the bandwidth proportion allocated to the most important microservice A is increased from 30% to 45%, closer to its weight of 50%; and the proportions of the secondary microservices are correspondingly reduced. In this way, the fairness of the business is greatly reflected while ensuring network efficiency.

[0060] The modified initial scheduling action can be a higher-level specific content and a digital representation of the action space, which is a continuous value space, mapped to a path selection probability distribution or a bandwidth allocation proportion. For example, the bandwidth allocation proportion: a tz1 = [bw_1, bw_2, …, bw_m] (proportions allocated to m services or links, summing to 1). The path selection probability: a tz2 = [p_1, p_2, …, p_n] (probabilities allocated to n selectable paths, summing to 1). In terms of path selection, for each data flow or microservice request, the optimal transmission path is selected according to the probability distribution output by the policy network; in terms of bandwidth allocation, different services or traffic are allocated corresponding network bandwidth resources according to the bandwidth allocation proportion value in the action space, to achieve effective scheduling of network traffic.

[0061] In an embodiment, the reinforcement learning model parameters for generating the traffic scheduling strategy include: encoding the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model to form an iterative particle; guiding the particle swarm to perform iterative search in the joint parameter space of the policy network parameters and the feature learning network parameters according to the particle swarm optimization algorithm and the fitness function, and querying the optimal reinforcement learning model parameters.

[0062] The reinforcement learning model in the present application is essentially based on DDPG as the algorithm core, but GAT replaces the first fully connected layer (or the entire perception module) of the state input end in the original DDPG. GAT here acts as a more powerful state encoder, which encodes the original, structured network topology information (nodes, links and their features) into a fixed-dimensional state vector that can deeply reflect the global and local relationships of the network and is passed to the subsequent layers of the DDPG policy network and value network for decision-making. Therefore, the specific model here is a hybrid model based on GAT-DDPG.

[0063] The feature learning network parameters of the graph model can be the trainable weights of the graph neural network itself. Specifically, they can include: the graph convolution layer weights of the weight matrix used to aggregate neighbor node information, the attention mechanism parameters that determine the importance of different neighbors when aggregating information, and the fully connected layer weights used to further transform the learned node embeddings. Different network topologies (e.g., data center networks, wide area networks) or different traffic patterns require different feature extraction methods. Optimizing these parameters means allowing the model to automatically learn the most suitable feature representation for the current traffic scheduling task.

[0064] The policy network is the "brain" of the reinforcement learning agent. It receives the state, such as the topology features processed by the graph model, real-time link utilization, traffic demand, etc., and then outputs an action, i.e., a scheduling decision, such as routing data packets from path A to path B or assigning traffic to server C. The policy network parameters of the reinforcement learning model can be the trainable weights of the policy neural network itself, which can specifically include the weights and biases of the multi-layer perceptron. The policy network parameters of the reinforcement learning model directly determine how the reinforcement learning model makes decisions based on the current state. Optimizing the policy network parameters of the reinforcement learning model to find an optimal policy that maximizes the long-term cumulative reward. The concatenation of the graph feature network parameters and the policy network parameters forms the iteration particles. Assuming the graph network has 3 parameters and the policy network has 4 parameters, the position of each particle is a (3+4)-dimensional vector.

[0065] where the particle swarm optimization algorithm is represented as: ; ; where v r k+1denotes the velocity vector of particle r at the k+1th iteration, ω denotes the adaptive inertia weight, k denotes the current iteration number, c1, c2 denote acceleration constants that control the influence of individual and social experience respectively, r1, r2 denote random numbers uniformly distributed in the interval [0, 1] and are used to increase the randomness of the search and avoid falling into local optimum, pbest denotes the best position that particle r has reached in history, gbest denotes the global best position that the particle swarm has reached in all iterations, , ρ r k denotes the position of particle r at the kth iteration, which directly encodes the policy network parameters θ of the reinforcement learning model and the feature learning network parameters of the graph model .

[0066] Here, denotes the adaptive inertia weight, c1 = c2 = 1.5, the size of the particle swarm can be 20, and the maximum number of iterations can be 50; by guiding the particle swarm to search efficiently in the joint parameter space ρ r k , find the optimal parameter combination that enables the reinforcement learning model to obtain the highest cumulative reward.

[0067] where the fitness function is represented as: ; Fitness(ρ r ) denotes the fitness value of the parameter ρ r represented by particle r, which represents the optimization objective of the IPSO algorithm, πρ r denotes the flow scheduling strategy defined by the parameter ρ r , (GAT-DDPG generated flow scheduling strategy), denotes a complete interaction trajectory composed of state, action, and reward sequence, denotes a discount factor that gives higher importance to recent rewards, r t denotes the reward value obtained by the reinforcement learning model parameters at time t, where the reward value is calculated according to the reward function.

[0068] The fitness function can be an IPSO function, and the purpose of the IPSO function is to quantitatively evaluate the pros and cons of each set of parameters ρ r , and the goal of IPSO is to maximize this fitness value.

[0069] First, a particle swarm is randomly generated (i.e. multiple sets of random joint parameters), and the position of each particle is randomly initialized (containing the weights of GAT and DDPG). For each particle, use its parameters ρ r kThe GAT-DDPG model is instantiated, run for multiple rounds in a simulation environment or historical data, and calculate its fitness value Fitness (p r ). The current fitness value is compared with the particle's own historical optimum (pbest) and the global historical optimum (gbest), and pbest and gbest are updated. According to the particle swarm optimization algorithm, the information of pbest and gbest is used to update the search direction (velocity) and position (parameter set) of each particle. The fitness value is calculated, pbest and gbest are updated, and the search direction (velocity) and position (parameter set) of each particle are updated, and the particle moves towards the direction of its own historical best position pbest and the global best position gbest of the population. This moving process is a search in the joint parameter space until the maximum number of iterations is reached. Finally, the parameter set p r is set as the parameter of the final GAT-DDPG model for actual online traffic scheduling.

[0070] For example, the particle swarm size can be set to 20 and the maximum number of iterations can be set to 50. The selection of the particle swarm size needs to balance between search efficiency and solution quality, and a too small size may lead to insufficient search, and a too large size will increase the calculation cost; the maximum number of iterations is set according to the complexity of the actual problem and the convergence speed requirement to ensure that the algorithm can converge to a relatively optimized solution within a reasonable time. It can be understood that the specific value of the particle swarm size is not limited to 20, and the specific value of the maximum number of iterations is not limited to 50, and the specific value can be set according to actual needs.

[0071] In the process of updating the particle velocity based on the particle swarm optimization algorithm, the adaptive inertia weight is dynamically adjusted according to factors such as the number of iterations during the algorithm running process, and the initial value can be set to about 0.9, which gradually decreases with the increase of the number of iterations, to ensure the global search ability of the particle in the early stage and enhance the local search ability in the later stage; c1, c2 = 1.5, r1, r2 are uniformly distributed random numbers in the interval [0, 1], which is used to increase the randomness and diversity of particle velocity update and prevent the particle from falling into local optimum.

[0072] The application sets up a collaborative optimization framework integrating GAT, DDPG and IPSO. By deeply integrating the graph attention network and the DDPG algorithm, it can fully understand the complex relationship of the network topology, thereby providing a state representation rich in structural information for decision-making. The parameters of the DDPG policy network and the feature extraction parameters of GAT are jointly encoded as the position of the particle in the improved particle swarm optimization algorithm, so that each particle represents a complete set of GAT-DDPG joint model parameter configuration. During the optimization process, IPSO acts as a meta-optimizer to guide the particle swarm to search iteratively in the entire joint parameter space. Each generation of particles is evaluated by its performance in the network simulation environment (i.e. the fitness function) to guide the population evolution to a better area. Finally, the global optimal particle position output by IPSO is the fully optimized model parameter combination. In this way, through the deep integration of reinforcement learning and meta-heuristic search, the problem of high-quality policy generation and stable training in a complex environment is solved. The above microservice priority weight, global state vector and predicted target traffic are jointly input into the reinforcement learning model, so that it can comprehensively weigh the business fairness, real-time performance and future load in the high-dimensional and continuous policy space to generate an approximately optimal traffic scheduling policy. Through the particle swarm optimization algorithm, the application automatically searches the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model as joint optimization targets. This design effectively overcomes the instability of deep reinforcement learning model training and the difficulty of falling into local optimum, significantly improving the convergence speed and final performance of the policy generation model.

[0073] Step S105: issuing the traffic scheduling policy to the network device and adjusting the traffic of the network device based on the traffic scheduling policy.

[0074] After issuing the traffic scheduling policy to the network device and before adjusting the traffic of the network device based on the traffic scheduling policy, it includes: obtaining actual network performance data after the network device executes the traffic scheduling policy; calculating an actual reward value based on the actual network performance data and the priority weight of the microservice; calculating a deviation value of the actual reward value and the preset reward value, in response to the deviation value being greater than a preset threshold, updating the decision experience of the reinforcement learning model according to the actual network performance data and the actual reward value, and training the reinforcement learning model based on the updated decision experience to adjust the policy network parameters of the reinforcement learning model.

[0075] Please refer to Figure 3, the DDPG final output action is translated into specific OpenFlow flow table items (such as setting queue bandwidth, modifying forwarding path) by the software defined network (SDN) controller. The flow table items are sent to the corresponding network device through the OpenFlow protocol. The controller sends the traffic scheduling strategy to the network device, and collects the actual network average delay after the network device executes the traffic scheduling strategy and the actual average link utilization after the network device executes the traffic scheduling strategy in real time.

[0076] The traceroute command is used on the server to verify whether the actual forwarding path of the packet is consistent with the path specified by the policy to ensure that the scheduling action output by the reinforcement learning model is correctly executed by the network device. The deviation value of the actual reward value from the preset reward value triggers the closed-loop feedback mechanism. If the deviation value is greater than the preset threshold, it means that the prediction of the model is seriously inconsistent with the actual situation, which means that the current scheduling strategy and the actual network status have a large deviation, and cannot meet the expected performance requirements. At this time, the closed-loop feedback mechanism is triggered, and the experience pool is updated. The current network state (topology information processed by GAT, real-time link utilization, device queue length, traffic demand, etc.), scheduling action, actual reward value calculated by measuring network performance data corresponding to the scheduling action of the executed scheduling strategy, and new state of the entire network system after the scheduling strategy corresponding to the executed scheduling action are stored in the experience replay pool.

[0077] The Actor and Critic networks (policy network parameters of the reinforcement learning model) of DDPG are retrained by sampling from the updated experience pool. Until the deviation value is less than or equal to the preset threshold, the encoding indicates that the effect of executing the current traffic scheduling strategy is good. The current policy is maintained, and monitoring is continued.

[0078] Among them, the network performance data includes network average delay, average link utilization, and the actual reward value is calculated based on the actual network performance data and the priority weight of the microservice, including: calculating the actual reward value based on the actual network average delay, the actual average link utilization, the priority weight of the microservice, and the reward value calculation formula.

[0079] The reward value calculation formula is represented as: ; Among them, r t represents the reward value, λ1 represents the delay weight, λ2 represents the utilization weight, λ3 represents the allocation fairness weight, delay t represents the network average delay at time t, the smaller the better, utilization t represents the network average link utilization at time t, the larger the better, (1-utilization t ) represents the penalty term, alloci a proportion of the actual bandwidth allocation for microservice i, w i a priority weight of microservice i, w an absolute error sum of the actual allocation and the theoretical allocation of the microservice, and the smaller the error sum, the more the scheduling strategy conforms to the priority weight corresponding to the microservice.

[0080] The traffic scheduling strategy generated by the reinforcement learning model guided by the reward function can simultaneously meet the requirements of minimizing the time delay and maximizing the utilization, thereby improving the overall performance and resource utilization of the network.

[0081] The precise delivery and closed-loop feedback of the strategy are realized through the software-defined network architecture. The software-defined SDN controller realizes flexible control over network devices based on the OpenFlow protocol, ensuring that the scheduling strategy can be quickly and accurately executed in the actual network. Moreover, the closed-loop feedback mechanism can update the DDPG experience pool and retrain the model in a timely manner according to the verification error, ensuring the continuous optimization and effectiveness of the scheduling strategy, so that the server network can always maintain an efficient and stable operating state when facing changing traffic demands.

[0082] In an embodiment, the overall performance indicators of the network can be monitored in real time, including the percentile of the network average time delay, the number of times the key link utilization rate threshold is exceeded, and the sliding window statistical value of the microservice priority satisfaction rate. If it is monitored that the percentile of the network average time delay continuously exceeds the preset average time delay threshold, the time delay weight is automatically increased, and the time delay is more inclined to be optimized. If it is monitored that the key link utilization rate frequently exceeds the safety threshold (such as 90%), the utilization rate weight is automatically increased, and the load balancing is more concerned. If it is monitored that the resource allocation satisfaction rate of high-priority microservices decreases, the fairness weight is automatically increased, and the principle of scheduling according to priority is strengthened. The time delay weight, utilization rate weight, and allocation fairness weight values are periodically or triggered adjusted, and the adjusted weights are applied to the reward value calculation, thereby guiding the reinforcement learning model to dynamically adapt to the network optimization focus in different periods. In this way, the reward function changes from a static optimization target to a dynamic adaptive optimization. In different stages of network operation, such as congestion avoidance during peak business hours, energy efficiency optimization during daily operation, and service level agreement stage to protect key services, the system can automatically adjust the focus of the optimization direction, thereby having stronger environmental adaptability and long-term effectiveness in complex multi-objective trade-offs.

[0083] Embodiments of the present application also provide an electronic device, such as Figure 4 as shown, comprising a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to perform the steps in any of the above traffic scheduling method embodiments.

[0084] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is arranged to execute the steps in any of the above flow scheduling method embodiments when running.

[0085] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile network device, a magnetic disk or an optical disk, and various media capable of storing a computer program.

[0086] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0087] The above has introduced in detail a flow scheduling method provided by the present application. The principles and implementation manners of the present application are described by applying specific examples in the present text, and the above example description is only used to help understand the method of the present application and its core idea. It should be pointed out that, for the ordinary skilled person in the technical field, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A traffic scheduling method, characterized in that, The traffic scheduling method includes: The priority weight of a microservice is calculated based on microservice information, which includes microservice traffic information, microservice dependencies, and microservice interaction frequency. Predict the target traffic of network devices in future periods based on historical traffic information of network devices in historical periods; A graph model is constructed based on the topology information of network devices and the dependencies of microservices. Embedding learning is performed on the nodes in the graph model to generate state vectors that represent the network state. The priority weights of the microservices, the state vectors, and the target traffic are used as inputs to the reinforcement learning model, which outputs a traffic scheduling strategy for the network device. The reinforcement learning model parameters used to generate the traffic scheduling strategy are determined by iterative search based on the policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model. The traffic scheduling policy is distributed to network devices, and the traffic of the network devices is adjusted based on the traffic scheduling policy.

2. The traffic scheduling method according to claim 1, characterized in that, The calculation of microservice priority weights based on microservice information includes: Obtain microservice traffic information, microservice dependencies, and microservice interaction frequency; The service weight of a microservice is calculated based on the traffic information of the microservice, the dependencies of the microservice, the interaction frequency of the microservice, and the service weight calculation formula. Obtain the total number of microservices, and calculate the priority weight of each microservice based on the total number of microservices, the service weight of each microservice, and the priority weight calculation formula. The service weight calculation formula is as follows: ； Where X1 represents the traffic information weight of the microservice, X2 represents the dependency weight of the microservice, X3 represents the interaction frequency weight of the microservice, T represents the preset time window length, and V i (t) represents the traffic information of microservice i collected at time t, D i Y represents the number of microservices that depend on microservice i, C represents the set of interacting services, which is the set of other microservices that interact with microservice i, and Y represents the number of microservices that depend on microservice i. （i,k） This represents the interaction frequency between microservice i and microservice k within a preset time window T; The priority weight calculation formula is as follows: ； Among them, w i This indicates the priority weight of microservice i. This represents the sum of the service weights of the microservices, where M is the total number of microservices.

3. The traffic scheduling method according to claim 1, characterized in that, The method of predicting the target traffic of network devices in future periods based on historical traffic information of network devices in historical periods includes: A short-term network traffic prediction model is constructed by inputting the historical traffic information of network devices within the historical period into the short-term network traffic prediction model, so as to output the target traffic of network devices in the future period through the short-term network traffic prediction model. The short-term network traffic prediction model is trained based on a loss function, which is shown below: ； Among them, L MSE The loss value is used to train a short-term network traffic prediction model, where N represents the number of training samples and V(t) represents the actual traffic value at time t. This represents the predicted flow rate of the model at time t.

4. The traffic scheduling method according to claim 1, characterized in that, The step of constructing a graph model based on the topology information of network devices and the dependencies of microservices, and performing embedding learning on the nodes in the graph model to generate state vectors representing the network state includes: The network devices and microservices in the system are abstracted as nodes in the graph model, and the topology information of the network devices and the dependencies of the microservices are abstracted as edges in the graph model to construct the graph model. The initial feature vector of the node is used as the input of the graph model, and the initial feature vectors of the node and its neighboring nodes are aggregated through the graph model. By using an attention mechanism to assign different aggregation weights to different neighbor nodes, and then generating the final embedded representation of the node through multi-layer propagation using a feature propagation formula; The final embedded representation of the node is pooled to generate a state vector representing the network state. The state vector representing the network state is expressed as: ； s t The state vector represents the network state, and Readout() is the graph readout function used to aggregate the node embeddings into a global graph representation. i (L) h represents the final embedded representation of node i after processing at the Lth layer of the graph model. i (L) The initial feature vectors of node i and all its neighboring nodes are included, and C represents the set of nodes; The feature propagation formula is expressed as follows: ； h i (l+1) The embedding representation of node i at layer l+1 of the graph model is obtained by aggregating the embedding representations of node i and its neighboring nodes j at layer l of the graph model. σ represents a nonlinear function, and α represents the embedding representation of node i at layer l. ij W represents the attention weight, which measures the importance of neighbor node j to node i. （l） Let h represent the trainable weight matrix. j (l) This represents the embedding representation of node i's neighbor node j in the l-th layer of the graph model.

5. The traffic scheduling method according to claim 4, characterized in that, The attention weights are represented as follows: ； Where exp() represents the exponential function and LeakfyReLU() represents the activation function. Let n(i) represent the attention vector, and n(i) represent the set of neighboring nodes j of node i. Let W(l) be the transpose of the attention vector, and let h be the trainable weight matrix. i (l) h represents the embedding representation of node i in the l-th layer of the graph model. j (l) This represents the embedding representation of node i's neighbor node j in the l-th layer of the graph model.

6. The traffic scheduling method according to claim 1, characterized in that, The step of using the priority weights of the microservices, the state vector, and the target traffic as input to the reinforcement learning model, and outputting the network device's traffic scheduling strategy through the reinforcement learning model, includes: The state vector is fused with the target flow to form an enhanced state representation; The enhanced state representation is input into the policy network of the reinforcement learning model to obtain the initial scheduling action; The initial scheduling action is weighted and modified using the priority weights of the microservices to output the final traffic scheduling strategy, which is expressed as a bandwidth allocation strategy or a data flow forwarding path strategy for network devices.

7. The traffic scheduling method according to claim 1, characterized in that, The method further includes: generating reinforcement learning model parameters for a traffic scheduling strategy, wherein the reinforcement learning model parameters for generating the traffic scheduling strategy include: The policy network parameters of the reinforcement learning model and the feature learning network parameters of the graph model are encoded to form iterative particles; The particle swarm optimization algorithm and fitness function guide the particle swarm to iteratively search in the joint parameter space of policy network parameters and feature learning network parameters to find the optimal reinforcement learning model parameters. The particle swarm optimization algorithm is expressed as: ； ； Among them, v r k+1 Let ω represent the velocity vector of particle r at the (k+1)th iteration, ω represent the adaptive inertia weight, k represent the current iteration number, c1 and c2 represent acceleration constants, controlling the influence of individual and social experience respectively, r1 and r2 represent uniformly distributed random numbers in the interval [0,1], used to increase the randomness of the search and avoid getting trapped in local optima, pbest represents the best position that particle r has reached historically, and gbest represents the global best position that the particle swarm has reached in all iterations. , ρ r k This represents the position of particle r in the k-th iteration, which directly encodes the policy network parameters θ of the reinforcement learning model and the feature learning network parameters of the graphical model. ; The fitness function is expressed as follows: ； Among them, Fitness (ρ r The particle r represents the parameter ρ. r The fitness value represents the optimization objective of the IPSO algorithm, πρ r Indicated by parameter ρ r Defined traffic scheduling strategy, A complete interaction trajectory is represented by a sequence of states, actions, and rewards. This represents a discount factor, giving greater importance to recent rewards. t This represents the reward value obtained by the reinforcement learning model parameters at time t.

8. The traffic scheduling method according to claim 1, characterized in that, After the traffic scheduling policy is distributed to the network device, but before adjusting the traffic of the network device based on the traffic scheduling policy, the following steps are included: Obtain actual network performance data after the network device executes the traffic scheduling strategy; The actual reward value is calculated based on the actual network performance data and the priority weight of the microservice. The deviation between the actual reward value and the preset reward value is calculated. In response to the deviation value being greater than the preset threshold, the decision experience of the reinforcement learning model is updated based on the actual network performance data and the actual reward value. The reinforcement learning model is then trained based on the updated decision experience to adjust the policy network parameters of the reinforcement learning model.

9. The traffic scheduling method according to claim 8, characterized in that, Network performance data includes average network latency and average link utilization. The calculation of the actual reward value based on the actual network performance data and the priority weights of the microservices includes: The actual reward value is calculated based on the actual average network latency, actual average link utilization, microservice priority weights, and the reward value calculation formula. The formula for calculating the reward value is expressed as follows: ； Where, r t λ1 represents the reward value, λ2 represents the delay weight, λ3 represents the utilization weight, and λ4 represents the allocation fairness weight. t The average network latency at time t is represented by the utilization. t This represents the average network link utilization at time t, (1-utilization) t ) represents the penalty term, alloc i w represents the actual bandwidth allocation ratio for microservice i. i This indicates the priority weight of microservice i. This represents the sum of the absolute errors between the actual allocation and the theoretical allocation of microservices.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the traffic scheduling method as described in any one of claims 1 to 9 when executing the computer program.

Images