Intelligent traffic orchestration method based on fusion of expert network and deep reinforcement learning

Abstract

The application discloses an intelligent traffic arrangement method based on a fusion of an expert network and deep reinforcement learning, comprising the following steps: collecting network node and link state data in real time, and constructing a time sequence input vector and a topological graph structure; using a time sequence neural network and a graph neural network to respectively extract traffic space-time features and node topological features, fusing the features, predicting future traffic through a classification network, and completing coarse-grained arrangement to different service level network slices; modeling resource scheduling as a multi-agent Markov decision process, designing a state space, an action space and a reward function; initializing a deep reinforcement learning agent, and training the agent through interactive experience; fusing a pre-trained expert strategy network, constructing a total loss function to optimize network parameters; and finally generating an intelligent strategy capable of dynamically optimizing traffic paths and resource allocation according to real-time states; and the application can realize efficient resource scheduling under differentiated service quality requirements of multiple services.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network traffic arrangement, and particularly relates to an intelligent traffic arrangement method based on a fusion of an expert network and deep reinforcement learning. BACKGROUND

[0002] With the continuous expansion of the power network system scale, the number of physical devices connected in the system rapidly increases, resulting in a sharp increase in the amount of data that needs to be transmitted, collected, stored and shared, and the problem of data transmission delay is increasingly prominent. In a resource-limited network environment, if there is a lack of effective traffic scheduling and resource management mechanism, the network service quality is difficult to meet the real-time, reliability and diversified business requirements of the power grid system. In the face of differentiated service requirements of low delay, high bandwidth and high reliability, the traditional static resource allocation and routing strategy has gradually become inadequate, and it is urgent to introduce a more intelligent and dynamic network management mechanism to adapt to the complex and changing operating environment.

[0003] To address the above challenges, network slicing technology is widely used in the prior art. As a core component of 5G and next-generation communication networks, this technology can divide the underlying physical network into multiple logically isolated networks according to business requirements, to respectively carry different types of services with different quality of service (QoS) requirements. Typical implementation methods include slice division and traffic scheduling mechanisms based on artificial rules or static policies, such as pre-defined bandwidth reservation strategies, fixed routing tables or threshold-based heuristic algorithms, to preliminarily divide and assign network resources. This kind of method to a certain extent realizes the isolation of multiple services and the differentiated service of resources.

[0004] However, the existing methods have a significant defect: the slice identification and traffic scheduling mechanism lacks dynamic perception and autonomous optimization capabilities. Due to the reliance on artificial preset rules or static configuration, it is difficult to respond to network state changes (such as node load fluctuations, link quality changes or burst traffic patterns) in real time, resulting in low resource utilization efficiency, unstable delay performance, and inability to adapt to the increasing requirements of real-time and reliability of power grid services. SUMMARY

[0005] The purpose of the present application is to provide an intelligent traffic arrangement method based on the fusion of an expert network and deep reinforcement learning, which can realize real-time perception of network device state and meet the differentiated service requirements of low delay, high bandwidth and high reliability.

[0006] Technical solution: The intelligent traffic arrangement method according to the present application comprises the following steps:

[0007] (1) Network state perception and input construction: Collect multi-dimensional real-time running state parameters of all nodes and links in the network, and construct a time series input vector for representing the spatio-temporal distribution of business traffic and a topological graph structure for representing the network connection relationship based on the collected data.

[0008] (2) Multi-dimensional feature extraction and fusion classification: Use a time series neural network model to process the time series input vector to extract the spatio-temporal features of the traffic, and use a graph neural network model to process the topological graph structure to extract the topological correlation features of the nodes; fuse the spatio-temporal features and topological correlation features, and process the fused features through a non-linear classification network to predict future traffic and complete the coarse-grained arrangement of dividing it into different service quality level logical network slices.

[0009] (3) Reinforcement learning environment modeling: Model the network resource scheduling problem as a multi-agent Markov decision process, define each resource node as an agent, and design the state space, action space and reward function for each agent; the state space includes node resource state, historical allocation record and capacity constraint, the action space includes adjustment instructions for various slice resources, and the reward function is used to quantitatively evaluate the comprehensive performance of the action.

[0010] (4) Agent initialization and interaction: Initialize the decision network, target network and experience storage unit of the deep reinforcement learning agent; the agent perceives the current state, outputs the value evaluation through the decision network and selects the resource adjustment action, executes the resource adjustment action to update the network resource allocation, then the environment feedbacks a reward signal, and stores the interaction experience corresponding to the agent's action and the environment's feedback in the experience storage unit.

[0011] (5) Expert knowledge fusion and network training: Pre-train an expert policy network with historical optimal decision data to simulate expert behavior; during the reinforcement learning training process, sample data from the experience storage unit to calculate the loss of the decision network, and calculate the difference loss between the decision network strategy and the output of the expert policy network; combine the loss of the decision network and the difference loss to construct a total loss function, and update the parameters of the decision network through back propagation, and synchronize the parameters of the target network regularly.

[0012] (6) Optimal strategy generation: Repeat the interaction and training process until the strategy converges, and finally output an intelligent model that can generate optimal traffic path and resource allocation strategy according to real-time network state.

[0013] The application realizes accurate perception of the space-time distribution of service traffic and the network connection relationship by constructing multi-dimensional real-time state input and network topology representation; with the joint feature extraction and fusion classification of time sequence and graph neural network, the accuracy and adaptability of traffic prediction and slice arrangement are effectively improved; through multi-agent reinforcement learning environment modeling, the complex resource scheduling problem is converted into a learnable decision-making process; relying on the agent interaction and experience storage mechanism, continuous decision-making and experience accumulation in a dynamic environment are realized; combined with expert knowledge fusion and hybrid loss training, the stability and generalization ability of strategy learning are significantly enhanced; finally, through strategy iteration optimization, an intelligent decision-making model that can adapt to real-time network state, efficiently coordinate resource allocation and traffic path is generated, and the network resource utilization efficiency and service quality are comprehensively improved.

[0014] Preferably, the collection of multi-dimensional real-time running state parameters of all nodes and links in the network in step 1 comprises:

[0015] Define the network node set as : Each node contains the state information of the network node: computing power , load , cache queue length , bandwidth utilization , energy consumption factor , reliability factor , task waiting time average , task priority density , cache data discard rate ; therefore, the node feature vector is represented as:

[0016] ;

[0017] Define the link set as , if node and node are connected, then define the connection characteristics as:

[0018]

[0019] wherein, is the link bandwidth, is the link delay, is the link jitter, is the packet loss rate, is the link stability index, is the hop distance factor, i.e. the path hop number from the source node to the connection.

[0020] By defining the multi-dimensional refined feature vectors of nodes and links, the application realizes comprehensive quantitative perception of the running state of network infrastructure: node features cover computing, load, cache, bandwidth, energy consumption, reliability and task-level indicators, link features integrate bandwidth, delay, jitter, packet loss, stability and path topology information, and together build a high-granularity, multi-dimensional network dynamic representation system, providing rich, accurate and interpretable environment state input for subsequent intelligent decision-making.

[0021] Preferably, the input vector and the construction of the topological graph structure in step 1 include:

[0022] The target area is divided into a grid, and three types of traffic, enhanced mobile broadband, ultra-reliable low-latency communication and large-scale machine communication, are collected respectively, a total of three channels, and the traffic of a type of service in each grid is counted every hour:

[0023]

[0024] The three types of heat maps are spliced into:

[0025]

[0026] A historical time sequence is input to the improved convolutional long short-term memory network:

[0027]

[0028] The shape of the input tensor is , wherein ;

[0029] To construct the input graph neural network required, the adjacency matrix is defined as:

[0030]

[0031] Wherein, .

[0032] The node feature matrix is defined as :

[0033] .

[0034] By gridding the target area and counting the traffic data of three typical services respectively, a multi-channel space-time heat map is constructed to achieve a fine description of the space-time distribution of traffic flow. Then, the historical time-series heat map sequence is converted into a tensor input, providing a high-dimensional feature representation with both spatial pattern and temporal dynamics for the convolutional long short-term memory network. At the same time, based on the physical connection relationship and multi-dimensional state characteristics of nodes, an adjacency matrix and a node feature matrix are constructed to form a graph structure input that preserves the topological association relationship, laying a solid foundation for the graph neural network to efficiently extract deep dependency relationships between nodes and comprehensively improving the model's perception of complex network states and business patterns.

[0035] Preferably, the step 2 of processing the time-series input vector by using a time-series neural network model to extract the space-time characteristics of traffic flow comprises:

[0036] The historical traffic heat map sequence is processed by using a convolutional long short-term memory network to extract space-time characteristics; the convolutional long short-term memory network unit comprises an input gate , a forget gate , an output gate , a memory cell and a hidden state ; the input tensor of the convolutional long short-term memory network is the historical heat map sequence , wherein represents the traffic heat map at time ; the input gate , the forget gate and the output gate are calculated respectively by using the input at each time step , the current input and the previous state are fused, and and are updated;

[0037] A dynamic gating mechanism is introduced based on the convolutional long short-term memory network, and a time-series attention mechanism is introduced in the input gate to focus on key historical states; the specific calculation formula is as follows:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] wherein, is the hidden state at time , and is the hidden state at time the cell state, is a convolution operation, and is a Hadamard product, is an activation function, is a hyperbolic tangent activation function, is a forget gate bias term, is an output gate bias term, is a bias term for cell state update, is a linear transformation matrix, which determines the influence of input, historical hidden state and memory, and is a learnable weight parameter together with the bias term, which is constantly updated during the training process through back propagation, and is used to calculate the gating value, state update and attention weight;

[0044] the final output is used as the extracted feature to predict the heat map of the next moment;

[0045] At the same time of inputting X, the topology modeling is input into the graph attention model to obtain the final graph embedding.

[0046] By fusing the convolutional long short-term memory network and the dynamic gating mechanism, and introducing the time sequence attention factor and the heat map change rate as enhanced input, the design can accurately capture the complex spatio-temporal dependence relationship contained in the traffic heat map; the time sequence attention mechanism can significantly improve the perception ability of important time nodes by adaptively weighting the key historical states, and the gating structure and the dynamic update of the cell state can effectively coordinate the fusion and forgetting process of the current input and historical information; the high-level features of the final output not only retain the spatio-temporal evolution law of the traffic, but also provide a discriminative representation for the subsequent deep fusion with the topology features, and comprehensively enhance the prediction and understanding ability of the model for the dynamic business traffic.

[0047] Preferably, in step 2, the graph neural network model is used to process the topological graph structure to extract the topological correlation features of the nodes, and a graph attention model is adopted, and the calculation process includes:

[0048] Initialization: input the graph adjacency matrix A and the optimal feature set into the graph attention model;

[0049] Calculate the attention coefficient: for each node , calculate the attention coefficient of the node and its neighbor node , the attention coefficient measures the importance of to the node ; the mutual influence between neighbors is introduced as the second-order attention information, and the final calculation attention coefficient is obtained by adaptively fusing the first-order and second-order attention coefficients, and the calculation formula is as follows:

[0050]

[0051]

[0052]

[0053] wherein, is a first-order attention coefficient, is a second-order attention coefficient, is a learnable weight matrix for mapping the node feature vector to a new feature space with dimension , is another learnable weight vector, denotes a concatenation operation, is a weight, is an input feature vector of a node and , is a link feature between and , LeakyReLU is an activation function defined as ;

[0054] Aggregating neighbor features: according to the attention weight , the features of the neighbor nodes are aggregated to obtain a new feature vector of the node , and the calculation formula is:

[0055]

[0056] wherein, is an activation function; in the multi-head attention mechanism, i.e., multiple single-head attention calculations are performed in parallel to obtain multiple new feature vectors , K is the number of heads; then the feature vectors are concatenated to obtain the final node feature vector .

[0057] By introducing an adaptive graph attention mechanism that fuses first-order and second-order attention coefficients, the model can accurately quantify the complex topological dependence and mutual influence between nodes; by using link features to enhance neighbor importance calculation and combining a multi-head parallel aggregation strategy, the discriminability and robustness of node feature expression are significantly improved, and the finally generated node embedding effectively preserves the global topological structure information and local correlation patterns of the network, providing high-precision topological correlation representation for subsequent cross-modal feature fusion and collaborative decision-making.

[0058] Preferably, the fusion classification of step 2 comprises:

[0059] The extracted feature vectors are concatenated:

[0060]

[0061] F is input into the Kolmogorov-Arnold Networks, and the predicted traffic matrix is output.

[0062] The shape of KAN is represented by an integer array:

[0063]

[0064] wherein is the number of nodes in the layer of the computational graph, and denotes the activation value of the neuron in the layer, and the calculation formula of the activation value is as follows:

[0065]

[0066]

[0067]

[0068]

[0069] wherein, is the activation value of the neuron in the layer, is the activation value of the neuron in the previous layer, is the activation function connecting the neuron in the layer and the neuron in the layer, which is a trainable function, is a mapping function that maps the input , i.e., the output of the neuron in the previous layer, to a certain value, is the activation part weight, is a linear combination of a plurality of B-spline basis functions is the weight of the B-spline basis function combination, is the coefficient corresponding to the basis function, are all trainable, is the function matrix corresponding to the layer of KAN, and when KAN is L layers, the output of the final KAN is:

[0070] ​​

[0071] wherein, The representative function composition is that the input feature F is mapped through multiple layers of function matrices, and high-order abstract representations are obtained layer by layer, and finally a traffic prediction of different network slice types of the network node at a time step is obtained.

[0072] By deeply fusing the space-time features and the topology correlation features, and using the Kolmogorov-Arnold Networks based on the trainable function matrix for nonlinear transformation, the model breaks through the limitation of the traditional linear combination, and realizes the accurate extraction and interaction of high-order abstract features; by using the B-spline basis function to flexibly fit the complex mapping relationship, the expression ability and the generalization performance of the traffic prediction are significantly improved, and finally a high-precision prediction result accurately reflecting the traffic distribution of the network node at different slice types at a future time is output, which provides reliable data support for subsequent resource arrangement.

[0073] Preferably, the reinforcement learning environment modeling in step 3 comprises:

[0074] The resource scheduling problem under the multi-resource node is modeled as a partially observable distributed multi-agent Markov decision process, and is defined by the following seven-tuple:

[0075]

[0076] wherein, is the agent set, and corresponds to the resource node in the system, each node is an agent, is the global environment state space, is the action space of the agent , is the state transition probability function, is the reward of the agent , is the discount factor, used to measure the weight of the future reward;

[0077] For each resource node , the state space is designed as:

[0078]

[0079] wherein, is the resource requested by the slice m at t-1, is the resource allocated to m by the node , is the resource adjustment action history, including increase, decrease and maintenance, is the node resource capacity, that is, the constraint on the node ;

[0080] Each resource agent selects the direction for adjusting resources across three business slices, including increasing, decreasing, and maintaining, i.e.:

[0081]

[0082] Through mapping function Map the action number to:

[0083] .

[0084] By modeling the distributed resource scheduling problem as a partially observable multi-agent Markov decision process, this framework achieves accurate abstraction and distributed decision support for complex network environments. By designing a multi-dimensional state space for each resource node agent, including resource requests, allocation records, action history, and capacity constraints, as well as a discrete adjustment action space oriented towards multiple service slices, it effectively characterizes the local observation and decision-making capabilities of the agents, provides a learnable and scalable mathematical foundation for collaborative resource optimization, and lays the theoretical foundation for generating efficient distributed resource scheduling strategies.

[0085] Preferably, the agent initialization and interaction in step 4 includes:

[0086] Initialization: Initialize a main network for each agent. and a target network The main network The target network is used to calculate the action Q-value based on the current state and select an action. This is used to evaluate the long-term benefits of the action, providing a stable training signal; simultaneously, an experience replay pool is initialized. Used to store the agent's interaction experience data and initialize parameters. ;

[0087] The intelligent agent targets each resource node resource types on Get its state The Q-value of each optional action is calculated through forward propagation, using the following formula:

[0088]

[0089] in, It refers to the current state value. This refers to extracting the advantage of each action. This function value is output by the neural network and does not directly depend on rewards or the environment. Instead, it is learned through backpropagation during training. Indicates the action of selection. This represents all actions in the action space, used to normalize the dominance function. Represent the advantage of all actions to average;

[0090] The main network adopts Strategy to select action: select action with probability Randomly, select action with probability Select the action with the maximum current estimated Q value, and the action space is , use the mapping function Convert the action to the slice resource adjustment amount, and directly select the maximum Q value action during deployment;

[0091] According to the action decision , update the resource allocation, the formula is as follows:

[0092]

[0093] Among them Indicates the updated estimated value, indicates the estimated resource share of slice Allocated to service type at time step , refers to slice eMBB, URLLC, mMTC; The number of slices is , refers to the mapping of action ;

[0094] Each agent aims to meet the service quality requirements of the business, minimize resource waste and congestion, and designs the following rewards. At time step , the immediate reward of node is:

[0095]

[0096] Among them, is the different reward weight, is the estimated resource allocated to slice m, is the demand of slice m, is the service delay of the th slice, is the unsatisfied resource request of slice , is a small value to avoid division by zero error, E is the unit resource brings the unit benefit, N is the total number of nodes, used for resource fairness index, is the total amount of resources obtained by slice m; Save the interaction experience to the experience pool:

[0097]

[0098]

[0099] Among them, ​​For resource nodes The environmental conditions, For nodes The space of motion For nodes Instant rewards For resource nodes The new environmental state.

[0100] By initializing the main network and target network for each agent and constructing an experience replay pool, this mechanism achieves synergy between distributed decision-making and stable training. Utilizing a Q-value calculation method based on state value and action advantage decomposition, it effectively improves the accuracy of action evaluation and the efficiency of policy learning. By balancing exploration and utilization through an ε-greedy policy and combining it with a resource adjustment mapping function to transform discrete actions into actual resource allocation amounts, it ensures the feasibility and practicality of the policy. A multi-objective reward function integrating resource utilization, service latency, demand satisfaction, and fairness is designed to guide agents to autonomously learn the optimal scheduling policy with the goal of minimizing resource waste and congestion. Finally, experience storage provides high-quality data support for subsequent training.

[0101] Preferably, the expert knowledge fusion and network training described in step 5 include:

[0102] A training dataset is constructed using historical best decision data, which includes historical states and corresponding expert decision actions; an expert policy network is trained using cross-entropy loss.

[0103]

[0104] in, It refers to the environmental conditions observed in history. Actions chosen by experts throughout history It is an expert policy network to be trained, and the value is the policy network in the state. Down Output Action The probability of;

[0105] B experience pairs are randomly sampled from the experience pool D to update the Q-value of the main network. The target Q-value is calculated for each experience pair using the following formula:

[0106]

[0107] in, In the state Execute action Instant rewards received For all possible actions in the future The Q-value vector; It involves selecting the action with the largest Q value from the Q-value vector predicted by the main network. After choosing the action for the main network, the target network is used to calculate the Q value of the action The Q value of the action is calculated The Q value of the action is calculated The Q value of the action is calculated

[0108] The loss function calculation formula is as follows:

[0109]

[0110] The formula is used to measure the difference between the current Q network output and the target value , wherein is the batch size sampled from the experience pool; is the target Q value, is the mean square error, used to make the predicted Q value of the main network as close as possible to the target Q value, and minimized through back propagation;

[0111] The expert guidance loss is added to make the Q network strategy close to the expert strategy , and the KL divergence is used:

[0112]

[0113] , wherein represents all actions in the action space, represents the state of node at time step , is the probability of the expert network choosing action in state , is the probability of the Q network derived strategy choosing action in state , represents the difference between the Q network strategy distribution and the expert distribution in state , is the average value of the KL divergence between the expert strategy and the Q strategy calculated by randomly sampling B experiences from the experience pool ;

[0114] The total loss function is finally obtained as follows:

[0115]

[0116]

[0117] , wherein and are constants, is a weight factor, which is a dynamic weight that changes over time It decreases as it increases, being large in the early stage and gradually decreasing in the later stage.

[0118] By integrating historical expert decision-making data with online interaction experience, this training mechanism effectively combines expert prior knowledge with reinforcement learning exploration capabilities. It utilizes cross-entropy loss to pre-train the expert policy network to simulate optimal decision-making behavior, and constructs expert-guided loss using KL divergence, ensuring that the agent's policy closely aligns with expert experience in the early stages of training, significantly improving learning efficiency and stability. Simultaneously, the calculation of the target Q-value and mean squared error loss guarantee the accuracy of the value function convergence, while the dynamically weighted strategy adaptively balances the contributions of imitation learning and autonomous exploration during training, ultimately achieving continuous optimization and reliable improvement of policy performance.

[0119] Preferably, the optimal strategy generation in step 6 includes:

[0120] By repeatedly executing the interaction process of step (4) and the training process of step (5), the agent's policy is continuously optimized until the preset convergence condition is met; after the policy converges, the parameters of the main Q network and the expert policy network are solidified to obtain the trained intelligent model.

[0121] During the application deployment phase, given the current observations... Weight Q-network computing Expert network computing Select an action:

[0122]

[0123] From numerous actions Obtained from This refers to traffic orchestration strategies.

[0124] Through iterative execution of interactive exploration and hybrid training processes, the agent's strategy is continuously optimized and eventually converges to a stable state. The solidified intelligent model after strategy convergence deeply integrates the adaptive decision-making ability of reinforcement learning with the reliability of expert knowledge. During the deployment phase, it can dynamically generate traffic orchestration actions that take into account both short-term benefits and long-term optimality by collaboratively utilizing the value assessment of the Q network and the policy suggestions of the expert network based on real-time observation status, thereby achieving efficient, accurate and autonomous scheduling of network resources.

[0125] Beneficial effects: compared with the prior art, the present application has the following remarkable advantages: 1. Through multi-dimensional real-time perception of network running state and accurate prediction of traffic space-time law, intelligent allocation and scheduling of network resources are realized, effectively meeting the differentiated service quality requirements of high transmission rate, low delay and high reliability of different service types such as enhanced mobile broadband, ultra-reliable low-latency communication and large-scale machine communication; 2. Multi-agent collaborative decision is adopted, with comprehensive performance as the optimization target, automatic learning of optimal resource allocation strategy, significant reduction of resource waste and congestion, and improvement of global resource utilization; 3. Historical expert experience and online environment interaction are fused, model convergence is accelerated through hybrid training mechanism, and the decision-making process has both expert reliability and environmental adaptability; 4. The final model can automatically generate and continuously optimize traffic path and resource allocation strategy according to real-time network state, with strong adaptive ability to cope with network fluctuations and service changes. BRIEF DESCRIPTION OF DRAWINGS

[0126] Figure 1 is a flowchart of the method of the present application;

[0127] Figure 2 is an intelligent traffic arrangement architecture diagram of the present application;

[0128] Figure 3 is a coarse-grained traffic arrangement flowchart of multi-dimensional data feature fusion of the present application;

[0129] Figure 4 is a deep reinforcement learning modeling flowchart of the present application;

[0130] Figure 5 is a D3QN algorithm traffic arrangement flowchart of the present application fusing expert networks. DETAILED DESCRIPTION

[0131] The technical solutions of the present application will be further described below in combination with the drawings.

[0132] The embodiment of the present application provides an intelligent traffic arrangement method based on fusion of expert networks and deep reinforcement learning, Figure 1 is a flowchart of the method, Figure 2For the intelligent traffic arrangement architecture of the method, the dynamic traffic arrangement optimization problem is proposed by the application, which considers the slice average efficiency and utility measurement. In addition, the application proposes an intelligent traffic arrangement framework based on a deep reinforcement learning of network slicing and fusion expert network. In the method, each resource configuration unit is managed by two controllers, and coarse-grained and fine-grained arrangement is combined in the traffic arrangement process. Convolutional neural network is used to extract and graph attention network to extract different dimensions of network node features. Through KAN network, the fusion features are classified by network slicing, and the network nodes are divided into three logical slices of large bandwidth, high reliability and low delay, and mass access. On this basis, deep reinforcement learning is introduced to combine expert network in the training process of slice path scheduling, resource selection and traffic scheduling. A large amount of historical data is used to find out the optimal decision to form expert trajectory. Expert network is used to optimize and correct the reinforcement learning result, and finally the intelligent traffic path arrangement strategy is output. The method comprises the following steps:

[0133] Step 1, coarse-grained traffic arrangement based on multi-dimensional data feature fusion.

[0134] Specifically, assuming that there are N network nodes in the network, the convolutional long short-term memory network and the graph attention model are improved, and the two models are used to extract the information of each network node and the features of the traffic flowing through the network node, respectively. The network slice type of the traffic flowing through the network node in the future time step t is predicted, and the coarse-grained arrangement of traffic arrangement is divided into three types of enhanced mobile broadband, ultra-reliable low latency communication and large-scale machine communication.

[0135] Figure 3 The flow chart of coarse-grained traffic arrangement based on multi-dimensional data feature fusion comprises:

[0136] Step 1.1, network state information acquisition.

[0137] Specifically, let the network node set be :

[0138] (1)

[0139] Each node contains the state information of the network node: the computing power , the load , the cache queue length , the bandwidth utilization , the energy consumption factor , the reliability factor , the task waiting time average , the task priority density , and the cache data discard rate . Therefore, the node feature vector is represented as:

[0140] (2)

[0141] Let the set of links be , if node and node are connected, then define the edge feature as:

[0142] (3)

[0143] where is the link bandwidth, is the link delay, is the link jitter, is the packet loss rate, is the link stability index, is the hop distance factor, i.e., the number of hops from the source node to the edge.

[0144] Step 1.2, input vector construction.

[0145] Specifically, the target area is divided into grid, and the traffic of three types of services (enhanced mobile broadband, ultra-reliable low-latency communication, and large-scale machine communication) is collected respectively, a total of three channels. The traffic of a certain type of service in each grid is counted every hour, and the following is obtained:

[0146] (4)

[0147] The three types of heat maps are spliced as follows:

[0148] (5)

[0149] The input of the improved convolutional long short-term memory network is a historical time series:

[0150] (6)

[0151] The shape of the input tensor is , where .

[0152] In order to construct the graph required by the input graph neural network, the adjacency matrix is defined as:

[0153] (7)

[0154] where .

[0155] The node feature matrix is defined as :

[0156] (8)

[0157] Step 1.3: Multidimensional feature extraction.

[0158] Specifically, a Convolutional Long Short-Term Memory (LSTM) network is a deep learning model that combines the mechanisms of convolutional neural networks and long short-term memory networks to process temporal spatial data. Each unit contains: an input gate. Forgotten Gate Output gate Memory cells and hidden state The input tensor of the convolutional long short-term memory network is a sequence of historical heatmaps. ,in Indicates at time A flow heatmap. Using the input for each time step... Calculate the input gates separately Forgotten Gate Output gate Combine the current input with the previous state, and then update. and By introducing a dynamic gating mechanism on top of the convolutional long short-term memory network, the nonlinear modeling capability is enhanced. An attention mechanism is introduced into the input gate to determine which historical states are more important to the model. The specific formula is as follows:

[0159] (9)

[0160] (10)

[0161] (11)

[0162] (12)

[0163] (13)

[0164] in, For a moment The hidden state, For a moment The cell state, ⊙ represents the convolution operation, and ⊙ represents the Hadamard product (element-wise multiplication). For activation function, The hyperbolic tangent activation function is used. For the forget gate bias term, For the output bias terms, This is a bias term for cell state updates. For linear transformation matrix (learnable parameter matrix), the influence of input, historical hidden state and memory is determined, which are all learnable weight parameters together with bias term, which are constantly updated through back propagation during training process, used to calculate gating value, state update, attention weight.

[0165] Final output Can be used as extracted features to predict the heat map of the next moment.

[0166] Model the topology at the same time as input X Input into the graph attention model to get the final graph embedding. The calculation of the graph attention model is divided into three steps:

[0167] (1) Initialization: First, input the graph adjacency matrix A and the optimal feature set Into the graph attention model.

[0168] (2) Calculate attention coefficient: for each node , calculate its attention coefficient with neighbor nodes , the attention coefficient measures the importance of to node . Introduce the mutual influence between neighbors as second-order attention information, and then obtain the final calculation attention coefficient by adaptively fusing first-order and second-order attention coefficients. The calculation formula is as follows:

[0169] (14)

[0170] (15)

[0171] (16)

[0172] Among them, is the first-order attention coefficient, is the second-order attention coefficient, is a learnable weight matrix used to map the node feature vector to a new feature space (dimension ), is another learnable weight vector, represents the concatenation operation, is the weight, is the input feature vector of node and , is the link feature between and , LeakyReLU is an activation function, usually defined as .

[0173] (3) Aggregated neighbor features: based on attention weights The features of the neighbor nodes are aggregated to obtain a new feature vector of the node . The calculation formula is:

[0174] (17)

[0175] wherein is an activation function. If a multi-head attention mechanism is used, i.e., multiple single-head attention calculations are performed in parallel to obtain multiple new feature vectors , K is the number of heads. Then the feature vectors are spliced to obtain the final node feature vector .

[0176] Step 1.4, multi-dimensional feature fusion and network slicing classification.

[0177] Specifically, the extracted feature vectors are spliced:

[0178] (18)

[0179] F is input into a Kolmogorov-Arnold Network (KAN) to output a predicted traffic matrix.

[0180] The shape of the KAN is represented by an integer array:

[0181] (19)

[0182] wherein is the number of nodes in the layer of the computation graph. The activation value of the th neuron in the layer, and the calculation formula of the activation value is as follows:

[0183] (20)

[0184] wherein is the activation value of the th neuron in the layer, is the activation value of the th neuron in the previous layer. is an activation function connecting the th neuron in the layer and the th neuron in the layer, and is a trainable function, which is different from the fixed activation function of a traditional DNN.

[0185] (21)

[0186] (22)

[0187] (23)

[0188] where is a mapping function that maps input (previous layer of neuron output) to a certain value, is the activation part of the weight, is a linear combination of multiple B-spline basis functions is the weight of the B-spline basis function combination, is the coefficient of the corresponding basis function, are all trainable, is the function matrix corresponding to the KAN layer, when KAN is L layers, the output of the final KAN is: (24)

[0189] (25)

[0190] where, represents the function composition, the input feature F is mapped through multiple layers of function matrix, and the high-order abstract representation is obtained layer by layer, and finally the traffic prediction of different network slice types flowing through the network nodes in the future time step.

[0191] Step 2, deep reinforcement learning modeling.

[0192] Specifically, a multi-agent reinforcement learning system is designed to realize intelligent resource arrangement, each resource node as an agent, selects actions according to state information, and designs state space and action space in detail, providing a basis for subsequent deep reinforcement learning process.

[0193] Figure 4 The flow chart of deep reinforcement learning modeling includes:

[0194] Step 2.1, Markov decision process modeling of multi-agent

[0195] Specifically, the resource scheduling problem under multi-resource nodes is modeled as a multi-agent Markov decision process, which is represented by the following seven-tuple: We model the resource scheduling problem under multi-resource nodes as a partially observable distributed multi-agent Markov decision process, which is represented by the following seven-tuple:

[0196] (25) ​​

[0197] wherein, is a set of agents, corresponding to resource nodes in the system, each node is an agent, is the global environment state space, is the action space of the agent , is the state transition probability function, is the reward of the agent , is the discount factor, used to measure the weight of future rewards.

[0198] Step 2.2, state space design

[0199] Specifically, for each resource node , the state space is designed as:

[0200] (26)

[0201] wherein is the resource requested by slice m at t-1, is the resource allocated to m by node , is the resource adjustment action history (increase, decrease, maintain), is the node resource capacity, i.e. the constraint on node .

[0202] Step 2.3, action space design

[0203] Specifically, each resource agent can choose the direction of resource adjustment for three business slices (increase, decrease, maintain), that is:

[0204] (27)

[0205] Through the mapping function , the action number is mapped to:

[0206] (28)

[0207] Step 3, D3QN of fusion expert network for resource arrangement

[0208] Specifically, through the D3QN algorithm to learn the optimal strategy, on the basis of network slice prediction to realize the dynamic adjustment of resources, to ensure that the resource allocation is both efficient and does not exceed the capacity limit.

[0209] Figure 5 is the D3QN algorithm flow chart of traffic arrangement of fusion expert network, including:

[0210] Step 3.1, Initialization

[0211] Specifically, initialize the two main neural network networks. and target network and initializing the experience replay pool initial parameters During training, the main network calculates the Q-value based on the current state, selects an action (such as the ε-greedy policy), and collects rewards and the next state after executing the action, thus updating the experience pool. The target network evaluates the long-term benefits of the action, thereby providing a stable training signal.

[0212] Step 3.2: Obtain the current state and select an action.

[0213] Specifically, for each resource node To obtain its environmental status Then The Q-value is calculated for each action by inputting it into the Q-value calculation formula. The specific formula is as follows:

[0214] (29)

[0215] in, It refers to the current state value. This refers to extracting the advantage function for each action. This function value is output by the neural network and does not directly depend on reward or the environment; rather, it is learned through backpropagation during training. Indicates the action of selection. This represents all actions in the action space and is used to normalize the dominance function. This means averaging the advantages of all actions.

[0216] Then the main network adopts Strategy for selecting actions: based on probability Randomly select actions, with probability Choose the action that currently yields the highest estimated Q value. The motion space is Then use the mapping function Convert the action into slice resource adjustment amount.

[0217] Step 3.3: Execute the action to update resources.

[0218] Specifically, it depends on the action. Update resource allocation using the following formula:

[0219] (30)

[0220] in, This represents the updated estimate, indicating the value at time step [time step]. slices assigned to service types , the estimated resource share of refers to slices (eMBB, URLLC, mMTC); is the number of slices (eMBB, URLLC, mMTC - 3 slices), refers to the mapping of actions .

[0221] Step 3.4, get rewards and store experience

[0222] Specifically, the goal of each agent is to meet the service quality requirements of services, minimize resource waste and congestion, and the reward can be designed as follows, in time , the immediate reward of node is:

[0223] (31)

[0224] wherein is the different reward weight, is the estimated resource allocated to slice m, is the demand of slice m, is the service delay of the th slice, is the unmet resource request of slice , is a small value to avoid division by zero error, E is the unit benefit brought by unit resource, N is the total number of nodes, and is used for resource fairness index, is the total amount of resources obtained by slice m. Then the interaction experience is saved to the experience pool

[0225] :

[0226] (32)

[0227] wherein is the environment state of the resource node (such as the load of the resource node, the bandwidth occupancy rate, the slice demand), is the action space of node , is the reward calculated by formula 31, is the new environment state of the resource node . Step 3.5, pre-training of expert policy network

[0228]

[0229] ​​Specifically, by a large number of historical trajectories, the past experienced actors are selected to make decisions. Then the expert policy network is trained using cross-entropy loss, which is The specific formula is as follows:

[0230] (33)

[0231] Wherein, is the observed environment state in history, is the action selected by the expert in history, is the expert policy network to be trained, the value is the probability of the action output by the policy network in state . The purpose of this step is to train the expert policy network to minimize the loss function , so that the expert network learns which states in history select which actions, and then assist in guiding the D3QN network.

[0232] Step 3.6, expert policy network assists in training D3QN network

[0233] Specifically, B experience pairs are randomly sampled from the experience pool to update the main network Q value (Q), and the specific formula is:

[0234] (34)

[0235] Wherein, is the reward obtained by performing action in state , is the Q value vector for all possible future actions , which itself only predicts the value of each action and does not directly tell which action to choose; is the action with the maximum Q value in the Q value vector predicted by the main network; is the Q value corresponding to the action selected by the main network, which is calculated by the target network , so is the "current reward + discounted value of future reward", and the Q network is trained to maximize .

[0236] The loss function used to train the Q network is calculated as follows:

[0237] (35)

[0238] The formula is to measure the difference between the current Q network output and the target value . Wherein​​​​ the batch size for sampling from the experience pool; is the target Q value, obtained by equation (34), is the mean squared error, used to make the predicted Q value of the main network as close as possible to the "target Q value", minimized by backpropagation.

[0239] Then add expert guidance loss, make the policy of Q network close to the expert policy , using KL divergence:

[0240] (36)

[0241] where, denotes all actions in the action space, denotes the state of node at time step , is the probability of the expert network choosing action in state , is the probability of the Q network-derived policy choosing action in state , denotes how different the Q network's policy distribution is from the expert distribution in state , is the average KL divergence between the expert policy and the Q policy calculated by randomly sampling B experiences from the experience pool .

[0242] The final total loss function is obtained:

[0243] (37)

[0244] (38)

[0245] where and are constants, is the weight factor, which is a dynamic weight that decreases with the increase of time , is large in the early stage (to strengthen the guidance of the expert), and gradually decreases in the later stage to reduce interference and facilitate autonomous learning of the agent.

[0246] Step 3.7, obtain the optimal traffic arrangement strategy

[0247] By continuously learning how the resource nodes should adjust their resources in different states, the overall network reward is maximized. When the network is trained well, it enters the deployment and use stage, given the current observation , the weight Q network computing Expert network computing You can then select actions:

[0248] (39)

[0249] From numerous actions Obtained from This refers to traffic orchestration strategies.

Claims

1. An intelligent traffic orchestration method based on fusion of expert network and deep reinforcement learning, characterized in that, The method comprises the following steps: (1) network state perception and input construction: collecting multi-dimensional real-time running state parameters of all nodes and links in the network, and constructing a time series input vector for representing the space-time distribution of business traffic and a topological graph structure for representing the network connection relationship based on the collected data; (2) multi-dimensional feature extraction and fusion classification: processing the time series input vector by using a time series neural network model to extract the space-time features of the traffic, processing the topological graph structure by using a graph neural network model to extract the topological correlation features of the nodes, fusing the space-time features and the topological correlation features, and processing the fused features by using a nonlinear classification network to predict future traffic and complete coarse-grained arrangement of the future traffic into different service quality level logical network slices; (3) reinforcement learning environment modeling: modeling the network resource scheduling problem as a multi-agent Markov decision process, defining each resource node as an agent, and designing the state space, action space and reward function for each agent; the state space includes node resource state, historical allocation record and capacity constraint, the action space includes adjustment instructions for various slice resources, and the reward function is used to quantitatively evaluate the comprehensive performance of the action; (4) agent initialization and interaction: initializing the decision network, target network and experience storage unit of the deep reinforcement learning agent; the agent perceives the current state, outputs a value evaluation through the decision network, selects a resource adjustment action, executes the resource adjustment action to update the network resource allocation, then the environment feeds back a reward signal, and stores the interaction experience corresponding to the agent execution action and the environment feedback in the experience storage unit; (5) expert knowledge fusion and network training: pre-training an expert policy network using historical optimal decision data to simulate expert behavior; During the reinforcement learning training process, the loss of the decision network is calculated by sampling data from the experience storage unit, and the difference loss between the decision network strategy and the output of the expert policy network is calculated; the loss of the decision network and the difference loss are combined to construct a total loss function, and the decision network parameters are updated through back propagation, and the target network parameters are updated regularly; (6) optimal strategy generation: repeatedly executing the interaction and training process until the strategy converges, and finally outputting an intelligent model capable of generating an optimal traffic path and resource allocation strategy according to real-time network state. 2.The intelligent traffic orchestration method of claim 1, wherein, The collection of multi-dimensional real-time running state parameters of all nodes and links in the network in step 1 comprises: A set of network nodes is defined as : Each node contains state information of the network node: computing power , load , cache queue length , bandwidth utilization , energy consumption factor , reliability factor , task waiting time average , task priority density , cache data discard rate ; therefore, the node feature vector is represented as: ； A set of links is defined as if node is connected to node , then the characteristics of this connection are defined as: ; wherein, is the link bandwidth, is the link latency, is the link jitter, is the packet loss rate, is the link stability index, is the hop distance factor, i.e. the number of path hops from the source node to this connection. 3.The intelligent traffic orchestration method of claim 1, wherein, The construction of the input vector and the topological graph structure in step 1 comprises: Divide the target area into Grid, collect traffic for three types of services, enhanced mobile broadband, ultra-reliable low-latency communication, and massive machine communication, respectively, a total of three channels, and count the traffic of each type of service in each grid per hour: ; the three types of heat maps are spliced into: ; input a historical time sequence into the improved convolutional long short-term memory network: ; input tensor shape of wherein ; For the graph required for constructing the input graph neural network, the adjacency matrix is defined as: ; wherein, ; The node feature matrix is defined as : 。 4.The intelligent traffic orchestration method of claim 1, wherein, The processing of the time series input vector by using a time series neural network model to extract the space-time features of the traffic in step 2 comprises: The convolutional long short-term memory network is used for processing historical traffic heat map sequences to extract space-time features The convolutional long short-term memory network unit comprises an input gate , a forget gate , an output gate , a memory cell and a hidden state ; the input tensor of the convolutional long short-term memory network is a historical heat map sequence , wherein represents a traffic heat map at time ; the input gate , the forget gate and the output gate are respectively calculated by using the input at each time step , the current input is fused with the previous state, and and are updated A dynamic gating mechanism is introduced on the basis of a convolutional long short-term memory network, and a time series attention mechanism is introduced in the input gate to pay attention to key historical states; the specific calculation formula is as follows: ; ; ; ; ;in, For a moment The hidden state, For a moment The cell state, This represents the convolution operation, and ⊙ represents the Hadamard product. For activation function, The hyperbolic tangent activation function is used. For the forget gate bias term, For the output bias terms, This is a bias term for cell state updates. It is a linear transformation matrix that determines the influence of input, historical hidden state, and memory. Together with the bias term, it is a learnable weight parameter that is continuously updated through backpropagation during training. It is used to calculate the gating value, state update, and attention weight. Final output Heat map as extracted features for predicting next time instant; Modeling the topology while inputting X Input into the graph attention model to get the final graph embedding. 5.The intelligent traffic orchestration method of claim 1, wherein, In step 2, the graph neural network model is used to process the topological graph structure to extract the topological correlation features of the nodes, and a graph attention model is adopted, and the calculation process comprises: Initialization: The graph adjacency matrix A and the optimal feature set are input into the graph attention model; Compute attention coefficients: for each node , compute its attention coefficients with neighbor nodes , the attention coefficients measure the importance of the node ; introduce the mutual influence between neighbors as the second-order attention information, and then obtain the final calculation attention coefficient by adaptively fusing the first-order and second-order attention coefficients, the calculation formula is as follows: ; ; ; wherein, is a first-order attention coefficient, is a second-order attention coefficient, is a learnable weight matrix for mapping the node feature vector to a new feature space with dimension , is another learnable weight vector, denotes a concatenation operation, is a weight, is an input feature vector of node and , is a link feature between and , LeakyReLU is an activation function defined as ; Aggregated neighbor features: according to attention weights The features of the neighbor nodes are aggregated to obtain a new feature vector of the node , and the calculation formula is:​ ; wherein, is an activation function; in multi-head attention mechanism, i.e. multiple single-head attention computations are performed in parallel, resulting in multiple new feature vectors , K is the number of heads; then the feature vectors are concatenated to obtain the final node feature vector . 6.The intelligent traffic orchestration method of claim 1, wherein, The fusion classification described in step 2 includes: The extracted feature vectors are spliced: ; input F into Kolmogorov-Arnold Networks, output predicted traffic matrix; The shape of KAN is represented by an integer array: ; where is the number of nodes in the layer of the computational graph, and represents the activation value of the th neuron in the th layer, which is calculated as follows: ; ; ; ;in, For the first Layer The activation value of each neuron. For the previous layer The activation value of each neuron. To connect the first Layer The first neuron and the first Layer The activation function of each neuron is a trainable function. It is a mapping function that maps the input... That is, the output of the previous layer of neurons is mapped to a certain value. for Activate some weights, For multiple B-spline basis functions linear combination, The weights are the combinations of B-spline basis functions. These are the coefficients of the corresponding basis functions. They are all trainable. It corresponds to the first The function matrix of the KAN layers, when the KAN has L layers, the final output of the KAN is: ; wherein, represents function composition, the input feature F is mapped through multiple layers of function matrix, and high-order abstract representation is obtained layer by layer, and finally the traffic prediction of different network slice types flowing through the network nodes at a time step is obtained. 7.The intelligent traffic orchestration method of claim 1, wherein, The reinforcement learning environment modeling described in step 3 includes: The resource scheduling problem under the multi-resource node is modeled as a partially observable distributed multi-agent Markov decision process, and is defined by the following six-tuple: ; wherein, is a set of agents, and corresponds to a resource node in the system, each node being an agent, is a global environment state space, is an action space of an agent , is a state transition probability function, is a reward of an agent , is a discount factor, used to measure the weight of future rewards; For each resource node , the state space is designed as: ; wherein, is the resource requested by slice m at time t-1, is the node to which m has been allocated, is the resource adjustment action history, including increase, decrease, maintain, is the node resource capacity, i.e. the constraint on the node ; Each resource agent selects the direction of adjusting resources for three service slices, including increasing, decreasing, and maintaining, that is: ; by mapping function , the action number is mapped to: 。 8.The intelligent traffic orchestration method of claim 1, wherein, The agent initialization and interaction described in step 4 includes: Initialization: Initialize a main network for each agent. and a target network The main network The target network is used to calculate the action Q-value based on the current state and select an action. This is used to evaluate the long-term benefits of the action, providing a stable training signal; simultaneously, an experience replay pool is initialized. Used to store the agent's interaction experience data and initialize parameters. ; The agent obtains the state of each resource type on each resource node , calculates the Q value of each optional action by forward propagation, and the calculation formula is: ​​ ; where, is the current state value, is the advantage of extracting each action, the function value is output by the neural network, which does not directly depend on the reward or the environment, but is learned through back propagation during the training process, represents the selected action, represents all actions in the action space, which is used to normalize the advantage function, represents the average of the advantages of all actions; The main network adopts Strategy to select action: select action with probability Randomly select action with probability Select the action with the maximum current estimated Q value, and the action space is Convert the action to the slice resource adjustment amount using the mapping function Directly select the action with the maximum Q value during deployment; According to the action decision , update the resource allocation, as follows: ; wherein denotes the updated estimate, denotes the estimated resource share of the slice assigned to the traffic type at time step , refers to the slice eMBB, URLLC, mMTC; is the number of slices, refers to the mapping of the action ; Each agent aims to satisfy the quality of service requirement of the business, minimize resource waste and congestion, and is designed with the following reward at time step , the immediate reward of node is: ; wherein, are different reward weights, is the estimated resource allocated to slice m, is the demand of slice m, is the service delay of the m-th slice, is the service delay of the m-th slice, is the resource request of slice m that is not satisfied, is the resource request of slice m that is not satisfied, is a tiny value to avoid division by zero error, E is the unit profit brought by unit resource, and N is the total number of nodes, used for resource fairness index, is the total amount of resources obtained by slice m; The interaction experience is saved to the experience pool: ; wherein, is a state of the environment of the resource node , is an action space of the node , is an immediate reward of the node , is a new state of the environment of the resource node . 9.The intelligent traffic orchestration method of claim 1, wherein, The expert knowledge fusion and network training described in step 5 includes: A training data set is constructed using historical optimal decision data, which contains historical states and corresponding expert decision actions; the expert policy network is trained using cross-entropy loss: ; where, is a historically observed environment state, is a historically expert-chosen action, is an expert policy network to be trained, with value being the probability that the policy network outputs action in state B experiences are randomly sampled from the experience pool D for updating the main network Q value, and the target Q value is calculated for each experience, with the specific formula being: ; wherein, is the state performs an action obtained immediate reward, is the Q-value vector for all future possible actions ; is the action with the maximum Q-value in the Q-value vector predicted by the main network; is the target network computes the Q-value corresponding to the action, is the "current reward + discounted future reward", and the Q-network is trained such that is maximized. The loss function calculation formula is: ; this formula measures the gap between the current Q-network output and the target value , where is the batch size of the sampled experience pool; is the target Q-value, is the mean squared error used to let the main network's predicted Q-values approximate the "target Q-values" as closely as possible, minimizing through backpropagation; Adding an expert guidance loss to let the policy of the Q network approach the expert policy with the KL divergence: ; wherein, denotes all actions in the action space, denotes the state of the node at time step , is the probability of the expert network choosing action in state , is the probability of the policy derived from the Q network choosing action in state , denotes the difference between the policy distribution of the Q network and the expert distribution in state , is the average value of the KL divergence between the expert policy and the Q policy calculated by randomly sampling B experiences from the experience pool ; The final total loss function is obtained : ; ; wherein, and are constants, is a weight factor, is a dynamic weight, which decreases with the increase of time , is large in the early stage and gradually decreases in the later stage. 10.The intelligent traffic orchestration method of claim 1, wherein, The optimal policy generation described in step 6 includes: By repeatedly executing the interaction process of step (4) and the training process of step (5), the agent's policy is continuously optimized until the preset convergence condition is met; after the policy converges, the parameters of the main Q network and the expert policy network are solidified, and the trained intelligent model is obtained; In the deploy application phase, given the current observation , weights , Q network computation , expert network computation Perform action selection: ; from a multitude of actions is a traffic orchestration policy.​

Images