Intelligent flow arrangement method based on fusion expert network and deep reinforcement learning

Abstract

The invention discloses an intelligent flow arrangement method based on fusion expert network and deep reinforcement learning, which comprises the following steps: collecting network node and link state data in real time, and constructing a time sequence input vector and a topological graph structure; a time sequence neural network and a graph neural network are used for extracting traffic spatial-temporal features and node topological features respectively, future traffic is predicted through a classification network after fusion, and coarse-grained arrangement of network slices of different service levels is completed; modeling resource scheduling into a multi-agent Markov decision process, and designing a state space, an action space and a reward function; a deep reinforcement learning agent is initialized, and training is carried out through interaction experience; fusing a pre-trained expert strategy network, and constructing a total loss function to optimize network parameters; and finally generating an intelligent strategy capable of dynamically optimizing the flow path and resource allocation according to the real-time state. According to the invention, efficient resource scheduling under multi-service differentiation service quality requirements can be realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network traffic arrangement, and particularly relates to an intelligent traffic arrangement method based on a fusion of an expert network and deep reinforcement learning. BACKGROUND

[0002] With the continuous expansion of the power network system scale, the number of physical devices connected in the system rapidly increases, resulting in a sharp increase in the amount of data that needs to be transmitted, collected, stored and shared, and the problem of data transmission delay is increasingly prominent. In a resource-limited network environment, if there is a lack of effective traffic scheduling and resource management mechanism, the network service quality is difficult to meet the real-time, reliability and diversified business requirements of the power grid system. In the face of differentiated service requirements of low delay, high bandwidth and high reliability, the traditional static resource allocation and routing strategy has gradually become inadequate, and it is urgent to introduce a more intelligent and dynamic network management mechanism to adapt to the complex and changing operating environment.

[0003] To address the above challenges, network slicing technology is widely used in the prior art. As a core component of 5G and next-generation communication networks, this technology can divide the underlying physical network into multiple logically isolated networks according to business requirements, to respectively carry different types of services with different quality of service (QoS) requirements. Typical implementation methods include slice division and traffic scheduling mechanisms based on artificial rules or static strategies, such as pre-defined bandwidth reservation strategies, fixed routing tables or threshold-based heuristic algorithms, to preliminarily divide and assign network resources. To some extent, this method achieves multi-service isolation and resource differentiated services.

[0004] However, the existing method has a significant defect: the slice identification and traffic scheduling mechanism lacks dynamic perception and autonomous optimization capabilities. Due to the reliance on artificial preset rules or static configuration, it is difficult to respond to network state changes (such as node load fluctuations, link quality changes or burst traffic patterns) in real time, resulting in low resource utilization efficiency, unstable delay performance, and inability to adapt to the increasing requirements of power grid services for real-time and reliability. SUMMARY

[0005] The purpose of the present application is to provide an intelligent traffic arrangement method based on the fusion of an expert network and deep reinforcement learning, which can perceive the state of network devices in real time and meet the differentiated service requirements of low delay, high bandwidth and high reliability.

[0006] Technical solution: The intelligent traffic arrangement method according to the present application comprises the following steps: (1) Network state perception and input construction: Collect multi-dimensional real-time running state parameters of all nodes and links in the network, and construct a time series input vector for representing the spatio-temporal distribution of business traffic and a topological graph structure for representing the network connection relationship based on the collected data.

[0007] (2) Multi-dimensional feature extraction and fusion classification: Use a time series neural network model to process the time series input vector to extract the spatio-temporal features of the traffic, and use a graph neural network model to process the topological graph structure to extract the topological correlation features of the nodes; fuse the spatio-temporal features and topological correlation features, and process the fused features through a non-linear classification network to predict future traffic and complete the coarse-grained arrangement of dividing it into different service quality level logical network slices.

[0008] (3) Reinforcement learning environment modeling: Model the network resource scheduling problem as a multi-agent Markov decision process, define each resource node as an agent, and design its state space, action space and reward function for each agent; the state space includes node resource state, historical allocation record and capacity constraint, the action space includes adjustment instructions for various slice resources, and the reward function is used to quantitatively evaluate the comprehensive performance of the action.

[0009] (4) Agent initialization and interaction: Initialize the decision network, target network and experience storage unit of the deep reinforcement learning agent; the agent perceives the current state, outputs the value evaluation through the decision network and selects the resource adjustment action, executes the resource adjustment action to update the network resource allocation, then the environment feedbacks a reward signal, and stores the interaction experience corresponding to the agent's action and the environment's feedback in the experience storage unit.

[0010] (5) Expert knowledge fusion and network training: Pre-train an expert policy network with historical optimal decision data to simulate expert behavior; during the reinforcement learning training process, sample data from the experience storage unit to calculate the loss of the decision network, and calculate the difference loss between the decision network strategy and the output of the expert policy network; combine the loss of the decision network and the difference loss to construct a total loss function, and update the parameters of the decision network through back propagation, and synchronize the parameters of the target network regularly.

[0011] (6) Optimal strategy generation: Repeat the interaction and training process until the strategy converges, and finally output an intelligent model that can generate optimal traffic path and resource allocation strategy according to real-time network state.

[0012] The application realizes accurate perception of the space-time distribution of service traffic and the network connection relationship by constructing multi-dimensional real-time state input and network topology representation; with the joint feature extraction and fusion classification of time series and graph neural networks, the accuracy and adaptability of traffic prediction and slice arrangement are effectively improved; through multi-agent reinforcement learning environment modeling, the complex resource scheduling problem is converted into a learnable decision-making process; relying on the agent interaction and experience storage mechanism, continuous decision-making and experience accumulation in a dynamic environment are realized; combined with expert knowledge fusion and hybrid loss training, the stability and generalization ability of strategy learning are significantly enhanced; finally, through strategy iteration optimization, an intelligent decision-making model that can adapt to real-time network state, efficiently coordinate resource allocation and traffic path is generated, and the network resource utilization efficiency and service quality are comprehensively improved.

[0013] Preferably, the collection of multi-dimensional real-time running state parameters of all nodes and links in the network in step 1 comprises: Define the network node set as : Each node contains the state information of the network node: computing power , load , cache queue length , bandwidth utilization , energy consumption factor , reliability factor , task waiting time average , task priority density , cache data discard rate ; therefore, the node feature vector is represented as: ; Define the link set as , if node and node are connected, then define the connection characteristics as:

[0014] wherein, is the link bandwidth, is the link delay, is the link jitter, is the packet loss rate, is the link stability index, is the hop distance factor, i.e. the path hop number from the source node to the connection.

[0015] By defining the multi-dimensional refined feature vectors of nodes and links, the application realizes comprehensive quantitative perception of the running state of network infrastructure: the node features cover computing, load, cache, bandwidth, energy consumption, reliability and task-level indicators, and the link features integrate bandwidth, delay, jitter, packet loss, stability and path topology information, which together build a high-granularity, multi-dimensional network dynamic representation system, providing rich, accurate and interpretable environmental state input for subsequent intelligent decision-making.

[0016] Preferably, the input vector and the construction of the topology graph structure in step 1 include: The target area is divided into a grid, and three types of traffic, enhanced mobile broadband, ultra-reliable low-latency communication, and large-scale machine communication, are collected respectively, a total of three channels, and the traffic of a type of service in each cell is counted every hour:

[0017] The three types of heat maps are spliced into:

[0018] A historical time series is input to the improved convolutional long short-term memory network:

[0019] The shape of the input tensor is , wherein ; The graph required for constructing the input graph neural network is defined as an adjacency matrix:

[0020] Wherein, .

[0021] The node feature matrix is defined as : .

[0022] By gridding the target area and respectively counting the traffic data of three types of typical services, a multi-channel spatiotemporal heat map is constructed, realizing the refined characterization of the spatiotemporal distribution of service traffic; further, the historical time series heat map sequence is converted into a tensor input, providing a high-dimensional feature representation with spatial pattern and temporal dynamics for the convolutional long short-term memory network; at the same time, based on the node physical connection relationship and multi-dimensional state features, an adjacency matrix and a node feature matrix are constructed, forming a graph structure input that preserves the network topology association relationship, laying a solid foundation for the graph neural network to efficiently extract deep dependency relationships between nodes, and comprehensively improving the perception ability of the model to complex network state and service mode.

[0023] Preferably, the step 2 of processing the time series input vector using a time series neural network model to extract the spatiotemporal characteristics of traffic includes: A convolutional long short-term memory network is used to process the historical traffic heat map sequence to extract spatiotemporal features; the convolutional long short-term memory network unit includes an input gate , Forget Gate , output gate , memory cells and hidden state ; The input tensor of the convolutional long short-term memory network is the historical heat map sequence ,in Indicates at time Traffic heat map; using the input for each time step Calculate the input gate separately , Forget Gate , output gate , fuse the current input with the previous state, and update and ; A dynamic gating mechanism is introduced based on the convolutional long short-term memory network, and a temporal attention mechanism is introduced in the input gate to focus on key historical states. The specific calculation formula is as follows: ; ; ; ; ; in, is the temporal attention factor, is the rate of change of the heat map in the previous step, For the moment The hidden state of For the moment The unit status, is the convolution operation, is the Hadamard product, is the sigmoid function, is the activation function, is the hyperbolic tangent activation function, is the forget gate bias term, is the output bias term, is the bias term for cell state update, It is a linear transformation matrix that determines the influence of input, historical hidden state, and memory. Together with the bias term, it is a learnable weight parameter that is continuously updated through backpropagation during training and is used to calculate gate values, state updates, and attention weights. Final output Heatmap for predicting next time; Modeling topology while inputting X Input into graph attention model to get final graph embedding.

[0024] By fusing convolutional long short-term memory network and dynamic gating mechanism, and introducing time series attention factor and heat map change rate as enhanced input, this design can accurately capture the complex spatio-temporal dependence relationship contained in the traffic heat map; the time series attention mechanism can adaptively weight the key historical state, which significantly improves the perception ability of important time nodes, and the dynamic update of the gating structure and cell state effectively coordinates the fusion and forgetting process of the current input and historical information; the high-level features of the final output not only retain the spatio-temporal evolution law of traffic, but also provide discriminative representation for the subsequent deep fusion with topology features, and comprehensively enhance the model's prediction and understanding ability of business traffic dynamics.

[0025] Preferably, the topology graph structure is processed by a graph neural network model in step 2 to extract the topology correlation features of the nodes, and a graph attention model is used, and the calculation process includes: Initialization: input the graph adjacency matrix A and the optimal feature set into the graph attention model; Calculate attention coefficient: for each node , calculate the attention coefficient of its neighbor node , the attention coefficient measures the importance of to node ; the mutual influence between neighbors is introduced as second-order attention information, and the final calculation attention coefficient is obtained by adaptively fusing the first-order and second-order attention coefficients, and the calculation formula is as follows: ; ; ; wherein, is the first-order attention coefficient, is the second-order attention coefficient, is a learnable weight matrix for mapping the node feature vector to a new feature space, and the dimension is , is another learnable weight vector, represents the splicing operation, is the weight, is the input feature vector of node and , is and between the link features, LeakyReLU is an activation function, defined as ; Aggregate neighbor features: aggregate the features of neighbor nodes according to the attention weights to obtain the new feature vector of the node , the calculation formula is: ; wherein, is an activation function; in the multi-head attention mechanism, that is, multiple single-head attention calculations are performed in parallel to obtain multiple new feature vectors , K is the number of heads; then the feature vectors are spliced to obtain the final node feature vector .

[0026] By introducing the adaptive graph attention mechanism that fuses first-order and second-order attention coefficients, the model can accurately quantify the complex topological dependence and mutual influence between nodes; by using link features to enhance neighbor importance calculation and combining a multi-head parallel aggregation strategy, the discriminability and robustness of node feature expression are significantly improved, and the finally generated node embedding effectively preserves the global topological structure information and local correlation pattern of the network, providing high-precision topological correlation representation for subsequent cross-modal feature fusion and collaborative decision-making.

[0027] Preferably, the fusion classification of step 2 comprises: Splice the extracted feature vectors:

[0028] Input F into Kolmogorov-Arnold Networks to output the predicted traffic matrix.

[0029] The shape of KAN is represented by an integer array:

[0030] wherein is the number of nodes in the layer of the computation graph, the activation value of the th neuron in the layer is denoted as , and the calculation formula of the activation value is as follows: ; ; ; ; wherein, is the th layer ​The activation value of a neuron, For the previous layer The activation value of a neuron, To connect Tier neurons and Tier The activation function of a neuron is a trainable function. is a mapping function that takes the input , that is, the output of the previous layer of neurons is mapped to a certain value, for Activate some weights, are multiple B-spline basis functions The linear combination of is the weight of the B-spline basis function combination, are the coefficients of the corresponding basis functions, They are all trainable. It corresponds to The function matrix of the KAN layer, when the KAN layer is L, the final KAN output is: ; in, Representing function composition, the input feature F is mapped through multiple layers of function matrices to obtain high-order abstract representations layer by layer, and finally a traffic prediction of different network slice types flowing through the network node in a time step is obtained.

[0031] By deeply fusing spatiotemporal features with topological correlation features and using Kolmogorov-Arnold Networks based on trainable function matrices for nonlinear transformation, the model breaks through the limitations of traditional linear combinations and achieves accurate extraction and interaction of high-order abstract features. It uses B-spline basis functions to flexibly fit complex mapping relationships, significantly improving the expressiveness and generalization performance of traffic prediction. The final output is a high-precision prediction result that can accurately reflect the traffic distribution of network nodes on different slice types in the future, providing reliable data support for subsequent resource orchestration.

[0032] Preferably, the reinforcement learning environment modeling described in step 3 includes: The resource scheduling problem under multiple resource nodes is modeled as a partially observable distributed multi-agent Markov decision process, which is defined by the following seven-tuple:

[0033] in, is a collection of intelligent agents, corresponding to resource nodes in the system, each node is an intelligent agent, is the global environment state space, For intelligent agents The action space, is the state transition probability function, For intelligent agents Rewards, is the discount factor used to measure the weight of future rewards; For each resource node , design state space for:

[0034] in, is the resource requested by slice m at time t-1, For nodes The resources allocated to m, Adjust the action history for resources, including increase, decrease, and maintain, is the node resource capacity, that is, the node constraints; Each resource agent chooses the direction of adjusting resources for the three business slices, including increasing, decreasing, and maintaining, namely:

[0035] Through the mapping function , mapping the action numbers to: .

[0036] By modeling the distributed resource scheduling problem as a partially observable multi-agent Markov decision process, the framework achieves accurate abstraction and distributed decision support for complex network environments; by designing a multi-dimensional state space for each resource node agent that includes resource requests, allocation records, action history and capacity constraints, as well as a discrete adjustment action space for multiple business slices, it effectively characterizes the local observation and decision-making capabilities of the agent, provides a learnable and scalable mathematical foundation for collaborative resource optimization, and lays the theoretical foundation for generating efficient distributed resource scheduling strategies.

[0037] Preferably, the agent initialization and interaction described in step 4 includes: Initialization: Initialize a main network for each agent and a target network , the main network It is used to calculate the action Q value and select the action according to the current state. Used to evaluate the long-term benefits of the action to provide a stable training signal; at the same time, initialize an experience replay pool Used to store the agent's interaction experience data and initialize parameters ; The agent obtains the state of each resource type on each resource node , calculates the Q value of each optional action by forward propagation, and the formula is as follows:

[0038] wherein, is the current state value, is the advantage of extracting each action, the function value is output by the neural network, and is not directly dependent on the reward or the environment, but is learned during the training process by back propagation, represents the selected action, represents all actions in the action space, which is used to normalize the advantage function, represents the average of the advantages of all actions; The main network adopts the policy to select actions: with a probability , an action is randomly selected, and with a probability , the action with the maximum current estimated Q value is selected, and the action space is , the action is converted into a slice resource adjustment amount by using the mapping function , and the action with the maximum Q value is directly selected during deployment; According to the action determined by , the resource allocation is updated, and the formula is as follows:

[0039] wherein represents the updated estimated value, and represents the estimated resource share of the slice eMBB, URLLC, and mMTC allocated to the business type at the time step , is the slice eMBB, URLLC, and mMTC, is the adjustment step length, which is used to control the amplitude of the allocated resources; is the number of slices, is the mapping of the action ; Each agent aims to minimize resource waste and congestion while meeting the service quality requirements of businesses, and the following reward is designed, and the immediate reward of the node at the time step is as follows:

[0040] wherein, is a different reward weight, is the estimated resource allocated to the slice m, is the demand of the slice m, ​​​The service delay of the slice, The service delay of the slice, The resource request of the slice The unmet resource request, is a micro value to avoid division by zero error, E is the unit resource, N is the total number of nodes, and is used for resource fairness index, The total amount of resources obtained by the slice m; Save the interaction experience to the experience pool:

[0041] Wherein, The environment state of the resource node , The action space of the node , The immediate reward of the node , The new environment state of the resource node .

[0042] By initializing the main network and the target network for each agent and constructing the experience replay pool, the mechanism realizes the cooperation of distributed decision-making and stable training; Using the Q value calculation method based on state value and action advantage decomposition, the accuracy of action evaluation and the efficiency of policy learning are effectively improved; Through the ε-greedy strategy to balance exploration and utilization, and combining with the resource adjustment mapping function to convert discrete actions into actual resource allocation, the feasibility and practicality of the strategy are ensured; The multi-objective reward function is designed to integrate resource utilization, service delay, demand satisfaction and fairness, which guides the agent to autonomously learn the optimal scheduling strategy with the goal of minimizing resource waste and congestion, and finally provides high-quality data support for subsequent training through experience storage.

[0043] Preferably, the expert knowledge fusion and network training of step 5 comprise: Use the historical optimal decision data to construct a training data set, which contains historical states and corresponding expert decision actions; Use cross-entropy loss to train the expert policy network:

[0044] Wherein, is the observed environment state in history, is the action selected by the expert in history, is the expert policy network to be trained, the value is the probability of the policy network outputting action Under state ; Randomly sample B experience pairs from the experience pool D for updating the Q value of the main network, and calculate the target Q value for each experience, the specific formula is:

[0045] wherein, is the state at time step , the action is the immediate reward obtained by taking action in state , the Q-value vector is the Q-value vector predicted by the main network , the action is the action with the maximum Q-value in the Q-value vector predicted by the main network , the target Q-value

[0046] The formula is used to measure the gap between the output of the current Q network and the target value , wherein is the batch size sampled from the experience pool; is the target Q-value, is the mean square error, which is used to make the predicted Q-value of the main network as close as possible to the target Q-value, and is minimized through back propagation; The expert guidance loss is added to make the strategy of the Q network close to the expert strategy , and the KL divergence is used:

[0047] wherein, represents all actions in the action space, represents the state of node at time step , the probability of the action selected by the expert network in state , the probability of the action selected by the strategy derived by the Q network in state , the difference between the strategy distribution of the Q network and the expert distribution in state , the average value of the KL divergence between the expert strategy and the Q strategy calculated by randomly sampling B experiences from the experience pool , and the total loss function is obtained: ​​​​​​

[0048] wherein, and are constants, is a weight factor, is a dynamic weight, which decreases with the increase of time , is large in the early stage and gradually decreases in the later stage.

[0049] By fusing historical expert decision data and online interaction experience, the training mechanism effectively combines expert prior knowledge and reinforcement learning exploration ability; the cross-entropy loss is used to pre-train the expert policy network to simulate optimal decision behavior, and the KL divergence is used to construct the expert guidance loss, so that the agent policy closely fits the expert experience in the early stage of training, significantly improving the learning efficiency and stability; at the same time, based on the target Q value calculation and the mean square error loss, the accuracy of the value function convergence is guaranteed, and the dynamic weighting strategy adaptively balances the contribution of imitation learning and autonomous exploration in the training process, and finally realizes the continuous optimization and reliable improvement of the policy performance.

[0050] Preferably, the optimal strategy generation of step 6 comprises: By repeatedly executing the interaction process of step (4) and the training process of step (5), the policy of the agent is continuously optimized until the preset convergence condition; after the policy converges, the parameters of the main Q network and the expert policy network are solidified, and the trained intelligent model is obtained; In the deployment application stage, given the current observation , the weight , the Q network calculates , the expert network calculates to select an action:

[0051] The obtained from a plurality of actions is the traffic arrangement strategy.

[0052] By iteratively executing the interactive exploration and hybrid training process, the policy of the agent is continuously optimized and finally converges to a stable state; after the policy converges, the solidified intelligent model deeply integrates the adaptive decision-making ability of reinforcement learning and the reliability of expert knowledge, and in the deployment stage, it can cooperatively use the value evaluation of the Q network and the policy suggestion of the expert network according to the real-time observation state, dynamically generate traffic arrangement actions that take into account short-term benefits and long-term optimality, and thus realize efficient, accurate and autonomous scheduling of network resources.

[0053] Beneficial effects: compared with the prior art, the present application has the following remarkable advantages: 1. Through multi-dimensional real-time perception of network running state and accurate prediction of traffic space-time law, intelligent allocation and scheduling of network resources are realized, effectively meeting the differentiated service quality requirements of high transmission rate, low latency and high reliability of different service types such as enhanced mobile broadband, ultra-reliable low-latency communication and large-scale machine communication; 2. Multi-agent collaborative decision is adopted, with comprehensive performance as the optimization target, automatic learning of optimal resource allocation strategy, significant reduction of resource waste and congestion, and improvement of global resource utilization; 3. Historical expert experience and online environment interaction are fused, model convergence is accelerated through hybrid training mechanism, and the decision-making process has both expert reliability and environmental adaptability; 4. The final model can automatically generate and continuously optimize traffic path and resource allocation strategy according to real-time network state, with strong adaptive ability to cope with network fluctuations and service changes. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flowchart of the method of the present application; Figure 2 is an intelligent traffic arrangement architecture diagram of the present application; Figure 3 is a coarse-grained traffic arrangement flowchart of multi-dimensional data feature fusion of the present application; Figure 4 is a deep reinforcement learning modeling flowchart of the present application; Figure 5 is a D3QN algorithm traffic arrangement flowchart of the present application fusing expert networks. DETAILED DESCRIPTION

[0055] The technical solutions of the present application will be further described below in combination with the drawings.

[0056] The embodiment of the present application provides an intelligent traffic arrangement method based on fusion of expert networks and deep reinforcement learning, Figure 1 is a flowchart of the method, Figure 2For the intelligent traffic arrangement architecture of the method, the dynamic traffic arrangement optimization problem is proposed by the application, which considers the slice average efficiency and utility measurement. In addition, the application proposes an intelligent traffic arrangement framework based on a deep reinforcement learning of network slicing and fusion expert network. In the method, each resource configuration unit is managed by two controllers, and coarse-grained and fine-grained arrangement is combined in the traffic arrangement process. Convolutional neural network is used to extract and graph attention network to extract different dimensions of network node features. Through KAN network, the fusion features are classified by network slicing, and the network nodes are divided into three logical slices of large bandwidth, high reliability and low delay, and mass access. On this basis, deep reinforcement learning is introduced to combine expert network in the training process of slice path scheduling, resource selection and traffic scheduling. A large amount of historical data is used to find out the optimal decision to form an expert trajectory. The expert network is used to optimize and correct the reinforcement learning result, and finally the intelligent traffic path arrangement strategy is output. The method comprises the following steps: Step 1, coarse-grained traffic arrangement based on multi-dimensional data feature fusion.

[0057] Specifically, assuming that there are N network nodes in the network, the improved convolutional long short-term memory network and the graph attention model are used to extract the information of each network node and the features of the traffic flowing through the network node, and to predict the network slice type of the traffic flowing through the network node in the future time step t. The traffic arrangement is coarse-grained and arranged into three types of enhanced mobile broadband, ultra-reliable low-latency communication and large-scale machine communication.

[0058] Figure 3 The flow chart of coarse-grained traffic arrangement based on multi-dimensional data feature fusion comprises: Step 1.1, network state information acquisition.

[0059] Specifically, let the network node set be : (1) Each node contains the state information of the network node: computing power , load , cache queue length , bandwidth utilization , energy consumption factor , reliability factor , task waiting time average , task priority density , cache data discard rate . Therefore, the node feature vector is represented as: (2) Let the link set be If node is connected to node , the edge feature is defined as: (3) where is the link bandwidth, is the link delay, is the link jitter, is the packet loss rate, is the link stability index, is the hop distance factor, i.e., the number of hops from the source node to the edge.

[0060] Step 1.2, input vector construction.

[0061] Specifically, the target area is divided into grid, and three types of traffic (enhanced mobile broadband, ultra-reliable low-latency communication, and large-scale machine communication) are collected, totaling three channels. The traffic of each type of service in each grid is counted every hour, resulting in: (4) The three types of heat maps are spliced as: (5) The input to the improved convolutional long short-term memory network is a historical time series: (6) The shape of the input tensor is , where .

[0062] To construct the graph required by the input graph neural network, the adjacency matrix is defined as: (7) where .

[0063] The node feature matrix is defined as : (8) Step 1.3, multi-dimensional feature extraction.

[0064] Specifically, the convolutional long short-term memory network is a deep learning model that combines convolutional neural networks and long short-term memory network mechanisms for processing spatial-temporal data. Each unit contains: input gate , forget gate , output gate , memory cell and hidden state , the input tensor of the convolutional long short-term memory network is the historical heat map sequence ,in Indicates at time Traffic heat map. Using the input for each time step Calculate the input gate separately , Forget Gate , output gate , fuse the current input with the previous state, and then update and , a dynamic gating mechanism is introduced based on the convolutional long short-term memory network to improve the nonlinear modeling capability, and an attention mechanism is introduced in the input gate to let the model pay attention to which historical states are more important. The specific formula is: (9) (10) (11) (12) (13) in, is the temporal attention factor, is the rate of change of the heat map in the previous step, For the moment The hidden state of For the moment The unit status, is the convolution operation, is the Hadamard product (element-wise multiplication), is the sigmoid function, is the activation function, is the hyperbolic tangent activation function, is the forget gate bias term, is the output bias term, is the bias term for cell state update, It is a linear transformation matrix (learnable parameter matrix) that determines the influence of input, historical hidden state and memory. Together with the bias term, it is a learnable weight parameter that is continuously updated through backpropagation during training and is used to calculate gate values, state updates, and attention weights.

[0065] Final Output It can be used as the extracted features to predict the heat map at the next moment.

[0066] While inputting X, the topology is modeled Input into the graph attention model to obtain the final graph embedding. The calculation of the graph attention model is divided into three steps: (1) Initialization: First, the graph adjacency matrix A and the optimal feature set are input into the graph attention model.

[0067] (2) Calculate attention coefficients: For each node , calculate its attention coefficient with neighbor nodes , which measures the importance of to node . Introduce the mutual influence between neighbors as second-order attention information, and then obtain the final calculation attention coefficient by adaptively fusing first-order and second-order attention coefficients. The calculation formula is as follows: (14) (15) (16) where is the first-order attention coefficient, is the second-order attention coefficient, is a learnable weight matrix used to map the node feature vector to a new feature space (dimension ), is another learnable weight vector, denotes the concatenation operation, is the weight, is the input feature vector of node and , is the link feature between and , and LeakyReLU is an activation function, usually defined as .

[0068] (3) Aggregate neighbor features: Aggregate the features of neighbor nodes according to the attention weight to obtain the new feature vector of node . The calculation formula is: (17) where is an activation function. If a multi-head attention mechanism is used, i.e., multiple single-head attention calculations are performed in parallel to obtain multiple new feature vectors , K is the number of heads. Then concatenate these feature vectors to obtain the final node feature vector .

[0069] Step 1.4, multi-dimensional feature fusion and network slice classification.

[0070] In particular, the extracted feature vectors are concatenated: (18) F is input into a Kolmogorov-Arnold Network (KAN), which outputs a predicted flow matrix.

[0071] The shape of a KAN is represented by an array of integers: (19) where N is the number of nodes in the i-th layer of the computational graph. The activation value of the j-th neuron in the i-th layer, denoted as aij, is calculated as follows: where a^ is the activation value of the j-th neuron in the i-th layer, a^ is the activation value of the j-th neuron in the previous layer, and f is an activation function connecting the j-th neuron in the i-th layer and the j-th neuron in the i+1-th layer, which is a trainable function, different from the fixed activation function of traditional DNNs. (20) where a^ is the activation value of the j-th neuron in the i-th layer, a^ is the activation value of the j-th neuron in the previous layer, and f is an activation function connecting the j-th neuron in the i-th layer and the j-th neuron in the i+1-th layer, which is a trainable function, different from the fixed activation function of traditional DNNs. (21) (22) (23) where is a mapping function that maps the input (output of the previous layer of neurons) to a certain value, is the activation part weight, is a linear combination of multiple B-spline basis functions

[0072] (21) (22) (23) where is a mapping function that maps the input (output of the previous layer of neurons) to a certain value, is the activation part weight, is a linear combination of multiple B-spline basis functions is a function matrix corresponding to the i-th KAN layer, and when the KAN has L layers, the output of the final KAN is: (24) where​​​​​​​​​​​​​​​​​​​ Representative function composition, input features F are mapped through multiple layers of function matrices, and high-order abstract representations are obtained layer by layer, and finally the traffic prediction of different network slice types flowing through the network node in the future time step can be realized.

[0073] Step 2, deep reinforcement learning modeling.

[0074] Specifically, a multi-agent reinforcement learning system is designed to realize intelligent resource arrangement, each resource node as an agent, selects actions according to state information, and designs state space and action space in detail, which provides a basis for subsequent deep reinforcement learning process.

[0075] Figure 4 The flow chart of deep reinforcement learning modeling includes: Step 2.1, multi-agent Markov decision process modeling Specifically, the resource scheduling problem under multi-resource nodes is modeled as a multi-agent Markov decision process, which is represented by the following seven-tuple: We model the resource scheduling problem under multi-resource nodes as a partially observable distributed multi-agent Markov decision process, which is represented by the following seven-tuple: (25) Among them, is the set of agents, corresponding to the resource nodes in the system, each node is an agent, is the global environment state space, is the action space of agent , is the state transition probability function, is the reward of agent , is the discount factor, which is used to measure the weight of future rewards.

[0076] Step 2.2, state space design Specifically, for each resource node , the state space is designed as: (26) Among them is the resource requested by slice m at t-1, is the resource allocated to m by node , is the resource adjustment action history (increase, decrease, maintain), is the node resource capacity, that is, the constraint on node .

[0077] Step 2.3, action space design Specifically, each resource agent can select the direction of adjusting resources (increase, decrease, maintain) for three service slices, that is: (27) By mapping function , the action number is mapped to: (28) Step 3, D3QN of fusion expert network for resource arrangement Specifically, the optimal strategy is learned through the D3QN algorithm, and the dynamic adjustment of resources is realized on the basis of network slice prediction, ensuring that the resource allocation is efficient and does not exceed the capacity limit.

[0078] Figure 5 The flow chart of the D3QN algorithm for traffic arrangement of the fusion expert network includes: Step 3.1, initialization Specifically, two neural network main networks and target networks are initialized, and the experience replay pool is initialized , the initial parameters In the training process, the main network calculates the Q value according to the current state, selects the action (such as the ε-greedy strategy), collects the reward and the next state after executing the action, and updates the experience pool. The target network evaluates the long-term benefits of the action, thereby providing stable training signals.

[0079] Step 3.2, get the current state and select the action Specifically, for each resource node , its environment state is obtained, and then is input into the Q value calculation formula to calculate the Q value of each action, and the specific formula is: (29) Where, is the current state value, is the advantage function of each action, and the function value is output by the neural network, which does not directly depend on the reward or the environment, but is learned through backpropagation in the training process, represents the selected action, represents all actions in the action space, which is used to normalize the advantage function. It represents the average of the advantages of all actions.

[0080] Then the main network adopts strategy to select the action: with a probability randomly select an action, and with a probability select the action with the maximum current estimated Q value The action space is Then, the action is converted to the slice resource adjustment amount by the mapping function .

[0081] Step 3.3, execute the action and update the resource Specifically, according to the action decision , update the resource allocation, the formula is as follows: (30) wherein represents the updated estimated value, represents the estimated resource share of the slice assigned to the service type at the time step , refers to the slice (eMBB, URLLC, mMTC), is the adjustment step length, used to control the amplitude of the allocated resource; is the number of slices (eMBB, URLLC, mMTC - 3 slices), refers to the mapping of the action .

[0082] Step 3.4, obtain the reward and store the experience Specifically, the goal of each agent is to meet the service quality requirements of the business, minimize resource waste and congestion, and the reward can be designed as follows: at time , the immediate reward of node is: (31) wherein is the different reward weight, is the estimated resource allocated to the slice m, is the demand of the slice m, is the service delay of the first type of slice, is the unmet resource request of the slice, is a small value to avoid division by zero error, E is the unit benefit brought by the unit resource, N is the total number of nodes, used for resource fairness index, is the total amount of resources obtained by the slice m.

[0083] Then save the interaction experience to the experience pool : (32) wherein is the environment state of the resource node (such as the load of the resource node, the bandwidth occupancy rate, the slice demand),​ For nodes The action space, is the reward calculated by formula 31, Resource Node The new environment state.

[0084] Step 3.5: Expert Strategy Network Pre-training Specifically, through a large number of historical trajectories, we select how experienced actors made decisions in the past. Then we use the cross entropy loss to train the expert policy network. The loss function is The specific formula is as follows: (33) in, is the historically observed state of the environment, moves chosen by experts in history, Is the expert policy network to be trained, and the value is the policy network in state Down output action The purpose of this step is to train the expert strategy network to minimize the loss function , let the expert network learn which states have selected which actions in history, and then Assist in guiding the D3QN network.

[0085] Step 3.6: Expert strategy network assisted training of D3QN network Specifically, from the experience pool Randomly sample B experience pairs to update the Q value of the main network ( ), the specific formula is: (34) in, In state Execute an action Rewards received, For all possible future actions The Q-value vector of , which itself only predicts the value of each action and does not directly tell which action to choose; It is to select the action with the largest Q value in the Q value vector predicted by the main network; After selecting the action for the main network, use the target network Calculate the Q value corresponding to the action, so is the "discounted value of current reward + future reward", and the Q network should be trained so that maximum.

[0086] The loss function calculation formula used in training the Q network is: (35) The formula is to measure the gap between the current Q network output and the target value . Where is the batch size of the sampled experience pool; is the target Q value, obtained by formula (34), is the mean square error, which is used to make the predicted Q value of the main network as close as possible to the "target Q value", and is minimized by back propagation.

[0087] Then add expert guidance loss to make the Q network strategy close to the expert strategy , using KL divergence: (36) Where, represents all actions in the action space, represents the state of node at time step , is the probability of the expert network choosing action in state , is the probability of the Q network derived strategy choosing action in state , represents how different the Q network strategy distribution is from the expert distribution in state , is the average value of the KL divergence between the expert strategy and the Q strategy calculated by randomly sampling B experiences from the experience pool .

[0088] The final total loss function is: (37) (38) Where and are constants, is the weight factor, which is a dynamic weight that decreases with the increase of time , and is large in the early stage (to strengthen the guidance of the expert) and gradually decreases to reduce interference and facilitate autonomous learning of the agent.

[0089] Step 3.7, get the best traffic arrangement strategy By continuously learning how the resource nodes should adjust their resources in different states to maximize the overall network reward. When the network is trained well, it enters the deployment and use stage, given the current observation , the weight , the Q network calculates , Expert Network Computing You can select the action: (39) From many actions Obtained from This is the traffic orchestration strategy.

Claims

1. An intelligent traffic orchestration method based on fusion of expert network and deep reinforcement learning, characterized in that, The method comprises the following steps: (1) network state perception and input construction: collecting multi-dimensional real-time running state parameters of all nodes and links in the network, and constructing a time series input vector for representing the space-time distribution of business traffic and a topological graph structure for representing the network connection relationship based on the collected data; (2) multi-dimensional feature extraction and fusion classification: processing the time series input vector by using a time series neural network model to extract the space-time features of the traffic, processing the topological graph structure by using a graph neural network model to extract the topological correlation features of the nodes, fusing the space-time features and the topological correlation features, and processing the fused features by using a nonlinear classification network to predict future traffic and complete coarse-grained arrangement of the future traffic into different service quality level logical network slices; (3) reinforcement learning environment modeling: modeling the network resource scheduling problem as a multi-agent Markov decision process, defining each resource node as an agent, and designing the state space, action space and reward function for each agent; the state space includes node resource state, historical allocation record and capacity constraint, the action space includes adjustment instructions for various slice resources, and the reward function is used to quantitatively evaluate the comprehensive performance of the action; (4) agent initialization and interaction: initializing the decision network, target network and experience storage unit of the deep reinforcement learning agent; the agent perceives the current state, outputs a value evaluation through the decision network, selects a resource adjustment action, executes the resource adjustment action to update the network resource allocation, then the environment feeds back a reward signal, and stores the interaction experience corresponding to the agent execution action and the environment feedback in the experience storage unit; (5) expert knowledge fusion and network training: pre-training an expert policy network using historical optimal decision data to simulate expert behavior; During the reinforcement learning training process, the loss of the decision network is calculated by sampling data from the experience storage unit, and the difference loss between the decision network strategy and the output of the expert policy network is calculated; the loss of the decision network and the difference loss are combined to construct a total loss function, and the decision network parameters are updated through back propagation, and the target network parameters are updated regularly; (6) optimal strategy generation: repeatedly executing the interaction and training process until the strategy converges, and finally outputting an intelligent model capable of generating an optimal traffic path and resource allocation strategy according to real-time network state. 2.The intelligent traffic orchestration method of claim 1, wherein, The collection of multi-dimensional real-time running state parameters of all nodes and links in the network in step 1 comprises: A set of network nodes is defined as : Each node contains state information of the network node: computing power , load , cache queue length , bandwidth utilization , energy consumption factor , reliability factor , task waiting time average , task priority density , cache data discard rate ; therefore, the node feature vector is represented as: ; define a set of links as , if node is connected to node , then define the characteristics of this connection as: ; wherein, is the link bandwidth, is the link latency, is the link jitter, is the packet loss rate, is the link stability index, is the hop distance factor, i.e. the number of path hops from the source node to this connection. 3.The intelligent traffic orchestration method of claim 1, wherein, The construction of the input vector and the topological graph structure in step 1 comprises: Divide the target area into Grid, collect traffic for three types of services, enhanced mobile broadband, ultra-reliable low-latency communication, and massive machine communication, respectively, a total of three channels, and count the traffic of each type of service in each grid per hour: ; the three types of heat maps are spliced as: ; a historical time sequence is input to the improved convolutional long short-term memory network: ; the shape of the input tensor is , wherein ; for constructing the graph required by the input graph neural network, the adjacency matrix is defined as: ; wherein, ; the node feature matrix is defined as : . 4.The intelligent traffic orchestration method of claim 1, wherein, The processing of the time series input vector by using a time series neural network model to extract the space-time features of the traffic in step 2 comprises: A convolutional long short-term memory network is used to process the historical traffic heat map sequence to extract spatiotemporal features; the convolutional long short-term memory network unit includes an input gate , Forget Gate , output gate , memory cells and hidden state ; The input tensor of the convolutional long short-term memory network is the historical heat map sequence ,in Indicates at time Traffic heat map; using the input for each time step Calculate the input gate separately , Forget Gate , output gate , fuse the current input with the previous state, and update and ; A dynamic gating mechanism is introduced on the basis of a convolutional long short-term memory network, and a time series attention mechanism is introduced in the input gate to pay attention to key historical states; the specific calculation formula is as follows: ； ； ； ； ; wherein, is a timing attention factor, is a heat map change rate of the previous step, is a hidden state at time , is a cell state at time , is a convolution operation, is a Hadamard product, is a sigmoid function, is an activation function, is a hyperbolic tangent activation function, is a forget gate bias term, is an output gate bias term, is a cell state update bias term, is a linear transformation matrix, determining the influence of input, historical hidden state and memory, both together with the bias term are learnable weight parameters, which are constantly updated during the training process through backpropagation, used to calculate the gating value, state update, attention weight; Final output Heat map as extracted features for predicting next time instant; Modeling the topology while inputting X Input into the graph attention model to get the final graph embedding. 5.The intelligent traffic orchestration method of claim 1, wherein, In step 2, the topological correlation features of the nodes are extracted by using a graph neural network model, and a graph attention model is used, and the calculation process comprises: Initialization: The graph adjacency matrix A and the optimal feature set are input into the graph attention model; Compute attention coefficients: for each node , compute its attention coefficients with neighbor nodes , the attention coefficients measure the importance of the node ; introduce the mutual influence between neighbors as the second-order attention information, and then obtain the final calculation attention coefficient by adaptively fusing the first-order and second-order attention coefficients, the calculation formula is as follows: ； ； ; wherein, is a first-order attention coefficient, is a second-order attention coefficient, is a learnable weight matrix for mapping the node feature vector to a new feature space with dimension , is another learnable weight vector, denotes a concatenation operation, is a weight, is an input feature vector of a node and , is a link feature between and , LeakyReLU is an activation function defined as ; Aggregate neighbor features: according to attention weights Aggregate features of neighbor nodes to get new feature vector of node , the calculation formula is:​ ; wherein, is an activation function; in multi-head attention mechanism, i.e. multiple single-head attention computations are performed in parallel, resulting in multiple new feature vectors , K is the number of heads; then the feature vectors are concatenated to obtain the final node feature vector . 6.The intelligent traffic orchestration method of claim 1, wherein, The fusion classification in step 2 comprises: The extracted feature vectors are spliced: ; input F into a Kolmogorov-Arnold Network that outputs a predicted flow matrix; the shape of KAN is represented by an integer array: ; where is the number of nodes in the layer of the computational graph, and represents the activation value of the th neuron in the th layer, which is calculated as follows: ； ； ； ; wherein, is the activation value of the i-th neuron of the j-th layer, is the activation value of the i-th neuron of the j-th layer, is the activation value of the i-th neuron of the j-th layer, is the activation value of the i-th neuron of the j-th layer, is the activation value of the i-th neuron of the j-th layer, is the activation function connecting the i-th neuron of the j-th layer and the i-th neuron of the j+1-th layer, is the activation function connecting the i-th neuron of the j-th layer and the i-th neuron of the j+1-th layer, is the activation function connecting the i-th neuron of the j-th layer and the i-th neuron of the j+1-th layer, is the activation function connecting the i-th neuron of the j-th layer and the i-th neuron of the j+1-th layer, is the activation function connecting the i-th neuron of the j-th layer and the i-th neuron of the j+1-th layer, is a mapping function mapping the input , i.e. the previous layer neuron output, to a certain value, is the activation part weight, is the activation part weight, is a linear combination of a plurality of B-spline basis functions is a linear combination of a plurality of B-spline basis functions is a linear combination of a plurality of B-spline basis functions is a linear combination of a plurality of B-spline basis functions is a linear combination of a plurality of B-spline basis functions is a linear combination of a plurality of B-spline basis functions is a linear combination of a plurality of B-spline basis functions ; wherein, represents function composition, the input feature F is mapped through multiple layers of function matrix, and high-order abstract representation is obtained layer by layer, and finally the traffic prediction of different network slice types flowing through the network node at a time step is obtained. 7.The intelligent traffic orchestration method of claim 1, wherein, The reinforcement learning environment modeling in step 3 comprises: The resource scheduling problem under the multi-resource node is modeled as a partially observable distributed multi-agent Markov decision process, and is defined by the following seven-tuple: ; wherein, is a set of agents, and corresponds to a resource node in the system, each node being an agent, is a global environment state space, is an action space of the agent , is a state transition probability function, is a reward of the agent , is a discount factor, used to measure the weight of future rewards; For each resource node , the state space is designed as: ; wherein, is the resource requested by slice m at time t-1, is the node allocated to m, is the resource adjustment action history, including increase, decrease, maintain, is the node resource capacity, i.e. the constraint on the node ; Each resource agent selects the direction of adjusting resources for three service slices, including increasing, decreasing, and maintaining, that is: ; by mapping function , the action number is mapped to: 。 8.The intelligent traffic orchestration method of claim 1, wherein, The agent initialization and interaction in step 4 comprises: Initialization: Initialize a main network for each agent and a target network , the main network It is used to calculate the action Q value and select the action according to the current state. Used to evaluate the long-term benefits of the action to provide a stable training signal; at the same time, initialize an experience replay pool Used to store the agent's interaction experience data and initialize parameters ; The agent obtains the state of each resource type on each resource node The Q value of each optional action is calculated by forward propagation, and the calculation formula is:​​ ; where, is the current state value, is the extracted advantage of each action, which is output by the neural network and does not directly depend on the reward or the environment, but is learned through backpropagation during the training process, denotes the selected action, denotes all actions in the action space, which is used to normalize the advantage function, represents the average of the advantages of all actions; The main network adopts Strategy to select action: select action with probability Randomly select action with probability Select the action with the maximum current estimated Q value, and the action space is , the mapping function Convert the action to the slice resource adjustment amount, and directly select the maximum Q value action during deployment; According to the action decision , update the resource allocation, as follows: ; wherein denotes the updated estimate, denotes the estimated resource share of the slice at time step is assigned to the slice of service type, refers to the slices eMBB, URLLC, mMTC, is the adjustment step size for controlling the magnitude of the allocated resources; is the number of slices, refers to the mapping of the action ; Each agent aims to satisfy the quality of service requirement of the business, minimize resource waste and congestion, and is designed with the following reward at time step , the immediate reward of node is: ;in, For different reward weights, is the estimated resource allocated to slice m, is the demand for slice m, For the Slice-like service delay, For slices Unsatisfied resource requests, is a small value to avoid division by zero error, E is the unit benefit brought by unit resources, and N is the total number of nodes, which is used as a resource fairness indicator. The total amount of resources obtained for slice m; The interaction experience is saved to the experience pool: ; wherein, is an environment state of a resource node , is an action space of a node , is an immediate reward of a node , is a new environment state of a resource node . 9.The intelligent traffic orchestration method of claim 1, wherein, The expert knowledge fusion and network training in step 5 comprises: A training data set is constructed using historical optimal decision data, the data set containing historical states and corresponding expert decision actions; the expert policy network is trained using cross-entropy loss: ; where, is a historically observed environment state, is a historically expert-chosen action, is an expert policy network to be trained, with value being the probability that the policy network outputs action in state B experiences are randomly sampled from the experience pool D for updating the main network Q value, and the target Q value is calculated for each experience, and the specific formula is: ; wherein, is the state performs an action obtained immediate reward, is the Q-value vector for all future possible actions ; is the action with the maximum Q-value in the Q-value vector predicted by the main network; is the target network computes the Q-value corresponding to the action, is the "current reward + discounted future reward", and the Q-network is trained such that is maximized. The loss function calculation formula is: ; this formula measures the gap between the current Q-network output and the target value is the batch size of the sampled experience pool; is the target Q-value, is the mean squared error used to let the main network's predicted Q-values approximate the "target Q-values" as closely as possible, minimizing through backpropagation;​ Adding an expert guidance loss to let the policy of the Q network approach the expert policy with the KL divergence: ; where, denotes all actions in the action space, denotes the state of the node at time step , is the probability of the expert network choosing action in state , is the probability of the policy derived from the Q network choosing action in state , denotes the difference between the policy distribution of the Q network and the expert distribution in state , is the average value of the KL divergence between the expert policy and the Q policy calculated by randomly sampling B experiences from the experience pool . The final total loss function is obtained : ; ; wherein, and are constants, is a weight factor, is a dynamic weight, which decreases with the increase of time , is large in the early stage and gradually decreases in the later stage. 10.The intelligent traffic orchestration method of claim 1, wherein, The optimal policy generation in step 6 comprises: By repeatedly performing the interaction process in step (4) and the training process in step (5), the policy of the agent is continuously optimized until the preset convergence condition; after the policy converges, the parameters of the main Q network and the expert policy network are fixed, and the trained intelligent model is obtained. In the deploy application phase, given the current observation , weights , Q network computation , expert network computation Perform action selection: ; from a multitude of actions is a traffic orchestration policy.​

Images