Network steady-state flow regulation and control method combining ARIMA model and DQN

Abstract

The invention discloses a network steady-state flow regulation and control method combining an ARIMA model and a DQN, and the method comprises the steps: inputting real-time flow data obtained from a network into an ARIMA time sequence model based on historical data, and predicting the future flow; and inputting a prediction result into the DQN, and performing regulation and control output including dynamic optimization routing selection and flow distribution. According to the invention, the traffic prediction model based on ARIMA is integrated with the DQN intelligent agent, so that active routing decision is realized, and the network can dynamically optimize the traffic path under the condition of considering the current and predicted congestion.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning in network security and artificial intelligence, and in particular to a network steady-state traffic control method combining an ARIMA model and DQN. Background Art

[0002] In modern communication systems, steady-state regulation of network traffic is critical to maintaining system stability and operational efficiency. By balancing and maintaining consistent data traffic, steady-state regulation can effectively avoid sudden increases or decreases in traffic, and prevent congestion or service interruptions caused by traffic fluctuations. A key advantage of steady-state regulation is its ability to dynamically allocate bandwidth based on real-time traffic conditions, ensuring that high-priority applications receive the resources they need while avoiding overloading the network. This adaptive mechanism not only reduces latency, but also significantly improves the quality of service for latency-sensitive applications such as video calls, online games, and real-time data analysis.

[0003] At present, steady-state regulation has been widely used in many fields, especially in data-heavy environments such as smart cities, IoT ecosystems, and large-scale cloud services, which can effectively prevent data bottlenecks and ensure a smooth user experience. For example, research by Trivedi et al. shows that steady-state regulation can improve overall network efficiency by reducing the risk of packet loss and optimizing resource utilization. In addition, by optimizing network performance and avoiding expensive infrastructure expansion to cope with traffic growth, steady-state regulation can also reduce operating costs, thereby achieving more sustainable network development. Moreover, during periods of low traffic, the network can adjust the operating intensity according to actual needs, thereby effectively reducing energy consumption.

[0004] Therefore, as the network scale expands and traffic grows, the computational complexity of the control plane increases exponentially, and the network stability of steady-state regulation also faces greater challenges. Song et al. pointed out that adapting to changes in network conditions and efficiently managing congestion and link overloads are still important challenges for steady-state regulation. To address these issues, the network needs to continuously and actively monitor latency and throughput to ensure that data packets can be quickly diverted to less congested links. However, current routing algorithms have limitations in convergence and predicting changes in network traffic, making it difficult to meet high-demand application scenarios such as military defense.

[0005] Existing technologies have proposed some optimization schemes. For example, Hyun et al. introduced the concept of self-driving networks, which directly collect and analyze telemetry data through the network data plane, enabling the network to make routing decisions autonomously. At the same time, reinforcement learning (RL) technology has also received widespread attention in routing optimization. The reinforcement learning method proposed by Boyan et al. gradually optimizes the routing strategy by allowing the agent to learn the mapping from state to action to maximize the numerical reward. Khodayari et al. further proposed a Q-routing method to reduce the average transmission time. However, one of the main limitations of RL technology is that it converges slowly and takes a long time to reach the optimal strategy, which makes it difficult to deploy in multi-scale and nearly infinite state networks.

[0006] However, the existing technologies have several shortcomings in traffic prediction and route optimization. First, although the use of mixed superposition neural network models (such as LSTM and fully connected layers) for traffic prediction can capture some complex patterns, it requires huge computing resources and takes a long time to train. In particular, it may face the problem of gradient vanishing when processing high-dimensional state space. At the same time, the prediction accuracy of the LSTM model may be affected when processing non-stationary time series data, resulting in certain errors. In addition, when performing route optimization based on deep reinforcement learning (such as the DDPG algorithm), the algorithm relies on a continuous action space, which limits its application in discrete action problems, and is relatively slow in exploration and convergence speed, making it difficult to quickly adapt to changes in dynamic network environments. Moreover, the DDPG algorithm fails to effectively consider the deviation between predicted traffic and actual traffic, resulting in unsatisfactory optimization effects when facing real-time traffic fluctuations.

[0007] On the other hand, the traffic prediction method based on the entropy weight method also has certain limitations. This method relies on historical data for weighted calculations and cannot flexibly respond to rapid changes in traffic, resulting in poor real-time performance. Moreover, due to the high sensitivity of the entropy weight method to factor weights, it may affect the stability of the prediction results. In terms of routing optimization, although the solution traversal method can simulate different networking optimization schemes, its computational complexity is high, especially when the network scale increases, the efficiency decreases significantly. In addition, the lack of support for reinforcement learning mechanism makes it impossible to interactively adjust the optimization process in real time, reducing the overall optimization efficiency and adaptability. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a network steady-state traffic control method combining the ARIMA model and DQN.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] A first aspect of the present invention provides a network steady-state traffic control method combining an ARIMA model and DQN, comprising:

[0011] The real-time traffic data obtained from the network is input into the ARIMA time series model based on historical data to predict future traffic;

[0012] The prediction results are input into DQN for control output including dynamic optimization of routing selection and traffic allocation.

[0013] Further, the real-time traffic data is from network components including mobile devices, security servers, service access points or base stations;

[0014] The real-time traffic data first undergoes utility evaluation, including basic analysis of data value, importance, or network status.

[0015] Furthermore, the implementation of the ARIMA time series model includes:

[0016] S101: Determine the difference order d:

[0017] By performing a difference transformation on the original time series of historical data used in the ARIMA model training phase, the difference original time series of the historical data is obtained, and the trend or periodic fluctuation is eliminated to keep the mean and variance stable; the difference calculation formula is:

[0018] ΔX t =X t -X t-1 .#(1)

[0019] In the formula, X t Represents the sequence value of the original time series at time t, X t-1 Represents the sequence value of the original time series at time t-1, ΔX t Represents the difference result value of the original time series at time t; after multiple differences, if the data is still not stable, continue to perform second-order or higher-order differences; use the unit root test to determine the stability of the data. If the series passes the test, it is considered that the difference order d is sufficient;

[0020] S102: Analyze the ACF and PACF characteristics, use the autocorrelation function ACF and the partial autocorrelation function PACF to analyze the differenced historical data, and calculate the appropriate values ​​of the parameter p as the autoregressive order and the parameter q as the sliding average order in the ARIMA time series model:

[0021] ACF measures the correlation between a time series and its lagged values ​​and is defined as follows:

[0022]

[0023] in is the mean of the original time series, N is the total number of observations, ΔX t+k Represents the sequence value of the differenced original time series at lag k;

[0024] PACF measures the direct correlation between the data and its lagged value, excluding the influence of intermediate terms; PACF is calculated using the Yule-Walker equation and is defined as:

[0025] p=PACF(k)=φ k,k #(3)

[0026] where φ k,k is the partial autocorrelation coefficient obtained by solving the Yule-Walker equation, where the Yule-Walker equation is as follows:

[0027]

[0028] In the formula, φ k represents the k-order partial autocorrelation coefficient, is the i-th order autoregressive coefficient, γ i represents the i-th order autocorrelation coefficient;

[0029] S103: Determine the structure of the ARIMA time series model based on the characteristics of ACF and PACF:

[0030] If PACF is truncated at lag p and ACF gradually decays, the ARIMA time series model is AR(p); if ACF is truncated at lag q and PACF gradually decays, the ARIMA time series model is MA(q); if both decay, the ARIMA time series model is ARMA(p,q); where truncation means that after a certain lag order, the correlation coefficient suddenly becomes close to 0; decay means that the correlation coefficient gradually decreases, showing an exponential decline, including the occurrence of alternating positive and negative patterns; it will not suddenly become 0, but gradually approach 0;

[0031] S104: Use model parameters p, difference orders d and q to construct an ARIMA time series model:

[0032] The ARIMA time series model represents the lag term as the past p observations D t-1 ,D t-2 ,…,D t-p and q past white noise terms e t-1 ,e t-2 ,…,e t-q The linear combination of and adding random errors is expressed as follows:

[0033] Dt =c+φ1D t-1 +…+φ p D t-p +e t +θ1e t-1 +…+θ q e t-q #(5)

[0034] Where D t is the data calculated based on the historical data after difference processing, c represents the function constant term; φ is the coefficient of the AR part, which corresponds to the k-order partial autocorrelation coefficient in step S102, and is obtained by solving the p-order Yule-Walker equation; θ is the coefficient of the MA part, which corresponds to the observation value, and is obtained by maximizing the conditional likelihood function; e t is a random error;

[0035] When the model is AR(p), θ is set to 0; when the model is MA(q), φ is set to 0;

[0036] S105: Residual Analysis:

[0037] The residuals of the ARIMA time series model established in step S104 are analyzed to verify whether they conform to the white noise characteristics:

[0038] S10501: Residual calculation formula: Among them, y t is the actual value, i.e. the current observed value of the real-time time series, is the predicted value of the ARIMA time series model, i.e. D t ;

[0039] S10502: mean test: the residual mean in step S10501 should be close to zero; residual mean = (e1+e+...+et) / t;

[0040] S10503: Autocorrelation test: Check whether there is significant autocorrelation in the residuals in step S10501 to ensure that the model has captured all the information of the time series:

[0041]

[0042] In the formula, ei represents the residual at time i, represents the mean of the residual sequence, t represents the total number of samples, and k represents the lag order;

[0043] S10504: Variance stability: The variance of the residual in step S10501 should remain stable throughout the series;

[0044] S106: Optimize the ARIMA time series model by goodness of fit and residual characteristics, and select the model with the smallest residual and satisfying the white noise characteristics as the final ARIMA time series model;

[0045] S107: Use the final ARIMA time series model in step S106 to predict future traffic values ​​based on the real-time data of the time series, including link traffic levels and global traffic distribution. These results are output in the form of a traffic matrix, where each element represents the predicted traffic size between two nodes.

[0046] Furthermore, inputting the prediction result into DQN includes:

[0047] The traffic matrix output by the ARIMA model as the prediction result is combined with the network topology information to form the state space input of the DQN agent.

[0048] Furthermore, the network structure of the DQN includes:

[0049] Input layer: receives state space data consisting of traffic matrix and network topology information, indicating the real-time traffic distribution and link status of the current network;

[0050] Hidden layer: Use the fully connected layer to extract high-dimensional state features and provide the necessary deep feature expression capabilities for decision-making;

[0051] Output layer: corresponds to the action space of the intelligent agent, and each output node represents the Q value of a routing adjustment action.

[0052] Furthermore, the core of the DQN agent includes:

[0053] Undirected graph: The network topology operated by the DQN agent is represented as an undirected graph G(V,E,C), where the node set V represents the network nodes, the edge set E represents the links, and the capacity set C represents the bandwidth capacity of the links;

[0054] State space: The state space (State,s) of the agent is represented as an n×n traffic matrix, which reflects the real-time traffic distribution between network nodes and provides key network state information for the agent's decision-making;

[0055] Action space: The action space (Action,a) of the agent is defined as the link weight vector. The agent influences the routing path of traffic by adjusting the link weight, thereby optimizing network performance and resource allocation.

[0056] Reward function: The agent evaluates the effect of the action through the reward function (Reward, r), taking into account the data rate The network performance indicators of network delay L and packet loss rate PL; delay L and packet loss rate PL are calculated by the following formulas respectively:

[0057]

[0058] The final reward function r is defined as the following formula to maximize the network performance:

[0059]

[0060] Among them, d i,j is the transmission delay of link (i, j) on the path, where P lost is the number of lost packets, P total is the total number of packets sent, is the average data rate of network transmission, α, β, γ∈[0,1] are weights that control the contribution of each indicator in the reward function.

[0061] Furthermore, the DQN optimization mechanism includes:

[0062] Experience replay: DQN stores historical decision data in the experience replay buffer and randomly extracts samples for training; this method reduces the impact of data correlation on the model and improves the stability and efficiency of training;

[0063] Loss function: In DQN, the loss function is defined using mean square error (MSE) to measure the difference between the Q value generated by the Q network and the target Q value; the loss function is defined as:

[0064] Loss=E[(yQ(s t ,a t ))2],#(9)

[0065] The target Q value y is calculated as follows:

[0066]

[0067] Q-value update: The agent updates the Q-value using the Q-learning algorithm; it iteratively optimizes the Q-value after performing actions, observing state changes, and receiving rewards according to the following formula:

[0068]

[0069] where α t is the learning rate, i.e., controls the learning step size, γ is the discount factor, s t is the current state, a t is the current action, r t is the reward value, s t+1 For the next state.

[0070] Further, the specific operation steps of DQN are:

[0071] S201: Initialization:

[0072] Initialize the parameters θ of the Q network and the target network Q and θ' Q , and create an experience replay buffer to store state, action, reward, and next state data; the structure and parameters of the Q network and the target network are exactly the same, θ Q and θ' Q Represent the parameter sets of the Q network and the target network respectively, including: weight w1 and bias b1 from the input layer to the hidden layer, weight w2 and bias b2 between hidden layers, weight w3 and bias b3 from the hidden layer to the output layer;

[0073] S202: Status collection and decision making:

[0074] The DQN agent receives the flow matrix and network topology output by the ARIMA time series model to form the state s t , calculate the Q value Q(s) of all actions through the Q network t ,a), and use the ∈-greedy strategy to select action a t : Select the action with the largest Q value with probability 1-∈

[0075] S203: Execute actions and calculate rewards:

[0076] Perform action a in the environment t , adjust the routing path and observe the next state s t+1 ; Calculate the instant reward r based on delay, packet loss rate and link utilization t ;

[0077] S204: Experience Storage:

[0078] The experience (s t ,a t ,r t ,s t+1 ) into the playback buffer;

[0079] S205: Network training:

[0080] Randomly draw a small batch of samples (s t ,a t ,r t ,s t+1 ), calculate the value of the loss function L according to formula (9), calculate the target Q value according to formula (10), and finally update the Q value to θ according to formula (11) Q ;

[0081] S206: Update target network:

[0082] In the above training process, only the parameters θ of the Q network are updatedQ ,θ' Q remain unchanged; after T training steps, the parameters θ of the Q network are Q Copy to the target network θ' Q , to stabilize the training process.

[0083] The beneficial effects of the present invention are:

[0084] An exemplary embodiment of the present invention implements active routing decisions by integrating an ARIMA-based traffic prediction model with a DQN agent, so that the network can dynamically optimize traffic paths taking into account current and predicted congestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A flowchart of a network steady-state traffic control method combining an ARIMA model and DQN provided in an exemplary embodiment of the present invention is shown;

[0086] Figure 2 A schematic diagram showing an implementation of an ARIMA time series model provided in an exemplary embodiment of the present invention is shown;

[0087] Figure 3 A schematic diagram of truncation and decay in an ARIMA time series model provided in an exemplary embodiment of the present invention is shown;

[0088] Figure 4 A schematic diagram of a network structure of a DQN provided in an exemplary embodiment of the present invention is shown;

[0089] Figure 5 A schematic diagram showing specific operation steps of a DQN provided in an exemplary embodiment of the present invention is shown;

[0090] Figure 6 A schematic diagram showing the influence of different behavior space sizes in the process of training a DQN agent provided in an exemplary embodiment of the present invention is shown;

[0091] Figure 7 A schematic diagram showing the influence of changes in different reward function parameters α, β and γ during the training of a DQN agent provided in an exemplary embodiment of the present invention is shown;

[0092] Figure 8 FIG. 4 is a schematic diagram showing a comparison of experimental results provided in an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0093] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0094] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0095] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0096] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0097] See also Figure 1 , Figure 1 A network steady-state traffic control method combining an ARIMA model and DQN provided in an exemplary embodiment of the present invention is shown, comprising:

[0098] The real-time traffic data obtained from the network is input into the ARIMA time series model based on historical data to predict future traffic;

[0099] The prediction results are input into DQN for control output including dynamic optimization of routing selection and traffic allocation.

[0100] Specifically, in this exemplary embodiment, real-time traffic data is first collected from the network, and the ARIMA time series model is used to analyze and predict future traffic. A DQN-based agent is introduced to monitor and adjust network resource allocation in real time. The DQN agent interacts with the network environment through reinforcement learning to dynamically optimize routing selection and traffic allocation.

[0101] This exemplary embodiment implements proactive routing decisions by integrating an ARIMA-based traffic prediction model with a DQN agent, enabling the network to dynamically optimize traffic paths taking into account current and predicted congestion.

[0102] The ARIMA model shows remarkable efficiency and prediction accuracy in traffic forecasting. Through time series analysis of historical data, the ARIMA model can accurately capture the trend and periodic changes of traffic, providing reliable support for short-term forecasting. Compared with the hybrid superposition neural network model (LSTM+fully connected layer) and the prediction method based on entropy weight method in the prior art, ARIMA has better performance in processing non-stationary data and identifying periodicity, and is particularly suitable for scenarios of rapid deployment and lightweight computing. In addition, the ARIMA model parameters can be intuitively determined by statistical methods, and the construction and interpretation are simple and easy to understand, meeting the requirements for high logic and causality of the prediction results. Compared with the complex neural network training in the prior art and the entropy weight method calculation based on weighted factors in the prior art, the ARIMA model is not only faster to deploy but also has lower implementation costs. At the same time, the ARIMA model shows strong reliability in diversified traffic patterns, especially in predicting traffic bottlenecks, providing more accurate support for subsequent routing optimization.

[0103] In addition, DQN demonstrates excellent real-time and adaptive capabilities in routing optimization. Through dynamic interaction between reinforcement learning and the network environment, DQN can quickly adjust routing weights and resource allocation to adapt to rapid changes in traffic. Compared with the DDPG algorithm used in existing technologies, DQN does not rely on continuous action space, the training process is more stable, and it is more efficient in high-dimensional state space; at the same time, compared with the networking optimization method based on solution traversal, DQN can reduce computing time through real-time decision-making and greatly improve response speed.

[0104] The following contents will describe various preferred exemplary embodiments in detail:

[0105] More preferably, in an exemplary embodiment, the real-time traffic data is from a network component including a mobile device, a security server, a service access point, or a base station;

[0106] The real-time traffic data first undergoes utility evaluation, including basic analysis of data value, importance, or network status.

[0107] Specifically, in this exemplary embodiment, real-time traffic data is collected from the network, and these data come from network components such as other mobile devices, security servers, service access points or base stations; at the same time, the collected network traffic data will first undergo utility evaluation, which may include basic analysis of data value, importance or network status to provide a reference for subsequent prediction and control.

[0108] More preferably, in an exemplary embodiment, the ARIMA model detects periodicity and trend patterns in time series data through a combination of autoregression (AR), differencing (I) and moving average (MA), providing high-precision traffic forecasting support for the next step of network regulation.

[0109] The implementation of the ARIMA time series model (first calculation and output). Figure 2 As shown, including:

[0110] S101: Determine the difference order d:

[0111] By performing a difference transformation on the original time series of historical data used in the ARIMA model training phase, the difference original time series of the historical data is obtained, and the trend or periodic fluctuation is eliminated to keep the mean and variance stable; the difference calculation formula is:

[0112] ΔX t =X t -X t-1 .#(1)

[0113] In the formula, X t Represents the sequence value of the original time series at time t, X t-1 Represents the sequence value of the original time series at time t-1, ΔX t Represents the difference result value of the differenced original time series at time t; after multiple differences, if the data is still not stable, continue to perform second-order or higher-order differences; use a unit root test (such as an ADF test) to determine the stationarity of the data. If the sequence passes the test, it is considered that the difference order d is sufficient, and d is usually 0, 1, or 2.

[0114] S102: Analyze the ACF and PACF characteristics, use the autocorrelation function ACF and the partial autocorrelation function PACF to analyze the differenced historical data, and calculate the appropriate values ​​of the parameter p as the autoregressive order and the parameter q as the sliding average order in the ARIMA time series model:

[0115] ACF measures the correlation between a time series and its lagged values ​​and is defined as follows:

[0116]

[0117] in is the mean of the original time series, N is the total number of observations, ΔX t+k Represents the sequence value of the differenced original time series at lag k;

[0118] PACF measures the direct correlation between the data and its lagged value, excluding the influence of intermediate terms; PACF is calculated using the Yule-Walker equation and is defined as:

[0119] p=PACF(k)=φ k,k #(3)

[0120] where φ k,k is the partial autocorrelation coefficient obtained by solving the Yule-Walker equation, where the Yule-Walker equation is as follows:

[0121]

[0122] In the formula, φ k represents the k-order partial autocorrelation coefficient, is the i-th order autoregressive coefficient, γ i represents the i-th order autocorrelation coefficient.

[0123] In a specific exemplary embodiment, the parameters in the PACF are obtained by solving the Yule-Walker equation (as shown in the figure), wherein the calculation algorithm and process are as follows (python):

[0124] Algorithm initial condition setting:

[0125]

[0126]

[0127] Recursive calculation process:

[0128]

[0129] S103: Determine the structure of the ARIMA time series model based on the characteristics of ACF and PACF:

[0130] If PACF is truncated at lag p and ACF gradually decays, the ARIMA time series model is AR(p); if ACF is truncated at lag q and PACF gradually decays, the ARIMA time series model is MA(q); if both decay, the ARIMA time series model is ARMA(p,q);

[0131] It should be noted that see Figure 3 , definition of truncation: after a certain lag order k, the correlation coefficient suddenly becomes close to 0; mathematical definition: the correlation coefficient after k orders falls within the confidence interval of ±2 / √n, where n is the sample size. Definition of decay: the correlation coefficient gradually decreases, usually showing an exponential decline; a positive and negative alternating pattern may appear; it will not suddenly become 0, but asymptotically approach 0.

[0132] S104: Use model parameters p, difference orders d and q to construct an ARIMA time series model:

[0133] Mathematically, the ARIMA time series model represents the lag term as the past p observations D t-1 ,D t-2 ,…,D t-p and q past white noise terms e t-1 ,e t-2 ,…,e t-q The linear combination of and adding random errors is expressed as follows:

[0134] D t =c+φ1D t-1 +…+φ p D t-p +e t +θ1e t-1 +…+θ q e t-q #(5)

[0135] Where D t is the data calculated based on the historical data after difference processing, c represents the function constant term (representing the average level or benchmark value of the time series); φ is the coefficient of the AR part, which corresponds to the k-order partial autocorrelation coefficient in step S102, and is obtained by solving the p-order Yule-Walker equation; θ is the coefficient of the MA part, which corresponds to the observation value and is obtained by maximizing the conditional likelihood function; e t is a random error;

[0136] When the model is AR(p), θ is set to 0; when the model is MA(q), φ is set to 0.

[0137] It should be noted that formula (5) is a complete ARMA model, while for the AR(p) model, D t =c+φ1D t-1 +…+φ p D t-p +e t ; For the MA(q) model, D t =c+e t +θ1e t-1 +…+θ q e t-q .

[0138] At the same time, for the parameter θ, the q value can be determined according to the ACF of S102 and then the conditional likelihood function is constructed based on the given data sequence {D1, D2, ..., Di}:

[0139]

[0140] By using numerical optimization methods (such as the Newton-Raphson method) to maximize this likelihood function, the value of θ is obtained:

[0141]

[0142] S105: Residual Analysis:

[0143] The residuals of the ARIMA time series model established in step S104 are analyzed to verify whether they conform to the white noise characteristics:

[0144] S10501: Residual calculation formula: Among them, y t is the actual value, i.e. the current observed value of the real-time time series, is the predicted value of the ARIMA time series model, i.e. D t ;

[0145] S10502: mean test: the residual mean in step S10501 should be close to zero (|residual mean|<0.01); residual mean = (e1+e+...+et) / t;

[0146] S10503: Autocorrelation test: Check whether there is significant autocorrelation in the residuals in step S10501 to ensure that the model has captured all the information of the time series:

[0147]

[0148] In the formula, ei represents the residual at time i, represents the mean of the residual sequence, t represents the total number of samples and k represents the lag order; the significance judgment standard is: n is the sample size,

[0149] S10504: Variance stability: The variance of the residuals in step S10501 should remain stable throughout the series; for example, using the p_levene test, a p-value > 0.05 indicates variance stability.

[0150] In a specific exemplary embodiment, p_levene is used, and the calculation process is:

[0151] Step 1: Data preparation ① The residual sequence {e1,e2,...,e t} divided into k groups②The sample size of each group is n i ③Total sample size N = ∑n i ;

[0152] Step 2: Calculate the deviation

[0153] Among them: ①e iis the jth observation value of the i-th group② is the mean (or median) of the i-th group;

[0154] Step 3: Calculation of test statistic W:

[0155]

[0156] in:

[0157] ① is the mean deviation of group i

[0158] ② is the overall mean of all deviations

[0159] Step 4: p_levene value calculation:

[0160] W follows an F distribution with degrees of freedom (k-1, Nk)

[0161] p_levene=P(F>W)

[0162] in:

[0163] ①F is a random variable with F distribution

[0164] ②W is the calculated test statistic

[0165] Step 5: Result judgment: p_levene>0.05 indicates that the variance is stable.

[0166] S106: Optimize the ARIMA time series model by goodness of fit and residual characteristics, and select the model with the smallest residual and satisfying the white noise characteristics as the final ARIMA time series model;

[0167] Specifically, the goodness of fit AIC is: AIC = 2k-2ln(L), where:

[0168] ①k=p+q+1(number of parameters); ②L is the likelihood function: ③σ 2 is the residual variance;

[0169] Optimization steps:

[0170] Step 1: Try different parameter combinations: ① change the p value (0 to pmax); ② change the q value (0 to qmax); ③ fix the d value (determined by the difference step);

[0171] Step 2: For each set of parameters (p, d, q): ① calculate the AIC value; ② perform residual tests: mean test (|mean|<0.01); autocorrelation test (|ρk|≤2 / √n); variance stability test (p_levene>0.05).

[0172] Step 3: Screening models: ① First, meet all the conditions of the residual test; ② Then select the model with the smallest AIC among the models that meet the conditions.

[0173] S107: Use the final ARIMA time series model in step S106 to predict future traffic values ​​based on the real-time data of the time series, including link traffic levels and global traffic distribution. These results are output in the form of a traffic matrix, where each element represents the predicted traffic size between two nodes, providing a basis for subsequent routing optimization.

[0174] More preferably, in an exemplary embodiment, inputting the prediction result into the DQN comprises:

[0175] The traffic matrix output by the ARIMA model as the prediction result is combined with the network topology information to form the state space input of the DQN agent.

[0176] Specifically, in this exemplary embodiment, by analyzing the future traffic predicted by the ARIMA model, the system can detect possible bottlenecks in advance so that corresponding control measures can be taken. Specifically, the traffic matrix provided by the ARIMA model is combined with the network topology information to form the state space input of the agent. At each decision iteration, the DQN agent analyzes the traffic information in the state space, identifies the bottleneck location and adjusts the traffic allocation strategy, thereby dynamically optimizing network performance. This process ensures that when potential congestion occurs in the network, DQN can make real-time adjustments, optimize traffic routing, and ensure the smooth operation of the network.

[0177] Preferably, in an exemplary embodiment, a DQN-based agent is introduced to monitor and adjust network resource allocation in real time. The DQN agent interacts with the network environment through reinforcement learning to dynamically optimize routing selection and traffic allocation.

[0178] The network structure of the DQN is as follows: Figure 4 As shown, including:

[0179] Input layer: receives state space data consisting of traffic matrix and network topology information, indicating the real-time traffic distribution and link status of the current network;

[0180] Hidden layer: Use the fully connected layer to extract high-dimensional state features, providing the necessary deep feature expression capabilities for decision-making;

[0181] Output layer: corresponds to the action space of the intelligent agent, and each output node represents the Q value of a routing adjustment action.

[0182] More preferably, in an exemplary embodiment, the core of the DQN agent includes:

[0183] Undirected graph: The network topology operated by the DQN agent is represented as an undirected graph G(V,E,C), where the node set V represents the network nodes, the edge set E represents the links, and the capacity set C represents the bandwidth capacity of the links;

[0184] State space: The state space (State,s) of the agent is represented as an n×n traffic matrix, which reflects the real-time traffic distribution between network nodes and provides key network state information for the agent's decision-making;

[0185] Action space: The action space (Action,a) of the agent is defined as the link weight vector. The agent influences the routing path of traffic by adjusting the link weight, thereby optimizing network performance and resource allocation.

[0186] Reward function: The agent evaluates the effect of the action through the reward function (Reward, r), taking into account the data rate The network performance indicators of network delay L and packet loss rate PL; delay L and packet loss rate PL are calculated by the following formulas respectively:

[0187]

[0188] The final reward function r is defined as the following formula to maximize the network performance:

[0189]

[0190] Among them, d i,j is the transmission delay of link (i, j) on the path, where P lost is the number of lost packets, P total is the total number of packets sent, is the average data rate of network transmission, α, β, γ∈[0,1] are weights that control the contribution of each indicator in the reward function.

[0191] More preferably, in an exemplary embodiment, the DQN optimization mechanism includes:

[0192] Experience replay: DQN stores historical decision data in the experience replay buffer and randomly extracts samples for training; this method reduces the impact of data correlation on the model and improves the stability and efficiency of training;

[0193] Loss function: In DQN, the loss function is defined using mean square error (MSE) to measure the difference between the Q value generated by the Q network and the target Q value; the loss function is defined as:

[0194] Loss=E[(yQ(s t ,a t )) 2 ],#(9)

[0195] The target Q value y is calculated as follows:

[0196]

[0197] Q-value update: The agent updates the Q-value using the Q-learning algorithm; it iteratively optimizes the Q-value after performing actions, observing state changes, and receiving rewards according to the following formula:

[0198]

[0199] where α t is the learning rate, i.e., controls the learning step size, γ is the discount factor, s t is the current state, a t is the current action, r t is the reward value, s t+1 For the next state.

[0200] Preferably, in an exemplary embodiment, the specific operation steps of DQN are as follows: Figure 5 As shown:

[0201] S201: Initialization:

[0202] Initialize the parameters θ of the Q network and the target network Q and θ' Q , and create an experience replay buffer to store state, action, reward, and next state data; the structure and parameters of the Q network and the target network are exactly the same, θ Q and θ' Q Represent the parameter sets of the Q network and the target network respectively, including: weight w1 and bias b1 from the input layer to the hidden layer, weight w2 and bias b2 between hidden layers, weight w3 and bias b3 from the hidden layer to the output layer;

[0203] S202: Status collection and decision making:

[0204] The DQN agent receives the flow matrix and network topology output by the ARIMA time series model to form the state s t , calculate the Q value Q(s) of all actions through the Q network t ,a), and use the ∈-greedy strategy to select action a t : Select the action with the largest Q value with probability 1-∈

[0205] Specifically, the ∈-greedy strategy (python code) is implemented as follows:

[0206]

[0207] The probability description is:

[0208] The ε-greedy strategy contains two options: a. With a probability of 1-ε: select the action with the largest Q value (greedy); b. With a probability of ε: randomly select an action (exploration).

[0209] For example, if ε = 0.1: a. 90% probability of selecting the optimal action; b. 10% probability of random exploration.

[0210] In formula (10),

[0211]

[0212] Y is the target Q value, r is the immediate reward, γ is the discount factor, is the maximum Q value of the next state.

[0213] S203: Execute actions and calculate rewards:

[0214] Perform action a in the environment t , adjust the routing path and observe the next state s t+1 ; Calculate the instant reward r based on delay, packet loss rate and link utilization t .

[0215] Specifically, the definition of actions includes:

[0216] ①Action a t Indicates the decision to adjust the link weight: a) adjust the weight value of each link in the network; b) affect the selection of routing path; c) change the distribution of network traffic;

[0217] ②Execution process: a) DQN outputs action at (link weight adjustment plan); b) applies the new weight to the network; c) observes the changes in network status after execution: 1) new traffic distribution 2) network performance indicators 3) link load conditions;

[0218] ③State transition: s t (Current state) → Execute a t (Adjust link weight) →s t+1 (New state). Each state contains: a) traffic distribution matrix; b) network topology information.

[0219] ④ Result evaluation, i.e. step S203.

[0220] S204: Experience Storage:

[0221] The experience (s t ,a t ,r t ,s t+1 ) is stored in the playback buffer; the entire playback buffer includes s1, a1, r1, s 1+1 )~(s t ,a t ,r t ,s t+1 ).

[0222] S205: Network training:

[0223] Randomly draw a small batch of samples (s t ,a t ,r t ,s t+1 ), calculate the value of the loss function L according to formula (9), calculate the target Q value according to formula (10), and finally update the Q value to θ according to formula (11) Q .

[0224] Specifically, for a batch of samples (taking 32 samples as an example), first calculate the target Q value (y1 to y32) of each sample, then calculate the current Q value (Q1 to Q32) of each sample, and then calculate the overall loss L, and update the network parameters θ at one time. Q .

[0225] S206: Update target network:

[0226] In the above training process, only the parameters θ of the Q network are updated Q ,θ' Q remain unchanged; after T training steps, the parameters θ of the Q network are Q Copy to the target network θ' Q , to stabilize the training process.

[0227] In simple terms: First, the data is stored in a playback buffer, and then a small batch of data is randomly extracted from the buffer as the model training input. The Q value is determined using the Q network, the target value is determined using the target network, and the loss function is calculated based on these values. During training, only the parameters of the Q network are modified. After a specified number of training iterations, the weights of the Q network are transferred to the target network to maintain the stability of the target value.

[0228] It should be noted that this method adds a dynamically adjusted reward mechanism based on the advantages and disadvantages of existing reinforcement learning-based routing optimization, so that the DQN agent can adapt to the deviation between actual traffic and predicted traffic, thereby enhancing the real-time responsiveness and robustness of traffic routing.

[0229] Finally, based on the output of the DQN agent, specific traffic control operations are performed. This step can include path adjustment, resource reallocation, etc. to ensure network stability and improve resource utilization.

[0230] It can also include cyclic detection: after completing a round of detection and regulation, the system will enter the next traffic detection cycle and continue to optimize network traffic control in real time.

[0231] The following exemplary embodiments have conducted corresponding experiments to verify the actual effects of the above exemplary embodiments of the present invention. The specific contents of the experiments are as follows.

[0232] First, a network performance measurement module (for monitoring latency and collecting statistics) and an active forwarding module were implemented in ONOS, a Java-based OpenFlow controller. Subsequently, the DQN agent and traffic prediction module were developed in Python and encapsulated in a Docker container so that the DQN agent and traffic prediction module can interact with the active forwarding module through the northbound API of ONOS. To simulate traffic between hosts, we use the Iperf tool to establish flows between a set of hosts (h1,…,h8). The performance measurement module collects statistics (such as latency, throughput, and traffic size) from the devices and sends these time series data to the InfluxDB database at intervals of 5 seconds.

[0233] In terms of model construction and training, we built a DQN model using the TensorFlow library, which includes two neural networks: one for the Q network and the other for the Q target, both of which use the same architecture. Specifically, both the Q network and the Q target contain two fully connected layers. In order to analyze the impact of the action space size on performance (defined by the number of network configurations for each input traffic matrix state), we trained three different DQN agents with action space sizes of 36, 96, and 180 configurations. The input state vector size of each neural network is 48, and the corresponding output is consistent with the action space size (36, 96, and 180). Therefore, the training architectures are (48,36), (48,96), and (48,180), respectively. Figure 6 Shown is the effect of different action space sizes in training a DQN agent. Figure 7 Shown is the effect of varying the reward function parameters α, β, and γ during training of the DQN agent.

[0234] During training, the ε-greedy strategy was used for action selection, and the final exploration rate was set to 0.2. The Q target parameters were synchronized with the Q network every 300 steps, the learning rate was set to 0.01, and the discount factor was 0.95. Each DQN agent was trained for a total of 180 rounds.

[0235] To quantitatively evaluate the performance of the prediction model, the root mean square error (RMSE) is used. RMSE measures accuracy by calculating the square root of the average of the squared differences between the predicted values ​​and the observed values. The formula is as follows:

[0236]

[0237] in, and They represent the normalized predicted value and actual value at the same time interval, respectively, $N$ is the total number of predictions or the prediction interval P i , the size is set to 5 seconds.

[0238] This paper compares the network performance of ARIMA+DQN, independent DQN and Baseline methods under different methods. The results are as follows: Figure 8 As shown. It can be seen that in most scenarios, the ARIMA+DQN method of the present invention achieves excellent results. In processing network traffic optimization tasks, the present invention can effectively reduce packet loss rate and delay, and improve link utilization, significantly improving the overall performance of network traffic management.

[0239] Obviously, the above embodiments are merely examples for clear explanation, and are not intended to limit the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A network steady-state traffic control method combining ARIMA model and DQN, characterized by: include: The real-time traffic data obtained from the network is input into the ARIMA time series model based on historical data to predict future traffic; The prediction results are input into DQN for control output including dynamic optimization of routing selection and traffic allocation.

2. According to claim 1, a network steady-state traffic control method combining ARIMA model and DQN is characterized in that: The real-time traffic data is from network components including mobile devices, security servers, service access points or base stations; The real-time traffic data first undergoes utility evaluation, including basic analysis of data value, importance, or network status.

3. According to claim 1, a network steady-state traffic control method combining ARIMA model and DQN is characterized in that: The implementation of the ARIMA time series model includes: S101: Determine the difference order d: By performing a difference transformation on the original time series of historical data used in the ARIMA model training phase, the difference original time series of the historical data is obtained, and the trend or periodic fluctuation is eliminated to keep the mean and variance stable; the difference calculation formula is: ΔX t =X t -X t-1 .#(1) In the formula, X t Represents the sequence value of the original time series at time t, X t-1 Represents the sequence value of the original time series at time t-1, ΔX t Represents the difference result value of the original time series at time t; after multiple differences, if the data is still not stable, continue to perform second-order or higher-order differences; use the unit root test to determine the stability of the data. If the series passes the test, it is considered that the difference order d is sufficient; S102: Analyze the ACF and PACF characteristics, use the autocorrelation function ACF and the partial autocorrelation function PACF to analyze the differenced historical data, and calculate the appropriate values ​​of the parameter p as the autoregressive order and the parameter q as the sliding average order in the ARIMA time series model: ACF measures the correlation between a time series and its lagged values ​​and is defined as follows: in is the mean of the original time series, N is the total number of observations, ΔX t+k Represents the sequence value of the differenced original time series at lag k; PACF measures the direct correlation between the data and its lagged value, excluding the influence of intermediate terms; PACF is calculated using the Yule-Walker equation and is defined as: p=PACF(k)=φ k,k #(3) where φ k,k is the partial autocorrelation coefficient obtained by solving the Yule-Walker equation, where the Yule-Walker equation is as follows: In the formula, φ k represents the k-order partial autocorrelation coefficient, is the i-th order autoregressive coefficient, γ i represents the i-th order autocorrelation coefficient; S103: Determine the structure of the ARIMA time series model based on the characteristics of ACF and PACF: If PACF is truncated at lag p and ACF gradually decays, the ARIMA time series model is AR(p); if ACF is truncated at lag q and PACF gradually decays, the ARIMA time series model is MA(q); if both decay, the ARIMA time series model is ARMA(p,q); where truncation means that after a certain lag order, the correlation coefficient suddenly becomes close to 0; decay means that the correlation coefficient gradually decreases, showing an exponential decline, including the occurrence of alternating positive and negative patterns; it will not suddenly become 0, but gradually approach 0; S104: Use model parameters p, difference orders d and q to construct an ARIMA time series model: The ARIMA time series model represents the lag term as the past p observations D t-1 ,D t-2 ,…,D t-p and q past white noise terms e t-1 ,e t-2 ,…,e t-q The linear combination of and adding random errors is expressed as follows: D t =c+φ1D t-1 +…+φ p D t-p +e t +θ1e t-1 +…+θ q e t-q #(5) Where D t is the data calculated based on the historical data after difference processing, c represents the function constant term; φ is the coefficient of the AR part, which corresponds to the k-order partial autocorrelation coefficient in step S102, and is obtained by solving the p-order Yule-Walker equation; θ is the coefficient of the MA part, which corresponds to the observation value, and is obtained by maximizing the conditional likelihood function; e t is a random error; When the model is AR(p), θ is set to 0; when the model is MA(q), φ is set to 0; S105: Residual Analysis: The residuals of the ARIMA time series model established in step S104 are analyzed to verify whether they conform to the white noise characteristics: S10501: Residual calculation formula: Among them, y t is the actual value, i.e. the current observed value of the real-time time series, is the predicted value of the ARIMA time series model, i.e. D t ; S10502: mean test: the residual mean in step S10501 should be close to zero; residual mean = (e1+e+...+et) / t; S10503: Autocorrelation test: Check whether there is significant autocorrelation in the residuals in step S10501 to ensure that the model has captured all the information of the time series: In the formula, ei represents the residual at time i, represents the mean of the residual sequence, t represents the total number of samples, and k represents the lag order; S10504: Variance stability: The variance of the residual in step S10501 should remain stable throughout the series; S106: Optimize the ARIMA time series model by goodness of fit and residual characteristics, and select the model with the smallest residual and satisfying the white noise characteristics as the final ARIMA time series model; S107: Use the final ARIMA time series model in step S106 to predict future traffic values ​​based on the real-time data of the time series, including link traffic levels and global traffic distribution. These results are output in the form of a traffic matrix, where each element represents the predicted traffic size between two nodes.

4. According to claim 3, a network steady-state traffic control method combining ARIMA model and DQN is characterized in that: The step of inputting the prediction result into DQN includes: The traffic matrix output by the ARIMA model as the prediction result is combined with the network topology information to form the state space input of the DQN agent.

5. According to claim 4, a network steady-state traffic control method combining ARIMA model and DQN is characterized in that: The network structure of the DQN includes: Input layer: receives state space data consisting of traffic matrix and network topology information, indicating the real-time traffic distribution and link status of the current network; Hidden layer: Use the fully connected layer to extract high-dimensional state features and provide the necessary deep feature expression capabilities for decision-making; Output layer: corresponds to the action space of the intelligent agent, and each output node represents the Q value of a routing adjustment action.

6. A network steady-state traffic control method combining ARIMA model and DQN according to claim 5, characterized in that: The core of the DQN agent consists of: Undirected graph: The network topology operated by the DQN agent is represented as an undirected graph G(V,E,C), where the node set V represents the network nodes, the edge set E represents the links, and the capacity set C represents the bandwidth capacity of the links; State space: The state space (State,s) of the agent is represented as an n×n traffic matrix, which reflects the real-time traffic distribution between network nodes and provides key network state information for the agent's decision-making; Action space: The action space (Action,a) of the agent is defined as the link weight vector. The agent influences the routing path of traffic by adjusting the link weight, thereby optimizing network performance and resource allocation. Reward function: The agent evaluates the effect of the action through the reward function (Reward, r), taking into account the data rate The network performance indicators of network delay L and packet loss rate PL; delay L and packet loss rate PL are calculated by the following formulas respectively: The final reward function r is defined as the following formula to maximize the network performance: Among them, d i,j is the transmission delay of link (i, j) on the path, where P lost is the number of lost packets, P total is the total number of packets sent, is the average data rate of network transmission, α, β, γ∈[0,1] are weights that control the contribution of each indicator in the reward function.

7. A network steady-state traffic control method combining ARIMA model and DQN according to claim 6, characterized in that: DQN optimization mechanisms include: Experience replay: DQN stores historical decision data in the experience replay buffer and randomly extracts samples for training; this method reduces the impact of data correlation on the model and improves the stability and efficiency of training; Loss function: In DQN, the loss function is defined using mean square error (MSE) to measure the difference between the Q value generated by the Q network and the target Q value; the loss function is defined as: Loss=E[(y-Q(s t ,a t )) 2 ],#(9) The target Q value y is calculated as follows: Q-value update: The agent updates the Q-value using the Q-learning algorithm; it iteratively optimizes the Q-value after performing actions, observing state changes, and receiving rewards according to the following formula: where α t is the learning rate, i.e., controls the learning step size, γ is the discount factor, s t is the current state, a t is the current action, r t is the reward value, s t+1 For the next state.

8. A network steady-state traffic control method combining ARIMA model and DQN according to claim 7, characterized in that: The specific operation steps of DQN are: S201: Initialization: Initialize the parameters θ of the Q network and the target network Q and θ' Q , and create an experience replay buffer to store state, action, reward, and next state data; the structure and parameters of the Q network and the target network are exactly the same, θ Q and θ' Q Represent the parameter sets of the Q network and the target network respectively, including: weight w1 and bias b1 from the input layer to the hidden layer, weight w2 and bias b2 between hidden layers, weight w3 and bias b3 from the hidden layer to the output layer; S202: Status collection and decision making: The DQN agent receives the flow matrix and network topology output by the ARIMA time series model to form the state s t , calculate the Q value Q(s) of all actions through the Q network t ,a), and use the ∈-greedy strategy to select action a t : Select the action with the largest Q value with probability 1-∈ S203: Execute actions and calculate rewards: Perform action a in the environment t , adjust the routing path and observe the next state s t+1 ; Calculate the instant reward r based on delay, packet loss rate and link utilization t ; S204: Experience Storage: The experience (s t ,a t ,r t ,s t+1 ) into the playback buffer; S205: Network training: Randomly draw a small batch of samples (s t ,a t ,r t ,s t+1 ), calculate the value of the loss function L according to formula (9), calculate the target Q value according to formula (10), and finally update the Q value to θ according to formula (11) Q ; S206: Update target network: In the above training process, only the parameters θ of the Q network are updated Q ,θ' Q remain unchanged; after T training steps, the parameters θ of the Q network are Q Copy to the target network θ' Q , to stabilize the training process.

Images