An intelligent network exception processing method, device, equipment, medium and product

Abstract

This invention provides an intelligent network anomaly handling method, apparatus, device, medium, and product. It collects network traffic characteristics to generate traffic data; constructs a virtual model based on the traffic data using a digital twin network; and determines the network state of the virtual model. When a network failure occurs, it generates an optimization strategy based on the virtual model using a preset strategy model. This application utilizes real-time traffic data to build a digital twin system, constructs a fully mapped virtual model, and completes the evaluation strategy based on a deep deterministic strategy algorithm, resulting in a decision-making method that outputs a fault prediction and repair plan. This method can automatically obtain fault handling solutions and achieve intelligent fault handling.

Description

Technical Field

[0001] This invention relates to the field of network fault handling technology, and more specifically, to an intelligent network anomaly handling method, apparatus, equipment, medium, and product. Background Technology

[0002] The rapid development of optical transport networks has led to a rapid increase in the number of devices in the network. In order to ensure the high efficiency and reliability of private network systems and further improve the service capabilities of private network services, network fault early warning technology and fault handling technology are particularly important.

[0003] Existing fault handling solutions can address the issue of fault early warning and management. However, these methods can only predict and issue alerts for network faults; subsequent fault handling processes still require human intervention and cannot achieve intelligent fault processing. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes an intelligent network anomaly handling method, apparatus, device, medium, and product that can automatically generate fault handling solutions and achieve intelligent fault handling.

[0005] This invention provides an intelligent network anomaly handling method, the method comprising:

[0006] Collect network traffic characteristics and generate traffic data;

[0007] Based on the traffic data, a virtual model is constructed using a digital twin network to determine the network status of the virtual model;

[0008] When the network status fails, an optimization strategy is generated based on the virtual model using a preset strategy model.

[0009] Preferably, the method further includes:

[0010] The virtual model is adjusted according to the optimization strategy, and the network state of the updated virtual model is redefined.

[0011] If no faults are found in the updated network status, the network is adjusted according to the latest optimization strategy.

[0012] As a preferred approach, network traffic characteristics are collected to generate traffic data, specifically including:

[0013] Data packets flowing through the access devices in the network are collected and preprocessed to obtain data packet information;

[0014] Based on the data packet information, the data traffic is classified into normal traffic and abnormal traffic;

[0015] The normal traffic is processed to generate normal traffic data.

[0016] The abnormal traffic is augmented to obtain abnormal traffic data.

[0017] Furthermore, the data traffic is classified according to the data packet information to obtain normal traffic and abnormal traffic, including:

[0018] Based on the deep packet inspection engine, the transport layer protocol of the data packet information is parsed to obtain the characteristic information of the transport layer;

[0019] The feature information is matched with application protocol features based on a preset protocol feature library to obtain matching results for normal or abnormal traffic.

[0020] Preferably, the normal traffic flow is processed to generate normal traffic data, including:

[0021] Load and initialize the characteristic attributes of network packets for the normal traffic;

[0022] Based on the distribution characteristics of various types of network data packets, determine the types of network data packets that need to be sent;

[0023] Based on the preset data packet type size distribution, determine the size of the network data packets to be sent and construct a payload of the corresponding size;

[0024] Based on the network packet type and the constructed payload, the transport layer header, network layer header, and data link layer header of the network packet are constructed to generate the normal traffic data.

[0025] Preferably, data augmentation processing is performed based on the abnormal traffic to obtain abnormal traffic data, including:

[0026] Calculate the weighted density of all sample points in the abnormal traffic;

[0027] The weighted density of each sample point is sorted, and a first preset number of outliers with increasing density are excluded based on the sorting results.

[0028] Several sample sets are obtained by randomly sampling from the remaining sample points;

[0029] The centroid set is determined sequentially using the maximum-minimum distance algorithm for each sample set;

[0030] Find several nearest neighbor samples for each generation center in the set of center points;

[0031] A weighted average of the nearest neighbor samples is selected using a weighted algorithm. Weights are assigned to each nearest neighbor sample to generate a new dataset, which serves as the abnormal traffic data.

[0032] Preferably, the optimization strategy is generated based on the virtual model using a preset strategy model, including:

[0033] The parameters of the decision network and the parameters of the value network of the temporal Markov decision process model are initialized according to the virtual model.

[0034] The evaluation network and noise selection strategy actions are based on the initial temporal Markov decision process model.

[0035] Execute the policy action, detect the change in reward of the temporal Markov decision process model, and record the new state;

[0036] The empirical quadruplets of the temporal Markov decision process model are stored in the experience pool;

[0037] M samples are randomly selected from the experience pool to estimate the target value, and the parameters are updated to reduce the loss.

[0038] The gradient policy update calculation is performed on the sampled samples in the experience pool using a preset gradient policy update formula;

[0039] Based on the evaluation network, update the target network parameters, perform network iteration, and output the best set of repair actions in conjunction with a preset information database;

[0040] The optimal repair action set is used to make action decisions and determine the optimization strategy.

[0041] As a preferred embodiment, determining the network state of the virtual model includes:

[0042] Port connection detection, packet loss detection, and network congestion detection are performed on the access devices of the virtual model;

[0043] When the port connection status index, packet loss rate, or network congestion index of the access device of the virtual model is not within the normal range of the corresponding settings, it is determined that the network status of the virtual model is faulty.

[0044] When the port connection status indicators, packet loss rate, and network congestion indicators of the access device of the virtual model are all within the normal range of the corresponding settings, it is determined that the network status of the virtual model is not faulty.

[0045] Another embodiment of the present invention provides an intelligent network anomaly handling device, the device comprising:

[0046] The data processing module is used to collect network traffic characteristics and generate traffic data;

[0047] The fault simulation module is used to construct a virtual model based on the digital twin network according to the traffic data, and to determine the network status of the virtual model;

[0048] The decision-making module is used to generate an optimization strategy based on the virtual model and a preset strategy model when the network status fails.

[0049] This invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the intelligent network anomaly handling method as described in any of the above embodiments.

[0050] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the intelligent network anomaly handling method as described in any of the above embodiments.

[0051] This invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0052] This invention provides an intelligent network anomaly handling method, apparatus, device, medium, and product. It collects network traffic characteristics to generate traffic data; constructs a virtual model based on the traffic data using a digital twin network; and determines the network state of the virtual model. When a network failure occurs, it generates an optimization strategy based on the virtual model using a preset strategy model. This application utilizes real-time traffic data to build a digital twin system, constructs a fully mapped virtual model, and completes the evaluation strategy based on a deep deterministic strategy algorithm, resulting in a decision-making method that outputs a fault prediction and repair plan. This method can automatically obtain fault handling solutions and achieve intelligent fault handling. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the intelligent network anomaly handling method provided in an embodiment of the present invention;

[0054] Figure 2 This is a flowchart illustrating the traffic data generation process provided in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the abnormal traffic data enhancement processing provided in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the strategy model provided in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the structure of an intelligent network anomaly handling device provided in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram illustrating the working principle of the intelligent network anomaly handling device provided in this embodiment of the invention;

[0059] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] To address the aforementioned technical issues, this paper provides an intelligent network anomaly handling method, see [link to relevant documentation]. Figure 1 This is a flowchart illustrating an intelligent network anomaly handling method provided in an embodiment of the present invention. The method includes:

[0062] Step S1: Collect network traffic characteristics and generate traffic data;

[0063] Step S2: Construct a virtual model based on the digital twin network according to the traffic data, and determine the network status of the virtual model;

[0064] Step S3: When the network status fails, an optimization strategy is generated based on the virtual model using a preset strategy model.

[0065] In this specific implementation, a digital twin system is first constructed based on the collected and processed data. Based on this data and model, the optical transport network undergoes efficient state awareness, scheme evaluation, scheme verification, and scientific decision-making. A unified data information database is built as the single source of fact for the digital twin network, efficiently storing historical and real-time data such as the optical transport network's configuration, topology, status, logs, and user services. Standardized interfaces connect the optical transport network's service applications and physical entities, enabling real-time information collection and processing of the physical network and providing timely diagnosis and analysis.

[0066] Decision optimization models based on deep reinforcement learning can serve fault diagnosis and repair solutions for optical bearer networks more flexibly and efficiently through learning and training.

[0067] It should be noted that the digital twin infrastructure platform is used to build virtual models of digital twin networks. It collects information such as the topology and link information, and network element configuration information of the IoT network, and constructs a virtual twin network that can simulate and predict the actual network state. The digital twin infrastructure platform also has a virtual process playback function, which allows virtually generated network data packets to be played on a specific network at a specified rate and other parameters. The propagation information of the network data packets in the virtual digital twin network is determined by the virtual topology, link state, and network element state. This can be used to simulate network failures and test the effectiveness of network failure recovery schemes.

[0068] The intelligent network anomaly handling method provided in this application utilizes real-time traffic data to build a digital twin system, constructing a fully mapped virtual model. Based on the Deep Deterministic Policy Gradient (DDPG) algorithm, it evaluates strategies, resulting in a decision-making method that uses basic network element configuration information, traffic information, network operating status, link topology information, and repair instances as samples, and outputs fault prediction and the optimal repair solution. It can effectively handle traffic data anomalies including, but not limited to, network link disconnections, abnormal access to attacked ports, network element configuration errors, network congestion, and packet loss.

[0069] By leveraging digital twin networks to efficiently simulate and fully verify network optimization solutions before deployment to physical networks, the cost and risk of real-world network deployment can be significantly reduced, while improving deployment efficiency. Simultaneously, the digital twin network platform enables low-cost, high-efficiency network fault simulation and prevention, as well as research into innovative network technologies. Furthermore, the platform allows for efficient simulation and thorough training of more costly network fault diagnosis and repair procedures, greatly reducing the risks associated with real-world implementation and verification, and minimizing the likelihood of errors during deployment.

[0070] This invention utilizes digital twin simulation combined with deep reinforcement learning technology to solve the problems of difficult-to-track hidden dangers and time-consuming and labor-intensive repairs in traditional network topology and traffic monitoring, greatly improving the operation and maintenance efficiency of optical transport networks. By using a digital twin system for holographic traffic monitoring, based on interactive simulation technology, the level of network fault presentation is significantly improved. Not only can various network elements and topology information in the real network be dynamically visualized, but information such as network traffic monitoring and the dynamic changes, real-time status, and evolution direction throughout the entire lifecycle can also be presented in a holographic manner using the digital twin network model. This virtual-real interactive mapping helps users perceive network status and fault details more clearly, transforming the physical network from a "black box" to a "white box," building a bridge for interactive interpretation between the virtual and real worlds.

[0071] In another embodiment of the present invention, when the method is specifically executed, after generating an optimization strategy through a preset strategy model, the method further includes the following steps:

[0072] Anomalies in optimization strategies need to be tested before they are actually implemented.

[0073] The virtual model is adjusted according to the optimization strategy, the optimal solution is tested, and the twin network is evaluated again.

[0074] The network elements are set as R = {R1, R2, R3...Rn}, and the ports of each network element are set as S = (S1, S2, S3...Sn). The traffic indicators of the ports are determined, with 1 / 0 representing the availability and disabling of the ports. The normal threshold of the traffic indicators is set as U = (U1, U2, U3...Un). The network status of the updated virtual model is then redefined, and the changes after the virtual execution test are compared.

[0075] If the indicators to be repaired reach the effect threshold Ui requirements (i.e., indicators of normal traffic recovery such as port connection restoration, network congestion relief, packet loss disappearance, etc.), the virtual side will issue an implementation command.

[0076] If the requirements are not met and the system remains within the abnormal indicator range, then with manual intervention, the system will be iterated again based on the generated processing results and updated real-time data parameters until the final optimization and implementation of the solution is completed.

[0077] When the updated network status does not show any faults, i.e. meets the requirements, the network is adjusted according to the latest optimization strategy.

[0078] This invention not only supports the construction of a digital twin system based on real-time traffic data to complete timely anomaly handling strategy decisions, but also allows for the manual mapping of normal and abnormal traffic data to the digital twin system and decision system model. This enables pre-training and simulation of the network, improving fault prediction and handling capabilities, and allowing for early prediction and prevention of similar anomalies or related risks in reality.

[0079] The feedback mechanism for network anomaly handling in this invention can better and faster handle network anomalies and complete recovery detection, and has great advantages in handling causal failures and structured risks.

[0080] Leveraging the simulation, analysis, and prediction capabilities of digital twin networks, corresponding network configurations are generated to achieve real-time closed-loop control. These network configurations can be adjusted and optimized within the twin network layer ("inner closed loop"), and also enable real-time control, feedback, and optimization across the three layers of the digital twin network ("outer closed loop"). Through these inner and outer closed loops, the network ultimately achieves self-learning, self-verification, and self-evolutionary real-time closed-loop control.

[0081] The lifecycle of equipment includes "design, development, testing, trial production, and release," while the lifecycle of network management includes "planning, construction, maintenance, and optimization." The responsible parties for networks and equipment differ, and lifecycle management is not well integrated, hindering network fault backtracking, fault prediction, and network optimization design. Digital twin networks encompass not only network functional models but also network element models. By analyzing the characteristics of network element models, the operational status of devices within the network can be predicted. When faults occur during network operation and maintenance, it is possible to trace back not only the network's "past" but also the "past" of network devices through the network element models, thus achieving lifecycle correlation analysis between networks and equipment. By tightly integrating the lifecycles of networks and equipment through digital twin networks and artificial intelligence, refined and intelligent management of the entire network and equipment process can be achieved.

[0082] In yet another embodiment provided by the present invention, see Figure 2 This is a flowchart illustrating the traffic data generation process provided in an embodiment of the present invention. Step S1 specifically includes the following steps:

[0083] Data packets flowing through the network access devices are collected and preprocessed to obtain data packet information. First, n network access devices are selected, and data packets are captured in real time, which are used as Dataset0. Dataset0 is then preprocessed, including data packet selection, protocol header splitting, and five-tuple classification.

[0084] Furthermore, the analyzed data packet information is stored in the database as Dataset1, including information such as the 5-tuple, the protocol of the stream, and the number of bytes in the data stream.

[0085] It should be noted that the physical network used for traffic collection includes the actual network topology and links, network element configuration, traffic transmission and reception information, device information, etc., providing the data information required by the digital twin network in real time, and allowing for actual line adjustments and network port / end optimization based on the solutions provided by the digital twin network.

[0086] Traffic identification and classification are performed, that is, the data traffic is classified according to the data packet information to obtain normal traffic and abnormal traffic.

[0087] Normal traffic and abnormal traffic are processed separately after being classified.

[0088] Specifically, the normal traffic is processed to generate normal traffic data, and the abnormal traffic is processed to augment data, resulting in abnormal traffic data.

[0089] It should be noted that during data collection, common network probes such as UNP, NETP, and NSP are mainly used to capture, filter, and analyze traffic characteristic data in the real network. The collected data is then sent to the data processing center for data preprocessing.

[0090] This invention utilizes traffic data collected from the real-world side to generate virtual traffic. For abnormal traffic, data augmentation methods are employed to simulate various real-world anomalies, effectively addressing the problem of unbalanced and poorly diverse abnormal traffic on the real-world side. Directly collected fault data is inaccurate and incomplete, making it difficult to select appropriate detection models based on actual faults. This invention, combined with the requirements of a data virtual twin network, can quantitatively generate normal and abnormal traffic data, greatly improving flexibility. Simultaneously, the use of data augmentation techniques to generate abnormal traffic effectively eliminates outliers in the data, improving data quality.

[0091] In yet another embodiment of the present invention, when classifying data to obtain normal traffic and abnormal traffic, the following steps are specifically performed:

[0092] Based on the deep packet inspection engine, the transport layer protocol of the data packet information is parsed to obtain the characteristic information of the transport layer; that is, based on the deep packet inspection engine DPI, the data Dataset1 in the database is further parsed, that is, the transport layer protocol is parsed to obtain relevant information.

[0093] The feature information is matched with application protocol features based on a preset protocol feature library to obtain matching results for normal or abnormal traffic.

[0094] After obtaining the transport layer feature information, based on the nDPI protocol feature library, application protocol feature matching is performed according to the feature information of the data packets to obtain the matching result of the application layer protocol. The normal traffic obtained by matching is used as Dataset2.

[0095] It should be noted that this invention selects the AC multi-mode matching algorithm to implement pattern matching of the feature library. Based on the application protocol feature matching results, normal traffic data is classified according to the application layer protocol type, such as FTP, TELNET, SMTP, DNS, HTTP, IMAP, HTTPS, etc.

[0096] For abnormal traffic data, DPI is used to analyze the contents of Dataset1 and match them with feature fields in the feature library to determine abnormal traffic.

[0097] It should be noted that common abnormal traffic includes distributed denial-of-service (DDoS) attacks, network worms, and other abnormal traffic. The categorized abnormal traffic data is stored in the database as Dataset3, containing three categories: DatasetA, DatasetB, and DatasetC.

[0098] In another embodiment of the present invention, the following steps are specifically performed when processing normal data:

[0099] First, load and initialize the network packet characteristic attributes of the normal traffic Dataset2;

[0100] Furthermore, based on the characteristic distribution of various types of network data packets in the normal traffic Dataset2, the type of network data packet to be sent is selected;

[0101] Based on the selected packet type size distribution, determine the size of the network packets to be sent and construct a payload of the corresponding size;

[0102] Furthermore, the transport layer header, network layer header, and data link layer header of the network data packet are constructed sequentially according to the network data packet type and the constructed payload;

[0103] Based on the required traffic volume on the virtual twin network side, repeat the above steps to generate normal traffic data Dateset4.

[0104] This invention not only supports the construction of digital twin systems based on real-time traffic data to make timely decisions on anomaly handling strategies, but also allows for the manual mapping of normal and abnormal traffic data to the digital twin system and decision system model. This enables pre-training and simulation of the network, improving fault prediction and handling capabilities, and allowing for early prediction and prevention of similar or related risks in reality. Furthermore, the feedback mechanism for network anomaly handling in this invention can better and faster process network anomalies and complete recovery detection actions, offering significant advantages in handling causal faults and structured risks.

[0105] In yet another embodiment provided by the present invention, see Figure 3 This is a schematic diagram of the abnormal traffic data enhancement processing provided in an embodiment of the present invention. The specific steps performed during abnormal traffic enhancement processing are as follows:

[0106] Considering that traffic data collected from n devices may have issues such as dataset imbalance, out-of-bounds generated samples, and poor randomness, data augmentation is performed on abnormal traffic data.

[0107] Based on the needs of various types of abnormal traffic on the virtual twin network side, data augmentation is performed on each type of abnormal traffic. It is assumed that the virtual side needs N abnormal traffic samples of DatasetA.

[0108] First, calculate the weighted density of all samples in DatasetA, and then sort the weighted densities of each sample point from largest to smallest. The density calculation method is as follows:

[0109] Density of the i-th sample point

[0110] d ij With d c Distance function

[0111] Where, d ij For the i-th sample point x i With the j-th sample point x j The distance between them, d c The threshold value set; ρ i Indicates falling into x i With d as the center, c The number of samples within a circle of radius .

[0112] The weighted densities of each sample point are sorted according to the calculation results, and m sample points with too low density are excluded, i.e., m outliers.

[0113] Several sample sets are obtained by simple random sampling from the remaining sample points;

[0114] The center point set is determined sequentially for each sample set using the maximum-minimum distance algorithm, that is, one arbitrary sample point v1 is selected from the selected sample set 1 and used as the data generation center;

[0115] Select the sample point v2 that is farthest from sample point v1 as another generation center;

[0116] Select the center point when there is a remaining l (l>2), calculate the Euclidean distance from the remaining sample points to the previous center point, and put the minimum distance into the set in turn;

[0117] The next center point will be the sample point corresponding to the maximum value in the set;

[0118] Repeat the above steps to calculate the remaining required center points, as shown in the following formula:

[0119] dist l=max{min(dist i1 ,dist i2 ,...,dist in )}(l,i=1,2,...,n);

[0120] dist i1 and dist i2 are the Euclidean distances from sample i to v1 and v2, respectively.

[0121] Output the set of center data points G1 generated from sample set 1;

[0122] Repeat the above steps to output the set of central data points of the sample set q, denoted as Gq.

[0123] Find several nearest neighbor samples for each generation center in the set of center points;

[0124] A weighted average of the nearest neighbor samples is selected using a weighted algorithm. A weight is assigned to each nearest neighbor sample to generate N new Class A abnormal traffic data sets, DatasetA1, which are used as the abnormal traffic data.

[0125] This invention utilizes traffic data collected from the real-world side to generate virtual traffic. For abnormal traffic, it employs data augmentation methods to simulate various real-world anomalies, effectively solving the problem of unbalanced and poorly diverse abnormal traffic on the real-world side. Directly collecting fault data is inaccurate and incomplete, making it difficult to select a suitable detection model based on actual faults. This invention, combined with the requirements of a data virtual twin network, can quantitatively generate normal and abnormal traffic data, greatly improving flexibility. Simultaneously, the use of data augmentation techniques to generate abnormal traffic effectively eliminates outliers in the data, improving data quality.

[0126] Virtual traffic is generated based on data collected from real-world network elements. By analyzing traffic characteristics, a data generation method is used to quantitatively expand the normal traffic dataset, enriching data diversity. Addressing the issues of data imbalance, numerous outliers, and inaccurate data in abnormal traffic, a data augmentation method is employed. Based on the maximum and minimum distances of the sample set, abnormal traffic data is quantitatively enhanced, effectively simulating various fault conditions on the real-world side with high flexibility.

[0127] In another embodiment provided by the present invention, when generating an optimization strategy based on the virtual model using a preset strategy model in step S3, see [reference needed]. Figure 4 This is a schematic diagram of the strategy model provided in the embodiments of the present invention.

[0128] In this embodiment, based on simulation information such as network element configuration, network topology, operating status, link topology, and network environment of the optical bearer network in the digital twin platform, a temporal Markov decision process model (MDP) is constructed. Deep reinforcement learning is then used to determine the handling methods for network anomalies. The deep reinforcement learning model includes a quadruple: <state space S, action space, activation function A R, state transition S'>.

[0129] Temporal Markov Decision Process (MDP) models typically employ Actor-Critic reinforcement learning for decision-making. The core and typical idea in reinforcement learning is the temporal difference method, which combines Monte Carlo and dynamic programming approaches. It can learn directly from interactions with the environment without knowing the environmental model itself; it's a method based on value and decision functions. Here, the decision function is represented by the actor (Actor), who gives the action; the value function is represented by the critic, who evaluates the outcome of the actor's action and generates a temporal difference signal to guide the updates of the value and decision functions.

[0130] Typically, temporal difference methods are further divided into Sarsa-based temporal difference under the same-track strategy and Q-learning temporal difference under the off-track strategy. The Q-learning decision method is defined as follows:

[0131]

[0132] Where π represents the strategy, Q represents the estimate of the state value function after updating under the strategy, s represents the state, a represents the action, r represents the payoff, the subscript t represents the current time, the constant discount factor γ∈[0,1), and the subscript t+1 represents the payoff time at the current time.

[0133] The estimated function is determined by observing the payoff at the next time step and incorporating state changes, and the value function is updated accordingly. α is the step size, ranging from 0 to 1, and the function converges when it reaches 0.

[0134] The Bellman optimal equation is an estimate of the action-value function Q under the optimal policy π*.

[0135] Deep Deterministic Policy Gradient (DDPG), an improvement on DQN, also comprises two networks: an Actor (policy network) and a Critic (value evaluation network). The policy network outputs actions, while the value network evaluates the effects of those actions. Each of these networks contains two Deep Q-Networks (DQNs): an evaluation network and a target network. The policy network is updated using gradient calculation formulas, while the value network is updated based on the target value.

[0136] It should be noted that the DQN network is a method based on the classic reinforcement learning algorithm Q-learning, using a deep neural network to fit the Q-value. DQN performs end-to-end fitting using a deep neural network, leveraging the processing power of deep networks for high-dimensional data input.

[0137] DDPG employs sample pooling and fixed-target network techniques. Sample pooling (experience reply): Collected samples are first placed into a sample pool, and then a sample is randomly selected from the pool for network training. This process breaks the correlation between samples, making them independent. Fixed-target network: Calculating the network's target value requires the existing Q-function value; a slower-updating network is used to provide this Q-value. Through these two techniques, the Actor and Critic networks each have a slower-changing copy, which provides the parameters needed for updating information, effectively improving training stability and convergence.

[0138] Deep policy gradient models consist of a policy network and an evaluation network.

[0139] The Actor outputs a specific action A. When A is input into the Critic, it can obtain the maximum Q value. The Actor is updated using a gradient ascent strategy.

[0140] It should be noted that the gradient policy update formula provided in the embodiments of the present invention can be designed according to the actual situation and in combination with the principle of gradient policy update.

[0141] Using fixed network technology, DDGP actually consists of four networks: Actor-slow, Critic-slow, Actor, and Critic. The Critic network is responsible for predicting Q. It has two inputs: action and state, which need to be input together into the Critic network. The loss of this network is the same as that of AC, namely TD-error.

[0142] A deep reinforcement learning quadruple is defined as <state S, action A, reward R, state transition S'>, where the state space S and action space A are composed of multi-dimensional network data information.

[0143] Therefore, S = {T(i), H(i), L(i)}, A = {J1, J2, J3, ..., Jn}.

[0144] Wherein, T represents the connectivity of each link topology in the fiber optic network. T(i) includes the device and connection information of each network node, as well as the channel transmission rate and communication status of different network blocks, which is obtained directly from the real-time captured data Dataset0 flowing through the network access device. H represents the network element configuration information. H(i) includes the network element address identifier and status, which is obtained from the 5-tuple information of the data packet, the protocol of the data flow, and the number of bytes, etc., in Dataset1. L represents the network operating status of each part. L(i) includes the traffic operating status and link utilization, which is obtained from the normal traffic data Dataset2 and the abnormal traffic data Dataset3 after DPI processing.

[0145] J1 to Jn represent various network anomaly handling methods in the action space, including but not limited to fiber optic line connectivity repair; network routing adjustment; traffic shaping (GTS) and interface rate limiting to adjust traffic output rate; traffic filtering, which involves allowing or denying packets that conform to flow classification based on source IP address, destination IP address, MAC address, and protocol number; attack prevention to prevent packet attacks on the CPU; congestion avoidance, which involves discarding packets to avoid network overload; queue scheduling and other methods to adjust queue policies and scheduling parameters; in addition, there are URL filtering and bandwidth adjustment processing methods.

[0146] When generating the optimization strategy, the parameters of the decision network Actor and the value network Critic of the temporal Markov decision process model are randomly initialized, with θ being the weight parameter.

[0147] Based on the current assessment network and noise N t Choose the strategy action for the current moment.

[0148] Execute the policy action, detect the change in reward r of the temporal Markov decision process model, and record the new state.

[0149] The experience quadruples (s) of the time-series Markov decision process model are stored in the experience pool. t a t r t s t+1 ).

[0150] M samples are randomly selected from the experience pool to estimate the target value. The parameters are updated using a preset target value update formula to reduce the loss.

[0151] It should be noted that the loss L is calculated using a preset loss calculation function.

[0152] The gradient policy update is performed on the sampled samples in the experience pool using a preset gradient policy update formula.

[0153] Based on the evaluated network, update the target network parameters, perform network iteration, and combine the solution experience from the information database to output the optimal set of repair actions A, where A = {J}. i J o ...}, (1 <i<o<n)。

[0154] Where J represents the result of each optimization action selection, representing the same or different choices in a total of n steps of decision-making. Each step can be optimized and iterated, and finally output to set A in sequence.

[0155] The optimal repair action set A is used to make action decisions to comprehensively determine the handling plan for the fault and abnormality, and to determine the optimization strategy.

[0156] This invention employs deep reinforcement learning to transform the fault repair problem into a Markov decision process. It combines n-step temporal difference with a deep neural network deterministic policy gradient algorithm to evaluate the policy, forming a decision-making method that uses basic network element configuration information, traffic information, network operating status, link topology information, and repair instances as samples, and outputs fault prediction and the optimal repair solution. It can effectively handle, but is not limited to, traffic data anomalies such as network link disconnection, abnormal access to attacked ports, network element configuration errors, network congestion, and packet loss. This effectively improves the automation and intelligence level of network anomaly handling and the efficiency of operation and maintenance management. Unlike traditional machine learning and reinforcement learning alone, this invention utilizes n-step temporal actions and combines DDPG in deep reinforcement learning for feedback and backtracking of the results of each action. This allows for better decision-making of the optimal repair solution from a global perspective and considering the correlation effects, and dynamically updates the solution. Therefore, the accuracy and adaptability of the network anomaly handling solution are effectively improved.

[0157] By mapping real-time data onto a digital twin system, and constructing a temporal Markov decision process model, the anomaly handling and judgment problem in optical transport networks is transformed into a machine learning problem. Through deep reinforcement learning, existing data and information are scientifically and effectively utilized to scientifically analyze and predict perception and decision-making within the digital twin system. This digital and intelligent approach empowers the operation and maintenance level of the optical transport network, achieving a unified management and control method integrating real-time monitoring, intelligent detection, route detection, and efficient repair.

[0158] In another embodiment of the present invention, the following steps are specifically performed when performing network status detection:

[0159] Port connection detection, packet loss detection, and network congestion detection are performed on the access devices of the virtual model;

[0160] When the port connection status indicators, packet loss rate, or network congestion indicators of the access device of the virtual model are not within the normal range of the corresponding settings, it is determined that the network status of the virtual model has failed. Then, with manual intervention, the system is iterated again based on the generated processing results and updated real-time data parameters.

[0161] If the indicators meet the effect threshold Ui requirement, and the port connection status indicators, packet loss rate, and network congestion indicators of the access device are all within the normal range, that is, the traffic recovery indicators such as port connection recovery, network congestion relief, and packet loss disappearance are all within the normal range, then it is determined that the network status of the virtual model has not failed.

[0162] See Figure 5 This is a schematic diagram of the structure of an intelligent network anomaly handling device provided in an embodiment of the present invention. The device includes:

[0163] The data processing module is used to collect network traffic characteristics and generate traffic data;

[0164] The fault simulation module is used to construct a virtual model based on the digital twin network according to the traffic data, and to determine the network status of the virtual model;

[0165] The decision-making module is used to generate an optimization strategy based on the virtual model and a preset strategy model when the network status fails.

[0166] See Figure 6 This diagram illustrates the working principle of the intelligent network anomaly handling device provided in this embodiment of the invention. First, a digital twin system is constructed based on the collected and processed data. Based on this data and model, the optical transport network undergoes efficient state perception, scheme evaluation, scheme verification, and scientific decision-making. A unified data information database is built as the single source of fact for the digital twin network, efficiently storing historical and real-time data such as the optical transport network's configuration, topology, status, logs, and user services. Standardized interfaces connect the optical transport network's service applications and physical entities, enabling real-time information collection and processing of the physical network and providing timely diagnosis and analysis. The decision optimization model based on deep reinforcement learning can be learned and trained to more flexibly and efficiently serve the fault diagnosis and repair schemes of the optical transport network.

[0167] Physical Information Acquisition Module: This module primarily uses common network probes (UNP), internet probes (NETP), and network security detection probes (NSP) to capture, filter, and analyze traffic characteristic data in real-world networks, and then collects the data for preprocessing at the data processing center.

[0168] The data processing module primarily performs data preprocessing, including data augmentation and data balancing, and completes preliminary anomaly diagnosis and fault location and delimitation based on traffic characteristic information and real-time network element status information. It consists of a traffic analysis unit, a traffic generation unit, a generation strategy unit, and a process storage unit.

[0169] Fault simulation module: mainly completes digital twin mapping, and realizes further network status assessment through network fault simulation. It consists of traffic playback unit, twin network element and twin link, mapping unit and control unit.

[0170] The decision-making module transforms the fault repair problem into a Markov decision process. It evaluates the strategy based on the Deep Deterministic Policy Gradient (DDPG) algorithm, forming a decision-making method that takes basic network element configuration information, traffic information, network operating status, link topology information, and repair instances as samples, and outputs fault prediction and the best repair solution.

[0171] It should be noted that the intelligent network anomaly handling device provided in this embodiment can execute all the steps and functions performed by the business server in the intelligent network anomaly handling method provided in any of the above embodiments. The specific functions of the device will not be described in detail here.

[0172] See Figure 7 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, such as an intelligent network anomaly handling program. When the processor executes the computer program, it implements the steps in the various embodiments of the intelligent network anomaly handling method described above, for example... Figure 1 The steps S1 to S3 are shown. Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-described device embodiments.

[0173] For example, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the intelligent network anomaly handling device. For example, the computer program can be divided into several modules, the specific functions of which have been described in detail in the intelligent network anomaly handling method provided in any of the above embodiments; therefore, the specific functions of the device will not be repeated here.

[0174] The terminal device described can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on an intelligent network anomaly handling device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0175] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the intelligent network anomaly handling device, connecting all parts of the device via various interfaces and lines.

[0176] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the intelligent network anomaly handling device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0177] If the integrated module of the intelligent network anomaly handling device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0178] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the functional network element implementing the method described in the above embodiments.

[0179] The computer program product provided in this embodiment can execute all the steps and functions of the intelligent network anomaly handling method provided in any of the above embodiments. The specific functions of the product will not be described in detail here.

[0180] It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered to be within the scope of protection of this invention.

Claims

1. An intelligent network anomaly handling method, characterized in that, The method includes: Collect network traffic characteristics and generate traffic data; Based on the traffic data, a virtual model is constructed using a digital twin network to determine the network status of the virtual model; When the network status fails, an optimization strategy is generated based on the virtual model using a preset strategy model.

2. The intelligent network anomaly handling method according to claim 1, characterized in that, The method further includes: The virtual model is adjusted according to the optimization strategy, and the network state of the updated virtual model is redefined. If no faults are found in the updated network status, the network is adjusted according to the latest optimization strategy.

3. The intelligent network anomaly handling method according to claim 1, characterized in that, The network traffic characteristics are collected to generate traffic data, specifically including: Data packets flowing through the access devices in the network are collected and preprocessed to obtain data packet information; Based on the data packet information, the data traffic is classified into normal traffic and abnormal traffic; The normal traffic is processed to generate normal traffic data. The abnormal traffic is augmented to obtain abnormal traffic data.

4. The intelligent network anomaly handling method according to claim 3, characterized in that, Based on the data packet information, the data traffic is classified into normal traffic and abnormal traffic, including: Based on the deep packet inspection engine, the transport layer protocol of the data packet information is parsed to obtain the characteristic information of the transport layer; The feature information is matched with application protocol features based on a preset protocol feature library to obtain matching results for normal or abnormal traffic.

5. The intelligent network anomaly handling method according to claim 3, characterized in that, Based on the normal traffic, traffic generation processing is performed to obtain normal traffic data, including: Load and initialize the characteristic attributes of network packets for the normal traffic; Based on the distribution characteristics of various types of network data packets, determine the types of network data packets that need to be sent; Based on the preset data packet type size distribution, determine the size of the network data packets to be sent and construct a payload of the corresponding size; Based on the network packet type and the constructed payload, the transport layer header, network layer header, and data link layer header of the network packet are constructed to generate the normal traffic data.

6. The intelligent network anomaly handling method according to claim 3, characterized in that, Data augmentation processing is performed on the abnormal traffic to obtain abnormal traffic data, including: Calculate the weighted density of all sample points in the abnormal traffic; The weighted density of each sample point is sorted, and a first preset number of outliers with increasing density are excluded based on the sorting results. Several sample sets are obtained by randomly sampling from the remaining sample points; The centroid set is determined sequentially using the maximum-minimum distance algorithm for each sample set; Find several nearest neighbor samples for each generation center in the set of center points; A weighted average of the nearest neighbor samples is selected using a weighted algorithm. Weights are assigned to each nearest neighbor sample to generate a new dataset, which serves as the abnormal traffic data.

7. The intelligent network anomaly handling method according to claim 1, characterized in that, Based on the virtual model, an optimization strategy is generated using a preset strategy model, including: The parameters of the decision network and the parameters of the value network of the temporal Markov decision process model are initialized according to the virtual model. The evaluation network and noise selection strategy actions are based on the initial temporal Markov decision process model. Execute the policy action, detect the change in reward of the temporal Markov decision process model, and record the new state; The empirical quadruplets of the temporal Markov decision process model are stored in the experience pool; M samples are randomly selected from the experience pool to estimate the target value, and the parameters of the time-series Markov decision process model are updated accordingly. The gradient policy update calculation is performed on the sampled samples in the experience pool using a preset gradient policy update formula; Based on the evaluation network, update the target network parameters, perform network iteration, and output the best set of repair actions in conjunction with a preset information database; The optimal repair action set is used to make action decisions and determine the optimization strategy.

8. The intelligent network anomaly handling method according to claim 1, characterized in that, Determining the network state of the virtual model includes: Port connection detection, packet loss detection, and network congestion detection are performed on the access devices of the virtual model; When the port connection status index, packet loss rate, or network congestion index of the access device of the virtual model is not within the normal range of the corresponding settings, it is determined that the network status of the virtual model is faulty. When the port connection status indicators, packet loss rate, and network congestion indicators of the access device of the virtual model are all within the normal range of the corresponding settings, it is determined that the network status of the virtual model is not faulty.

9. An intelligent network anomaly handling device, characterized in that, The device includes: The data processing module is used to collect network traffic characteristics and generate traffic data; The fault simulation module is used to construct a virtual model based on the digital twin network according to the traffic data, and to determine the network status of the virtual model; The decision-making module is used to generate an optimization strategy based on the virtual model and a preset strategy model when the network status fails.

10. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the intelligent network anomaly handling method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the intelligent network anomaly handling method as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 8.

Images