Network fault adaptive detection system based on machine learning

Abstract

The invention discloses a network fault adaptive detection system based on machine learning, which relates to the technical field of network security and comprises a data acquisition module, a feature extraction and selection module, a model training and updating module, a fault detection module, a fault positioning and diagnosis module and an adaptive optimization module. According to the network fault adaptive detection system based on machine learning, by generating new features, the expression ability of a feature space is significantly enhanced, and the detection precision of a machine learning model is improved; the deep learning model based on the causal relationship is combined with the graph neural network and time sequence analysis, so that the complex causal relationship in the network is effectively processed, and the fault diagnosis is more accurate; through the combination of the multi-objective evolutionary algorithm and the fuzzy logic control algorithm, the network optimization strategy is more flexible and efficient, the network performance is improved, the weight of the optimization objective can be dynamically adjusted according to the actual demand, and the performance of the system in a complex network environment is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a network fault adaptive detection system based on machine learning. Background Art

[0002] As modern communication networks become increasingly complex, the number and types of devices in the network continue to increase, and the diversity and dynamics of network traffic also increase. This makes it difficult for traditional network fault detection and diagnosis methods to meet current needs. Traditional fault detection methods often rely on predefined rules and thresholds, which show obvious limitations when facing complex and changing network environments, and are prone to false positives and false negatives. In addition, traditional methods lack the ability to analyze multi-level and multi-dimensional causal relationships in the network, making it difficult to effectively identify and infer the root cause of the fault, which in turn affects the efficiency and accuracy of troubleshooting.

[0003] In terms of network optimization, existing technologies mostly use single-objective optimization algorithms, which cannot take into account the optimization of multiple network performance indicators at the same time, resulting in inflexible optimization strategies and difficulty in meeting the diverse needs under different network environments. In addition, current optimization methods usually fail to effectively combine the ambiguity and uncertainty of network operation, resulting in unsatisfactory performance in practical applications.

[0004] In the prior art, there are the following deficiencies:

[0005] Limitations of feature extraction and selection: Traditional feature extraction methods rely on manual design or statistical-based methods, which cannot fully capture the complex and hidden feature patterns in network data. This results in the detection capabilities of machine learning models being limited, especially when dealing with abnormally complex and diverse network traffic, and may not be able to effectively identify potential faults.

[0006] Insufficient fault location and diagnosis: Most existing fault diagnosis methods rely on static causal relationship models and cannot effectively handle dynamic, multi-level, and multi-dimensional causal relationships in the network. This method is difficult to accurately locate the source of the fault in a complex network environment and is prone to ignoring potential and hidden causes of the fault.

[0007] Limitations of network optimization strategies: Traditional network optimization methods usually only optimize a single performance indicator and fail to effectively balance multiple indicators such as bandwidth utilization, latency, and packet loss rate. In addition, the lack of flexible optimization strategies and the failure to fully consider the uncertainty and ambiguity in network operation lead to poor optimization results. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a network fault adaptive detection system based on machine learning to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions: a network fault adaptive detection system based on machine learning, comprising a data acquisition module, a feature extraction and selection module, a model training and updating module, a fault detection module, a fault location and diagnosis module and an adaptive optimization module;

[0010] The data collection module is based on distributed sensors or agents and is deployed on different nodes of the network to collect various data from the network in real time, including flow data, log data, network topology information, and device performance indicators;

[0011] The feature extraction and selection module extracts features that are helpful for fault detection from the original data based on time domain analysis, frequency domain analysis, and automatic feature extraction methods in deep learning; when selecting features, the most representative features are selected from a large number of features based on feature selection algorithms to reduce the complexity of the model;

[0012] In the feature extraction and selection module, new features are also generated through generative adversarial networks (GANs) or variational autoencoders (VAEs) to enrich the original feature space and enhance the detection ability of the model;

[0013] The model training and updating module is used to train the machine learning model and continuously update the model according to changes in the network; the model hyperparameters are optimized through cross-validation and grid search methods to train the model, and the model is updated based on adaptive learning so that the model dynamically adapts to new network environments and fault types;

[0014] The fault detection module is based on machine learning algorithms and detection strategies to analyze network status in real time and detect potential faults, including using time series analysis to detect abnormal fluctuations in network traffic and using classifiers to identify abnormal behavior in logs;

[0015] The fault location and diagnosis module can quickly determine the node or link where the fault occurs by analyzing the network topology and quickly find the potential cause of the fault by reasoning about the causal relationship.

[0016] In the fault location and diagnosis module, a deep learning model based on causality is built in to build and explain the causal chain of faults in complex network environments. Graph neural networks (GNNs) are used to model nodes and links in the network. Combined with time series analysis, a dynamic causal relationship diagram between faults is constructed to process multi-level and multi-dimensional causal relationships and find out the potential causes of faults.

[0017] The adaptive optimization module is used to dynamically adjust network configuration and resource allocation according to the detected fault type and frequency, so as to prevent similar faults from happening again;

[0018] In the adaptive optimization module, a multi-objective evolutionary algorithm and a fuzzy logic control algorithm are built in; the multi-objective evolutionary algorithm simultaneously optimizes multiple network performance indicators, including bandwidth utilization, delay, and packet loss rate, and is used to improve the overall network performance when adjusting the network configuration; the fuzzy logic control algorithm is introduced to assign different weights to each optimization target to provide a flexible optimization strategy.

[0019] To further optimize the technical solution, the data acquisition module has a built-in preprocessing module to perform operations including data cleaning, denoising, and normalization on redundant or incomplete information in the collected data;

[0020] The preprocessing module designs different strategies to process different types of data, including a regularization processing strategy for log data and a segmentation analysis strategy for traffic data; and a built-in anomaly detection algorithm is built into the preprocessing module to preliminarily filter out obviously abnormal data.

[0021] To further optimize the technical solution, in the feature extraction and selection module, the generative adversarial network trains the generator and the discriminator by minimizing the loss function of the generator and maximizing the loss function of the discriminator. The function is as follows:

[0022]

[0023] in,( ) represents a generator function; ( ) is derived from a random noise distribution ( ) is a random vector sampled from ( ) represents the discriminator function; ( ) is the input feature sample, is the real sample ( ) or generate samples ( );( ) indicates the expected value; ( ) is the value function of the generative adversarial network.

[0024] Further optimizing the technical solution, in the feature extraction and selection module, the variational autoencoder generates new features by learning the potential distribution of the input data, and the encoder converts the input feature ( ) is mapped to the distribution parameters in the latent space ( )and( ), and sample the latent variables ( ), then the decoder will ( ) is converted into a new feature ( ), the training objective is to maximize the marginal likelihood function, which is achieved by minimizing the following loss function:

[0025]

[0026] in,( ) is the latent variable ( )’s approximate posterior distribution; ( ) are the parameters of the encoder; ( ) are the decoder parameters; ( ) is the expected value; ( ) is the Kullback-Leibler divergence, which measures the difference between two distributions; ) is the prior distribution, which is preset to be the standard normal distribution.

[0027] To further optimize the technical solution, the model training and updating module includes the following process when updating the model:

[0028] Weight update: When the system receives new data, it dynamically adjusts the model weights according to the incremental gradient descent model, so that the model can adapt to changes in the network environment in real time.

[0029] Transfer learning fine-tuning: When adapting to new failure modes, the system uses a transfer learning model to fine-tune the parameters of the pre-trained model to quickly adapt the system to new tasks.

[0030] Model architecture adjustment: The system searches the model through the architecture and automatically adjusts the structural parameters of the model to adapt to new network conditions or fault types, so that the detection capability of the model is always in the best state.

[0031] To further optimize the technical solution, the fault detection module has a built-in confidence-based detection model, which triggers an alarm only when the detected anomaly exceeds a certain confidence threshold, and supports alarm settings of different levels to adjust the alarm sensitivity according to actual needs;

[0032] The confidence-based detection model includes the following contents:

[0033] Confidence measures the probability value of the model's output for the detection result. Assume that the system uses multiple classifiers or time series models to detect network failures, and each model ( ) will output a probability of failure occurring ( ), the overall confidence ( ) is calculated by weighted average of multiple model outputs:

[0034]

[0035] in,( ) represents the number of models involved in the detection; ( ) is the first ( ) model weight, reflecting the reliability or historical performance of the model; ) is the first ( ) model output failure probability;

[0036] The system will compare the confidence level ( ) and preset alarm thresholds ( ), if the confidence exceeds the threshold, the detection result is considered reliable and an alarm is triggered:

[0037]

[0038] in:

[0039] ( ) is the alarm threshold, which is set according to the system sensitivity requirements;

[0040] Based on the fact that different types of faults require different sensitivity settings, a dynamic threshold adjustment mechanism is introduced. Assuming that the threshold value ( )According to the current network status ( ) is adjusted, then:

[0041]

[0042] in,( ) is the initial basic threshold;

[0043] ( ) is the current network status parameter, such as traffic load and error rate;

[0044] ( ) is a state function used to dynamically adjust the threshold. For example, the higher the network load, the lower the threshold, making the system more sensitive to anomalies;

[0045] Based on different levels of alarm settings, a multi-level threshold system is introduced to define multiple thresholds ( ), corresponding to different alarm levels ( ),but:

[0046]

[0047] in,( ) is the first ( ) level alarm threshold;

[0048] ( ) represents the alarm level, such as level 1 for minor alarm, level 2 for moderate alarm, and level 3 for severe alarm.

[0049] To further optimize the technical solution, in the fault location and diagnosis module, a deep learning model based on causality is used to find out the potential cause of the fault, including:

[0050] Collect status information and historical data of all relevant devices and links in the network, including traffic, delay, packet loss rate, and fault logs;

[0051] Preprocess the data, such as normalization, denoising, feature extraction, and generate node feature vectors ( );

[0052] According to the network topology, a network graph is constructed, where nodes represent devices, links or fault events, and edges represent the connection relationship between nodes;

[0053] Use the collected historical data to train the graph neural network, update the node representation through multiple rounds of message passing, and introduce time features through time series analysis;

[0054] During the training process, the model parameters are optimized through back propagation, so that the model can accurately predict and infer causal relationships;

[0055] When a fault is detected in the network, the system uses the trained GNN model to generate a causal relationship matrix between fault events ( );

[0056] Based on the causal relationship matrix, the system generates a causal chain and determines the priority and causal order between faults;

[0057] The generated causal chain is converted into a graphical representation through an interpretable algorithm;

[0058] Quickly understand the root cause of the failure, its propagation path, and potential associated factors through a visual diagram of the cause-effect chain.

[0059] To further optimize the technical solution, the adaptive optimization module formulates the optimal network configuration strategy through reinforcement learning or Bayesian optimization method;

[0060] When the adaptive optimization module detects that a link is frequently congested, it automatically adjusts the bandwidth allocation of the link or replans the traffic path to alleviate the congestion.

[0061] To further optimize the technical solution, the system also includes a visualization module to monitor and manage the operating status of the system and generate detailed fault detection reports and charts;

[0062] It supports visualization of real-time data, including network traffic, device status, and fault distribution; and provides analysis of historical data to understand the long-term trends and patterns of faults.

[0063] To further optimize the technical solution, the system also includes an edge intelligent analysis module, which distributes part of the data processing tasks in the data acquisition module and part of the fault detection tasks in the fault detection module to the edge nodes, and uses federated learning and micro neural networks to perform data analysis on the edge devices.

[0064] Compared with the prior art, the present invention provides a network fault adaptive detection system based on machine learning, which has the following beneficial effects:

[0065] This network fault adaptive detection system based on machine learning significantly enhances the expression ability of the feature space and improves the detection accuracy of the machine learning model by generating new features; the deep learning model based on causality combines with graph neural network and time series analysis to effectively handle the complex causal relationship in the network, making fault diagnosis more accurate; the combination of multi-objective evolutionary algorithm and fuzzy logic control algorithm makes the network optimization strategy more flexible and efficient, which not only improves the network performance, but also can dynamically adjust the weight of the optimization target according to actual needs; the system's performance in dealing with complex network environments is significantly improved, the probability of false alarms and missed alarms is greatly reduced, and the efficiency of fault diagnosis and network optimization is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of the structure of a network fault adaptive detection system based on machine learning proposed by the present invention;

[0067] Figure 2 A deep learning model flow chart of a fault location and diagnosis module in a network fault adaptive detection system based on machine learning proposed by the present invention;

[0068] Figure 3 This is an incremental gradient descent model diagram of a network fault adaptive detection system based on machine learning proposed by the present invention;

[0069] Figure 4 A transfer learning model diagram of a network fault adaptive detection system based on machine learning proposed by the present invention;

[0070] Figure 5 This is an architecture search model diagram of a network fault adaptive detection system based on machine learning proposed by the present invention;

[0071] Figure 6 This is a structural schematic diagram of an edge intelligent analysis module in a network fault adaptive detection system based on machine learning proposed by the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Embodiment 1:

[0074] See also Figure 1 ,A network fault adaptive detection system based on machine learning includes a data acquisition module, a feature extraction and selection module, a model training and updating module, a fault detection module, a fault location and diagnosis module, and an adaptive optimization module.

[0075] The data acquisition module is based on distributed sensors or agents and is deployed on different nodes of the network to collect various data from the network in real time, including flow data, log data, network topology information, and equipment performance indicators.

[0076] In this embodiment, the data acquisition module has a built-in preprocessing module to perform operations including data cleaning, denoising, and normalization on redundant or incomplete information in the collected data to ensure the quality and consistency of the data.

[0077] The preprocessing module designs different strategies to process different types of data, including a regularization processing strategy for log data and a segmentation analysis strategy for traffic data; and a built-in anomaly detection algorithm is built into the preprocessing module to preliminarily filter out obviously abnormal data.

[0078] The feature extraction and selection module extracts features that are helpful for fault detection from the original data based on time domain analysis, frequency domain analysis, and automatic feature extraction methods in deep learning; when selecting features, based on feature selection algorithms, such as information gain-based selection methods, L1 regularization, etc., the most representative features are selected from a large number of features to reduce the complexity of the model.

[0079] In the feature extraction and selection module, new features are also generated through generative adversarial networks (GANs) or variational autoencoders (VAEs) to enrich the original feature space and enhance the detection capability of the model.

[0080] In this embodiment, the generative adversarial network trains the generator and the discriminator by minimizing the loss function of the generator and maximizing the loss function of the discriminator. The function is as follows:

[0081]

[0082] in,( ) represents a generator function; ( ) is derived from a random noise distribution ( ) is a random vector sampled from ( ) represents the discriminator function; ( ) is the input feature sample, is the real sample ( ) or generate samples ( );( ) indicates the expected value; ( ) is the value function of the generative adversarial network.

[0083] When the model is used:

[0084] Initialization: The parameters of the generator and discriminator are randomly initialized;

[0085] Training process: The generator receives random noise ( ), generate new features ( ). The discriminator receives the real features ( ) and generated features ( ), and try to label the real features as 1 and the generated features as 0;

[0086] Loss optimization: By optimizing the value function of GAN ( ), the generator and discriminator are constantly improving, the generator learns to generate features that are closer to the real features, and the discriminator learns to better distinguish the real features from the generated features;

[0087] Feature integration: The final generated features ( ) are integrated into the feature space to enrich the model input and improve the fault detection capability.

[0088] The variational autoencoder generates new features by learning the potential distribution of the input data. The encoder converts the input features ( ) is mapped to the distribution parameters in the latent space ( )and( ), and sample the latent variables ( ), then the decoder will ( ) is converted into a new feature ( ), the training objective is to maximize the marginal likelihood function, which is achieved by minimizing the following loss function:

[0089]

[0090] in,( ) is the latent variable ( )’s approximate posterior distribution; ( ) are the parameters of the encoder; ( ) are the decoder parameters; ( ) is the expected value; ( ) is the Kullback-Leibler divergence, which measures the difference between two distributions; ) is the prior distribution, which is preset to be the standard normal distribution.

[0091] When the model is used:

[0092] Encoding phase: Input features ( ) is mapped to the latent space by the encoder, generating the mean ( ) and variance ( ), which represents the distribution of latent variables;

[0093] Sampling and decoding: Sampling from the potential distribution ( ), and then generate new features through the decoder ( );

[0094] Loss optimization: By minimizing the loss function ( ), train the encoder and decoder so that the generated new features ( ) retains the key information of the original features while having a certain degree of diversity;

[0095] Feature enhancement: New features generated ( ) are added to the feature set and used together with the original features to train the machine learning model to improve the performance of fault detection.

[0096] The model training and updating module is used to train the machine learning model and continuously update the model according to changes in the network; the hyperparameters of the model are optimized through cross-validation and grid search methods, the model is trained, and at the same time, it is updated based on adaptive learning. The model dynamically adapts to new network environments and fault types. For example, when a new fault mode is detected, the module can automatically adjust the weights or update part of the model structure to maintain the effectiveness of the detection system.

[0097] In this embodiment, when updating the model, the following process is included:

[0098] Weight update, when the system receives new data, it updates the weights according to the incremental gradient descent model (such as Figure 3 As shown in Figure 2), the model weights are adjusted dynamically to enable the model to adapt to changes in the network environment in real time.

[0099] Figure 3 middle:

[0100] ( ) means at time ( ) model parameters (such as weights) at the moment;

[0101] ( ) is the learning rate, which controls the update step size;

[0102] ( ) is the loss function ( ) for the parameter ( )’s gradient;

[0103] ( ) is the model input ( ) under the prediction output;

[0104] ( ) is the true label or target value.

[0105] The model states that whenever new data ( ) arrives, the system will update the weights according to the prediction error of the current model, so that the model can gradually adapt to the new data distribution.

[0106] Transfer learning fine-tuning, when it is necessary to adapt to new failure modes, the system uses transfer learning models (such as Figure 4 As shown in Figure 3), by fine-tuning the parameters of the pre-trained model, the system can quickly adapt to new tasks.

[0107] Figure 4 middle:

[0108] ( ) represents the model parameters after fine-tuning;

[0109] ( ) are the parameters of the pre-trained model;

[0110] ( ) is the loss function defined on the new dataset;

[0111] ( ) are the labels of the new data.

[0112] By fine-tuning on new datasets, the system can quickly adapt to new patterns while retaining its ability to identify old failure patterns.

[0113] Model architecture adjustment, the system searches for models through architecture (such as Figure 5 As shown in the figure), the structural parameters of the model are automatically adjusted to adapt to new network conditions or fault types, so that the detection capability of the model is always in the best state.

[0114] Figure 5 middle:

[0115] ( ) is the loss function of the model on the validation set;

[0116] ( ) represents the model architecture, which is represented by the architecture parameters ( ) and weight parameters ( )Decide;

[0117] ( ) is the complexity of the model (such as the number of parameters, the amount of calculation, etc.), which is used to avoid overly complex models;

[0118] ( ) is a trade-off term, controlling the balance between loss and complexity.

[0119] By optimizing this objective function, the system can automatically search for the optimal model architecture, ensuring high performance while maintaining reasonable complexity.

[0120] The fault detection module is based on machine learning algorithms and detection strategies to analyze network status in real time and detect potential faults, including using time series analysis to detect abnormal fluctuations in network traffic and using classifiers to identify abnormal behaviors in logs.

[0121] In this embodiment, the fault detection module has a built-in confidence-based detection model. An alarm is triggered only when the detected anomaly exceeds a certain confidence threshold. Different levels of alarm settings are supported, and the sensitivity of the alarm is adjusted according to actual needs.

[0122] The confidence-based detection model includes the following contents:

[0123] Confidence measures the probability value of the model's output for the detection result. Assume that the system uses multiple classifiers or time series models to detect network failures, and each model ( ) will output a probability of failure occurring ( ), the overall confidence ( ) is calculated by weighted average of multiple model outputs:

[0124]

[0125] in,( ) represents the number of models involved in the detection; ( ) is the first ( ) model weight, reflecting the reliability or historical performance of the model; ) is the first ( ) model output failure probability;

[0126] The system will compare the confidence level ( ) and preset alarm thresholds ( ), if the confidence exceeds the threshold, the detection result is considered reliable and an alarm is triggered:

[0127]

[0128] in:

[0129] ( ) is the alarm threshold, which is set according to the system sensitivity requirements; when the system detects certain key indicators (such as an abnormal and sharp increase in traffic), the confidence level may rise rapidly, causing an alarm to be triggered.

[0130] Based on the fact that different types of faults require different sensitivity settings, a dynamic threshold adjustment mechanism is introduced. Assuming that the threshold value ( )According to the current network status ( ) is adjusted, then:

[0131]

[0132] in,( ) is the initial basic threshold;

[0133] ( ) is the current network status parameter, such as traffic load and error rate;

[0134] ( ) is a state function used to dynamically adjust the threshold. For example, the higher the network load, the lower the threshold, making the system more sensitive to anomalies;

[0135] Under high load conditions, the system may lower the threshold ( ), making the system more sensitive and providing early warning of possible failures.

[0136] Based on different levels of alarm settings, a multi-level threshold system is introduced to define multiple thresholds ( ), corresponding to different alarm levels ( ),but:

[0137]

[0138] in,( ) is the first ( ) level alarm threshold;

[0139] ( ) represents the alarm level, such as level 1 for minor alarm, level 2 for moderate alarm, and level 3 for severe alarm.

[0140] When the confidence level is 70%, a Level 1 alarm is triggered, indicating that attention is needed; when the confidence level reaches 90%, a Level 3 alarm is triggered, prompting emergency treatment.

[0141] The fault location and diagnosis module can quickly determine the node or link where the fault occurs by analyzing the network topology, and can quickly find out the potential cause of the fault by reasoning about the causal relationship.

[0142] In the fault location and diagnosis module, a causal relationship-based deep learning model is built in to build and explain the fault causal chain in a complex network environment. The graph neural network (GNN) is used to model the nodes and links in the network, and combined with time series analysis, a dynamic causal relationship diagram between faults is constructed to process multi-level and multi-dimensional causal relationships and find out the potential causes of the faults.

[0143] In this embodiment, the fault location and diagnosis module finds out the potential cause of the fault based on the deep learning model of causality, including:

[0144] like Figure 2 As shown, the status information and historical data of all relevant devices and links in the network are collected, including traffic, delay, packet loss rate, and fault logs;

[0145] Preprocess the data, such as normalization, denoising, feature extraction, and generate node feature vectors ( );

[0146] According to the network topology, a network graph is constructed, where nodes represent devices, links or fault events, and edges represent the connection relationship between nodes;

[0147] Use the collected historical data to train the graph neural network, update the node representation through multiple rounds of message passing, and introduce time features through time series analysis;

[0148] During the training process, the model parameters are optimized through back propagation, so that the model can accurately predict and infer causal relationships;

[0149] When a fault is detected in the network, the system uses the trained GNN model to generate a causal relationship matrix between fault events ( );

[0150] Based on the causal relationship matrix, the system generates a causal chain and determines the priority and causal order between faults;

[0151] The generated causal chain is converted into a graphical representation through an interpretable algorithm;

[0152] Quickly understand the root cause of the failure, its propagation path, and potential associated factors through a visual diagram of the cause-effect chain.

[0153] The adaptive optimization module is used to dynamically adjust network configuration and resource allocation according to the detected fault type and frequency, so as to prevent similar faults from happening again.

[0154] In the adaptive optimization module, a multi-objective evolutionary algorithm and a fuzzy logic control algorithm are built in; the multi-objective evolutionary algorithm simultaneously optimizes multiple network performance indicators, including bandwidth utilization, delay, and packet loss rate, and is used to improve the overall network performance when adjusting the network configuration; the fuzzy logic control algorithm is introduced to assign different weights to each optimization target to provide a flexible optimization strategy.

[0155] In this embodiment, the adaptive optimization module formulates the optimal network configuration strategy through reinforcement learning or Bayesian optimization method;

[0156] When the adaptive optimization module detects that a link is frequently congested, it automatically adjusts the bandwidth allocation of the link or replans the traffic path to alleviate the congestion. To ensure the effectiveness of the optimization strategy, the module will also continuously monitor the performance of the optimized network and make further adjustments based on feedback information. This dynamic optimization capability enables the system to not only passively detect and handle faults, but also actively improve the health of the network.

[0157] Embodiment 2:

[0158] In the network fault adaptive detection system based on machine learning described in Example 1, the system also includes a visualization module to monitor and manage the operating status of the system and generate detailed fault detection reports and charts.

[0159] It supports visualization of real-time data, including network traffic, device status, and fault distribution; and provides analysis of historical data to understand the long-term trends and patterns of faults.

[0160] At the same time, a report is generated, which covers all detected faults, including time, location, cause and treatment suggestions. In order to improve the visualization effect, the module uses multi-dimensional data display technology, such as heat map, timeline map and relationship map, so that complex data can be seen at a glance. Administrators can freely filter, sort and drill data through the interactive interface to better understand the network status and make decisions.

[0161] like Figure 6 As shown, the system also includes an edge intelligent analysis module, which distributes part of the data processing tasks in the data acquisition module and part of the fault detection tasks in the fault detection module to the edge nodes, and uses federated learning and micro neural networks to perform data analysis on the edge devices.

[0162] At the same time, in this embodiment, the system can also have a built-in fault prediction and prevention module, which can take preventive measures in advance to avoid failures by predicting impending failures. This module uses sequence models in deep learning, such as LSTM (Long Short-Term Memory Network) and Transformer model, combined with Bayesian Networks to analyze historical data and real-time data to predict possible future failure points. Unlike traditional prediction models, this module also introduces an anomaly pattern generator, which can train the model by simulating various abnormal patterns in the network to make it more predictive. The module can automatically trigger the adaptive optimization module based on the prediction results to adjust the network configuration to prevent potential failures. This combination of prediction and prevention not only improves the reliability of the network, but also reduces the frequency and cost of emergency fault handling.

[0163] The beneficial effects of the present invention are:

[0164] This network fault adaptive detection system based on machine learning significantly enhances the expression ability of the feature space and improves the detection accuracy of the machine learning model by generating new features; the deep learning model based on causality combines with graph neural network and time series analysis to effectively handle the complex causal relationship in the network, making fault diagnosis more accurate; the combination of multi-objective evolutionary algorithm and fuzzy logic control algorithm makes the network optimization strategy more flexible and efficient, which not only improves the network performance, but also can dynamically adjust the weight of the optimization target according to actual needs; the system's performance in dealing with complex network environments is significantly improved, the probability of false alarms and missed alarms is greatly reduced, and the efficiency of fault diagnosis and network optimization is improved.

[0165] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0166] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A network fault adaptive detection system based on machine learning, characterized in that: It includes data acquisition module, feature extraction and selection module, model training and updating module, fault detection module, fault location and diagnosis module, and adaptive optimization module; The data collection module is based on distributed sensors or agents and is deployed on different nodes of the network to collect various data from the network in real time, including flow data, log data, network topology information, and device performance indicators; The feature extraction and selection module extracts features that are helpful for fault detection from the original data based on time domain analysis, frequency domain analysis, and automatic feature extraction methods in deep learning; when selecting features, the most representative features are selected from a large number of features based on feature selection algorithms to reduce the complexity of the model; In the feature extraction and selection module, new features are also generated through generative adversarial networks (GANs) or variational autoencoders (VAEs) to enrich the original feature space and enhance the detection ability of the model; The model training and updating module is used to train the machine learning model and continuously update the model according to changes in the network; the model hyperparameters are optimized through cross-validation and grid search methods to train the model, and the model is updated based on adaptive learning so that the model dynamically adapts to new network environments and fault types; The fault detection module is based on machine learning algorithms and detection strategies to analyze network status in real time and detect potential faults, including using time series analysis to detect abnormal fluctuations in network traffic and using classifiers to identify abnormal behavior in logs; The fault location and diagnosis module can quickly determine the node or link where the fault occurs by analyzing the network topology and quickly find the potential cause of the fault by reasoning about the causal relationship. In the fault location and diagnosis module, a deep learning model based on causality is built in to build and explain the causal chain of faults in complex network environments. Graph neural networks (GNNs) are used to model nodes and links in the network. Combined with time series analysis, a dynamic causal relationship diagram between faults is constructed to process multi-level and multi-dimensional causal relationships and find out the potential causes of faults. The adaptive optimization module is used to dynamically adjust network configuration and resource allocation according to the detected fault type and frequency, so as to prevent similar faults from happening again; In the adaptive optimization module, a multi-objective evolutionary algorithm and a fuzzy logic control algorithm are built in; the multi-objective evolutionary algorithm simultaneously optimizes multiple network performance indicators, including bandwidth utilization, delay, and packet loss rate, and is used to improve the overall network performance when adjusting the network configuration; the fuzzy logic control algorithm is introduced to assign different weights to each optimization target to provide a flexible optimization strategy.

2. The network fault adaptive detection system based on machine learning according to claim 1 is characterized in that: The data acquisition module has a built-in preprocessing module to perform operations including data cleaning, denoising, and normalization on redundant or incomplete information in the collected data; The preprocessing module designs different strategies to process different types of data, including a regularization processing strategy for log data and a segmentation analysis strategy for traffic data; And an anomaly detection algorithm is built into the preprocessing module to initially filter out obviously abnormal data.

3. The network fault adaptive detection system based on machine learning according to claim 1 is characterized in that: In the feature extraction and selection module, the generative adversarial network trains the generator and the discriminator by minimizing the loss function of the generator and maximizing the loss function of the discriminator. The function is as follows: ； in,( ) represents a generator function; ( ) is derived from a random noise distribution ( ) is a random vector sampled from ( ) represents the discriminator function; ( ) is the input feature sample, is the real sample ( ) or generate samples ( );( ) indicates the expected value; ( ) is the value function of the generative adversarial network.

4. The network fault adaptive detection system based on machine learning according to claim 1 is characterized in that: In the feature extraction and selection module, the variational autoencoder generates new features by learning the potential distribution of the input data. The encoder converts the input features ( ) is mapped to the distribution parameters in the latent space ( )and( ), and sample the latent variables ( ), then the decoder will ( ) is converted into a new feature ( ), the training objective is to maximize the marginal likelihood function, which is achieved by minimizing the following loss function: ； in,( ) is the latent variable ( )’s approximate posterior distribution; ( ) are the parameters of the encoder; ( ) are the decoder parameters; ( ) is the expected value; ( ) is the Kullback-Leibler divergence, which measures the difference between two distributions; ) is the prior distribution, which is preset to be the standard normal distribution.

5. The network fault adaptive detection system based on machine learning according to claim 1 is characterized in that: In the model training and updating module, when updating the model, the following processes are included: Weight update: When the system receives new data, it dynamically adjusts the model weights according to the incremental gradient descent model, so that the model can adapt to changes in the network environment in real time. Transfer learning fine-tuning: When adapting to new failure modes, the system uses a transfer learning model to fine-tune the parameters of the pre-trained model to quickly adapt the system to new tasks. Model architecture adjustment: The system searches the model through the architecture and automatically adjusts the structural parameters of the model to adapt to new network conditions or fault types, so that the detection capability of the model is always in the best state.

6. The network fault adaptive detection system based on machine learning according to claim 1, characterized in that: The fault detection module has a built-in confidence-based detection model. An alarm is triggered only when the detected anomaly exceeds a certain confidence threshold. It also supports alarm settings of different levels and adjusts the alarm sensitivity according to actual needs. The confidence-based detection model includes the following contents: Confidence measures the probability value of the model's output for the detection result. Assume that the system uses multiple classifiers or time series models to detect network failures, and each model ( ) will output a probability of failure occurring ( ), the overall confidence ( ) is calculated by weighted average of multiple model outputs: ； in,( ) represents the number of models involved in the detection; ( ) is the first ( ) model weight, reflecting the reliability or historical performance of the model; ) is the first ( ) model output failure probability; The system will compare the confidence level ( ) and preset alarm thresholds ( ), if the confidence exceeds the threshold, the detection result is considered reliable and an alarm is triggered: ； in: ( ) is the alarm threshold, which is set according to the system sensitivity requirements; Based on the fact that different types of faults require different sensitivity settings, a dynamic threshold adjustment mechanism is introduced. Assuming that the threshold value ( )According to the current network status ( ) is adjusted, then: ； in,( ) is the initial basic threshold; ( ) is the current network status parameter, such as traffic load and error rate; ( ) is a state function used to dynamically adjust the threshold. For example, the higher the network load, the lower the threshold, making the system more sensitive to anomalies; Based on different levels of alarm settings, a multi-level threshold system is introduced to define multiple thresholds ( ), corresponding to different alarm levels ( ),but: ； in,( ) is the first ( ) level alarm threshold; ( ) represents the alarm level, such as level 1 for minor alarm, level 2 for moderate alarm, and level 3 for severe alarm.

7. The network fault adaptive detection system based on machine learning according to claim 1, characterized in that: In the fault location and diagnosis module, the potential causes of the fault are found based on the deep learning model of causality, including: Collect status information and historical data of all relevant devices and links in the network, including traffic, delay, packet loss rate, and fault logs; Preprocess the data, such as normalization, denoising, feature extraction, and generate node feature vectors ( ); According to the network topology, a network graph is constructed, where nodes represent devices, links or fault events, and edges represent the connection relationship between nodes; Use the collected historical data to train the graph neural network, update the node representation through multiple rounds of message passing, and introduce time features through time series analysis; During the training process, the model parameters are optimized through back propagation, so that the model can accurately predict and infer causal relationships; When a fault is detected in the network, the system uses the trained GNN model to generate a causal relationship matrix between fault events ( ); Based on the causal relationship matrix, the system generates a causal chain and determines the priority and causal order between faults; The generated causal chain is converted into a graphical representation through an interpretable algorithm; Quickly understand the root cause of the failure, its propagation path, and potential associated factors through a visual diagram of the cause-effect chain.

8. The network fault adaptive detection system based on machine learning according to claim 1, characterized in that: The adaptive optimization module formulates the optimal network configuration strategy through reinforcement learning or Bayesian optimization method; When the adaptive optimization module detects that a link is frequently congested, it automatically adjusts the bandwidth allocation of the link or replans the traffic path to alleviate the congestion.

9. The network fault adaptive detection system based on machine learning according to claim 1, characterized in that: The system also includes a visualization module to monitor and manage the system's operating status and generate detailed fault detection reports and charts; Supports visualization of real-time data, including network traffic, device status, and fault distribution; It also provides historical data analysis functions to understand the long-term trends and patterns of failures.

10. The network fault adaptive detection system based on machine learning according to claim 1, characterized in that: The system also includes an edge intelligent analysis module, which distributes part of the data processing tasks in the data acquisition module and part of the fault detection tasks in the fault detection module to the edge nodes, and uses federated learning and micro neural networks to perform data analysis on edge devices.

Images