Network perception anomaly detection system based on big data

Abstract

The invention, which relates to the technical field of network awareness anomaly detection, discloses a network awareness anomaly detection system based on big data, comprising a data acquisition module, a feature fusion module, a map construction module, a model calculation module, a root cause reasoning module, a threshold decision module and a response control module. The data acquisition module receives network flow data, equipment state data and system log data and outputs a standardized feature set; the feature fusion module is connected with the data acquisition module, dynamically calculates a weight coefficient of each data source based on information entropy, performs feature aggregation of privacy protection through a federated learning framework, and outputs a fusion feature vector; the atlas construction module is connected with the feature fusion module, maintains a network equipment node set and a communication edge set in real time, and updates a space-time association atlas according to a topology change event; and the model calculation module is connected with the atlas construction module, extracts topological features through a space-time diagram convolutional network, and updates a detection model based on an incremental learning mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of network perception anomaly detection, and in particular to a network perception anomaly detection system based on big data. Background Art

[0002] Network-aware anomaly detection technology utilizes advanced data analysis, machine learning, graph theory, and other techniques to collect, process, and analyze data from multiple sources, including network traffic, device status, and system logs, to identify abnormal behavior or potential threats within the network. Therefore, utilizing advanced technologies to enhance the intelligence and security of network-aware anomaly detection has become a pressing issue.

[0003] In the field of network-aware anomaly detection, traditional centralized learning methods require uploading all data to a central server for processing, which not only increases communication overhead but also easily leads to privacy leakage risks. In addition, many existing graph-based network anomaly detection systems rely on static topology structures and cannot reflect changes in network topology in a timely manner, resulting in insufficient ability to identify new attack patterns. At the same time, most traditional anomaly detection systems use fixed anomaly judgment thresholds, which are prone to high false alarm or missed alarm rates when network behavior changes over time and load. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a network-aware anomaly detection system based on big data to solve the problem that traditional centralized learning methods require uploading all data to a central server for processing, which not only increases communication overhead but also easily leads to privacy leakage risks. In addition, many existing graph-based network anomaly detection systems rely on static topology structures and cannot reflect changes in network topology in a timely manner, resulting in insufficient ability to recognize new attack patterns.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a network perception anomaly detection system based on big data, comprising: Data acquisition module, feature fusion module, graph construction module, model calculation module, root cause reasoning module, threshold decision module and response control module; The data acquisition module receives network traffic data, device status data and system log data, and outputs a standardized feature set; The feature fusion module is connected to the data acquisition module, dynamically calculates the weight coefficient of each data source based on information entropy, performs privacy-preserving feature aggregation through the federated learning framework, and outputs a fused feature vector; The graph construction module is connected to the feature fusion module to maintain the network device node set and the communication edge set in real time, and update the spatiotemporal correlation graph according to the topology change event; The model calculation module is connected to the graph construction module, extracts topological features through the spatiotemporal graph convolutional network, updates the detection model based on the incremental learning mechanism, and outputs the anomaly probability value; The root cause reasoning module is connected to the model calculation module to construct a causal reasoning graph to calculate the node impact factor, and generates the root cause analysis result in combination with the security knowledge base; The threshold decision module is connected to the model calculation module to analyze the historical anomaly probability distribution characteristics and dynamically adjust the anomaly determination threshold parameters; The response control module is connected to the root cause reasoning module and the threshold decision module, and triggers the execution of the disposal strategy when the abnormal probability exceeds the current threshold.

[0007] As a preferred solution of the network perception anomaly detection system based on big data of the present invention, wherein: receiving network traffic data, device status data and system log data, and outputting a standardized feature set, the specific steps are as follows: Perform field cleaning and structured extraction on the raw network traffic data obtained by the receiving port to obtain a set of network traffic feature vectors. Perform numerical unification and missing value processing on the device operation status data obtained by the receiving port to obtain a set of device status feature vectors. Perform semantic parsing and keyword extraction on the system log data obtained by the receiving port to obtain a set of log feature vectors. Standardize the features in the network traffic feature vector set, the device status feature vector set, and the log feature vector set, and perform dimension alignment to obtain a standardized feature set; Substituting the standardized network traffic feature vector set, device status feature vector set, and log feature vector set into the above expression, we can obtain the compressed vector representation of the three types of data. The concatenation function is used to merge the three types of compressed feature vectors and output a complete standardized feature set.

[0008] As a preferred solution of the network-aware anomaly detection system based on big data of the present invention, the specific steps are as follows: dynamically calculating the weight coefficients of each data source based on information entropy, performing privacy-preserving feature aggregation through a federated learning framework, and outputting a fused feature vector. For the three types of standardized feature sets, their feature dimensions are extracted and the information entropy values ​​are calculated. The following information entropy function is used to obtain the information entropy value sequences of the three types of data sources. The information entropy values ​​of the three types of data sources are normalized using the weight distribution function to obtain the dynamic weight coefficient of each data source. ; The three types of data sources are deployed on different edge nodes, and each node runs a local feature extraction model; The following process is performed using the federated learning communication protocol: Each node receives the global model parameters sent by the central server; Use the local data set to update the local model parameters to obtain updated parameters; Perform differential privacy scrambling on the updated parameters to generate encrypted gradients; Upload the encrypted gradient to the federated coordination server; Perform weighted aggregation function on encrypted gradients and dynamic weights; Finally, update the global model parameters; The feature mapping function is used to project the feature space corresponding to the updated global model parameters and output the fused feature vector.

[0009] As a preferred solution of the network perception anomaly detection system based on big data of the present invention, wherein: the real-time maintenance of the network device node set and the communication edge set, and the updating of the spatiotemporal correlation graph according to the topology change event, the specific steps are as follows: Extracting device identification fields and interface connection status from device status data; Add the device identifier to the initial network device node set, and establish a communication edge record for the device pairs that have communication behavior to form an initial communication edge set, and then use the initial communication edge set to construct the initial network graph structure; Parse the source IP and destination IP fields in the system log feature vector set and update the node connection relationship in the communication edge set; Identify changes in the current network device node set and communication edge set, and generate a list of topology change events; The spatiotemporal graph update mechanism is used to adjust the graph structure of each type of change in the topology change event list and update the spatiotemporal correlation graph. Specifically: for Each new node in the graph is added to the graph node set ; for For each old node in the graph, remove it from the graph node set Remove; for Each new edge in , add it to the graph edge set ; for For each old edge in , remove it from the graph edge set delete; At the same time, a timestamp is introduced to mark the active time period of each edge to construct a spatiotemporal correlation graph; The graph structure evolves dynamically over time to reflect real changes in network topology; Serialize and save the updated spatiotemporal correlation graph and output the final graph structure .

[0010] As a preferred solution of the network-aware anomaly detection system based on big data of the present invention, wherein: the topological features are extracted by the spatiotemporal graph convolutional network, the detection model is updated based on the incremental learning mechanism, and the anomaly probability value is output, the specific steps are as follows: Graph Structure Execute the graph adjacency matrix construction function to generate the graph adjacency matrix; The graph adjacency matrix construction function; Perform node mapping on the fused feature vector to generate a graph feature matrix; A spatiotemporal graph convolutional network is used to jointly process the graph adjacency matrix and the graph feature matrix to extract the network topology feature representation; The graph adjacency matrix and graph feature matrix Input into the multi-layer spatiotemporal graph convolutional network and perform graph convolution operation; After multi-layer propagation, the topological feature representation is output; The time aggregation function is used to perform sequence modeling on the topological features within a continuous time period to generate a spatiotemporal fusion feature vector; Execute the time aggregation function on the topological features of multiple consecutive time windows and output the spatiotemporal fusion feature vector; Adopting incremental learning mechanism to update detection model parameters online, obtain real-time optimized detection model, normalize the output of real-time detection model, and output abnormal probability value; Output abnormal probability value .

[0011] As a preferred solution of the network perception anomaly detection system based on big data of the present invention, the steps of constructing a causal reasoning graph to calculate the node impact factor and generating the root cause analysis result in combination with the security knowledge base are as follows: Extract alarm record sets from the historical alarm database, where each alarm record contains timestamp, device ID, and alarm type information; Use the association rule mining algorithm Apriori to perform pattern recognition on alarm records, find frequent item sets, deduce the causal relationship between alarms, and output a list of potential causal relationships; Input the potential causal relationship list into the causal reasoning graph construction function to create a directed graph; Calculate the impact factor of each node in the causal reasoning graph; For detected anomalies, based on the key nodes involved, the security knowledge base query function is used to retrieve relevant background information from the pre-built security knowledge base; The impact factor of the key node Corresponding background information Input into the comprehensive analysis module to conduct multi-dimensional analysis, which considers factors such as impact, urgency and repair cost; Generate a root cause analysis report covering the main problem points, possible causes and recommended measures to assist operation and maintenance personnel in quickly locating the root cause of the problem and taking action.

[0012] As a preferred solution of the network perception anomaly detection system based on big data of the present invention, wherein: the analysis of historical anomaly probability distribution characteristics and dynamic adjustment of anomaly determination threshold parameters are specifically carried out as follows: Extract the historical probability values ​​within a continuous time period from the abnormal probability value sequence, perform sliding collection in fixed time windows, and construct a historical abnormal probability set. ; For the historical anomaly probability set Apply the kernel density estimation method to construct the abnormal probability density function; Analyze the tail area of ​​the abnormal probability density function and set the initial abnormality judgment threshold; For continuous The initial anomaly judgment threshold sequence of each time window executes the trend change detection function and outputs the threshold adjustment signal; The initial anomaly determination threshold is weighted and modified in combination with trend change information to output a dynamic anomaly determination threshold; The threshold decision function is executed on the abnormal probability value output at the current moment and the dynamic abnormality judgment threshold.

[0013] As a preferred solution of the network perception anomaly detection system based on big data of the present invention, wherein: when the anomaly probability exceeds the current threshold, the processing strategy is triggered to execute, and the specific steps are: Compare the current abnormal probability value output by the model calculation module with the dynamic abnormality judgment threshold to generate an abnormality judgment result; Obtain the root cause analysis results of the current abnormal event, search for matching policy numbers in the predefined disposal policy library based on the type of each key node and its impact factor, and output a list of policy numbers; For the strategy number list, the priority sorting function is performed in combination with the key node impact factors corresponding to each strategy, and an ordered processing strategy queue is output; Input the ordered processing strategy queue into the automated execution engine, and call the processing actions corresponding to the strategy numbers in sequence; The actions performed include automatically isolating high-risk devices, starting backup links or switching redundant devices, sending SMS / email alerts, and triggering deep log collection tasks; Evaluate and log the effectiveness of each completed disposal strategy and generate a disposal feedback report.

[0014] In a second aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the network-aware anomaly detection system based on big data as described in the first aspect of the present invention is implemented.

[0015] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the network-aware anomaly detection system based on big data as described in the first aspect of the present invention is implemented.

[0016] The beneficial effects of the present invention are: through the standardized processing of multi-source heterogeneous data, privacy-preserving feature fusion based on information entropy and federated learning, dynamic spatiotemporal correlation graph modeling, and an adaptive threshold mechanism driven by abnormal probability distribution, a complete network-aware anomaly detection system is constructed. The system not only improves the accuracy and real-time performance of anomaly identification, but also enhances the adaptability and robustness to complex network environments; at the same time, combined with causal reasoning and automated response mechanisms, it realizes closed-loop control of root cause analysis and policy disposal, effectively ensuring the security and stability of the network system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 Schematic diagram of the network perception anomaly detection system based on big data in Example 1.

[0019] Figure 2 This is an architectural diagram of the big data-based network perception anomaly detection system in Example 1.

[0020] Figure 3 Schematic diagram of the feature fusion process in Example 1.

[0021] Figure 4 Flowchart for ST-GCN model calculation in Example 1.

[0022] Figure 5This is a flow chart of the response control closed loop in Example 1. DETAILED DESCRIPTION

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0024] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0025] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0026] Example, see Figure 1-Figure 5 , is an embodiment of the present invention, which provides a network perception anomaly detection system based on big data, including: Data acquisition module, feature fusion module, graph construction module, model calculation module, root cause reasoning module, threshold decision module and response control module; The data acquisition module receives network traffic data, device status data, and system log data, and outputs a standardized feature set; Furthermore, the raw network traffic data obtained by the receiving port is cleaned and structured to obtain a set of network traffic feature vectors. The device operation status data obtained by the receiving port is numerically unified and missing values ​​are processed to obtain a set of device status feature vectors. The system log data obtained by the receiving port is semantically parsed and keyword extracted to obtain a set of log feature vectors. The features in the network traffic feature vector set, device status feature vector set, and log feature vector set are standardized and dimensionally aligned to obtain a standardized feature set. The expression is: ; in, is the standardized feature vector of the final output, The first principal components, For the The eigenvectors corresponding to the principal components are is the normalized feature of the input, For the The weighted coefficients of the principal components, is the attenuation factor, is the current time step, is the base of natural logarithm; Substituting the standardized network traffic feature vector set, device status feature vector set, and log feature vector set into the above expression, we can obtain the compressed vector representation of the three types of data. The concatenation function is used to merge the three types of compressed feature vectors and output a complete standardized feature set. The expression is: ; in, Represents vector concatenation operation, is the standardized feature set of the final output; It should be noted that the data acquisition module of the present invention not only realizes the unified and standardized processing of three types of heterogeneous data: network traffic, device status and system logs, but also introduces a time decay factor through principal component analysis, thereby enhancing the timeliness and representativeness of feature expression. The design takes into account the characteristics of network behavior evolving over time, avoiding the lag problem of traditional static feature extraction methods in dynamic environments, thereby improving the accuracy of subsequent feature fusion and anomaly detection. In addition, the vector splicing operation ensures the semantic consistency of different data sources in a unified space, laying the foundation for distributed modeling under the federated learning framework.

[0027] The feature fusion module is connected to the data acquisition module, dynamically calculates the weight coefficients of each data source based on information entropy, performs privacy-preserving feature aggregation through the federated learning framework, and outputs a fused feature vector; Furthermore, for the three types of standardized feature sets, their feature dimensions are extracted and the information entropy values ​​are calculated. The following information entropy function is used to obtain the information entropy value sequences of the three types of data sources. The expression is: ; ; in, is a set of features of a certain type, is the number of samples in the feature set of this type, For the The frequency distribution of a sample in the entire sample, To prevent taking small constants whose logarithms are infinite; The information entropy values ​​of the three types of data sources are normalized using the weight distribution function to obtain the dynamic weight coefficient of each data source. ; The three types of data sources are deployed on different edge nodes, and each node runs a local feature extraction model; The following process is performed using the federated learning communication protocol: Each node receives the global model parameters sent by the central server; Use the local data set to update the local model parameters to obtain updated parameters; Perform differential privacy scrambling on the updated parameters to generate encrypted gradients; Upload the encrypted gradient to the federated coordination server; The weighted aggregation function is performed on the encrypted gradient and dynamic weight, and the expression is: ; in, is the aggregated global model gradient update, For the Dynamic weights of class data sources, For the Encrypted gradients of class data sources; Finally update the global model parameters: ; in, is the learning rate, which is used to control the model update step size; The feature mapping function is used to project the feature space corresponding to the updated global model parameters and output the fused feature vector, which is expressed as: ; in, is the fused feature vector, is the feature mapping matrix, is the bias term, is the Sigmoid activation function, is the standardized input feature after splicing; It should be noted that the information richness of various data sources is dynamically evaluated through the information entropy mechanism, and fusion weights are allocated accordingly, so that the model can adaptively focus on data dimensions with more discriminative capabilities. At the same time, a combination of federated learning and differential privacy is adopted to complete global model updates while ensuring the privacy of edge node data, effectively solving the problem of difficulty in centralized training of cross-institutional and cross-domain data. The fusion strategy not only improves the generalization ability of the system, but also enhances its robustness and security in complex network environments. It is especially suitable for scenarios with high data security requirements such as finance and government affairs.

[0028] The graph construction module is connected to the feature fusion module to maintain the network device node set and communication edge set in real time, and update the spatiotemporal correlation graph according to topology change events; Furthermore, the device identification field and the interface connection status are extracted from the device status data; Add the device identifier to the initial network device node set, and establish communication edge records for the device pairs with communication behavior to form the initial communication edge set. Then, use the initial communication edge set to construct the initial network graph structure. The expression is: ; in, represents the network diagram at the initial moment, is a collection of network device nodes, is the set of communication edges between devices; Parse the source IP and destination IP fields in the system log feature vector set and update the node connection relationship in the communication edge set; Identify changes in the current network device node set and communication edge set, and generate a list of topology change events; The spatiotemporal graph update mechanism is used to adjust the graph structure of each type of change in the topology change event list and update the spatiotemporal correlation graph. Specifically: for Each new node in the graph is added to the graph node set ; for For each old node in the graph, remove it from the graph node set Remove; for Each new edge in , add it to the graph edge set ; for For each old edge in , remove it from the graph edge set delete; At the same time, a timestamp is introduced to mark the active time period of each edge to construct a spatiotemporal correlation graph. The expression is: ; in, A timestamp set for each edge, used to record the time period when the communication occurred; The graph structure evolves dynamically over time to reflect the real changes in network topology; Serialize and save the updated spatiotemporal correlation graph and output the final graph structure ; It should be noted that the graph construction module of the present invention constructs a dynamic association graph with time and space perception capabilities by maintaining the network device node set and communication edge set in real time, combined with the topology change event recognition mechanism. The graph not only records the connection relationship between network entities, but also retains the time series information of communication behavior through the timestamp mechanism, providing high-fidelity structural support for subsequent graph neural network modeling. Compared with traditional static graph modeling methods, this solution can better reflect the actual operating status of the network, help capture hidden attack paths and abnormal communication patterns, and significantly improve the accuracy and response speed of anomaly detection.

[0029] The model calculation module is connected to the graph construction module, and the topological features are extracted through the spatiotemporal graph convolutional network. Update the detection model based on the incremental learning mechanism and output the abnormal probability value; Furthermore, for the graph structure Execute the graph adjacency matrix construction function to generate the graph adjacency matrix; The graph adjacency matrix construction function is expressed as: ; in, is the graph adjacency matrix, Representation node With node Is there a communication relationship? is the communication edge set at the current moment; Perform node mapping on the fused feature vector to generate a graph feature matrix; A spatiotemporal graph convolutional network is used to jointly process the graph adjacency matrix and the graph feature matrix to extract the network topology feature representation; The graph adjacency matrix and graph feature matrix Input into the multi-layer spatiotemporal graph convolutional network and perform graph convolution operation. The expression is: ; in, For the The hidden state of the layer, is the adjacency matrix with self-loops added, is the degree matrix, For the The trainable parameter matrix of the layer, is the ReLU activation function; After multi-layer propagation, the output topological feature representation is: ; in, is the number of graph convolution layers, is the extracted topological feature matrix; The time aggregation function is used to perform sequence modeling on the topological features within a continuous time period to generate a spatiotemporal fusion feature vector; The time aggregation function is performed on the topological features of multiple consecutive time windows to output the spatiotemporal fusion feature vector, which is expressed as: ; in, is the time window length, is the time attenuation coefficient, For the The topological characteristics of the time window, It is the spatiotemporal fusion feature after time weighting; The incremental learning mechanism is used to update the detection model parameters online to obtain a real-time optimized detection model. The output of the real-time detection model is normalized and the abnormal probability value is output. The expression is: ; in, is the probability of anomaly occurrence, is the score output by the model, is the Sigmoid function; Output abnormal probability value ; It should be noted that the model calculation module of the present invention deeply models the network topology features based on the spatiotemporal graph convolutional network ST-GCN, and introduces a time aggregation function to perform weighted fusion of multi-time window features, thereby enhancing the model's ability to understand the evolution trend of network behavior. At the same time, the incremental learning mechanism is adopted to enable the model to be continuously optimized online to avoid performance degradation caused by changes in the network environment. The Sigmoid normalized output ensures that the abnormal probability value has a clear probabilistic meaning, which facilitates threshold judgment and strategy linkage. As the core intelligent engine of the entire system, the module provides a stable and reliable decision-making basis for abnormality identification and root cause reasoning.

[0030] The root cause reasoning module connects to the model calculation module to build a causal reasoning graph to calculate the node impact factor. Generate root cause analysis results by combining with the security knowledge base; Furthermore, an alarm record set is extracted from the historical alarm database, where each alarm record contains timestamp, device ID and alarm type information; Use the association rule mining algorithm Apriori to perform pattern recognition on alarm records, find frequent item sets, deduce the causal relationship between alarms, and output a list of potential causal relationships; Input the potential causal relationship list into the causal reasoning graph construction function to create a directed graph; Calculate the influence factor of each node in the causal reasoning graph. The expression is: ; in, Indicates pointing to a node The set of predecessor nodes, Represents an edge The weight of is the attenuation factor; For detected anomalies, based on the key nodes involved, the security knowledge base query function is used to retrieve relevant background information from the pre-built security knowledge base; The impact factor of the key node Corresponding background information Input into the comprehensive analysis module to conduct multi-dimensional analysis, which considers factors such as impact, urgency and repair cost; Generate a root cause analysis report covering the main problem points, possible causes, and recommended measures to help operation and maintenance personnel quickly locate the root cause of the problem and take action; It should be noted that through causal reasoning graph modeling, combined with impact factor calculation and security knowledge base assisted analysis, a key transition from "anomaly discovery" to "cause location" has been achieved. The module uses the causal relationship mined from historical alarm data to construct a directed graph structure, and then quantifies the influence of each node in the network through a recursive propagation mechanism, so as to accurately identify the most likely source of the fault. Combined with the predefined security knowledge base, the system can provide highly targeted and executable handling suggestions, greatly shortening the troubleshooting time of operation and maintenance personnel and improving the self-healing ability and operational efficiency of the network system.

[0031] The threshold decision module is connected to the model calculation module to analyze the historical anomaly probability distribution characteristics and dynamically adjust the anomaly judgment threshold parameters; Furthermore, the historical probability values ​​within a continuous time period are extracted from the abnormal probability value sequence, and sliding collection is performed in units of fixed time windows to construct a historical abnormal probability set. ; For the historical anomaly probability set Apply the kernel density estimation method to construct the abnormal probability density function, which is expressed as follows: ; in, represents the number of samples in the set, is the bandwidth parameter, which controls the degree of smoothing. is the Gaussian kernel function , is the historical abnormal probability value; Analyze the tail area of ​​the abnormal probability density function and set the initial abnormality judgment threshold; For continuous The initial anomaly judgment threshold sequence of each time window executes the trend change detection function and outputs the threshold adjustment signal; The initial anomaly determination threshold is weighted and modified in combination with trend change information to output a dynamic anomaly determination threshold; The threshold decision function is executed on the abnormal probability value output at the current moment and the dynamic abnormality judgment threshold. The expression is: ; in, For abnormal judgment results, Indicates that the current status is abnormal. Indicates that the current status is normal; It should be noted that by modeling the historical anomaly probability distribution through the kernel density estimation method, and combining the tail area analysis and trend change detection mechanism, the dynamic adjustment of the anomaly judgment threshold is realized. Compared with the fixed threshold or simple sliding average method, this scheme can more scientifically adapt to the periodic and sudden changes in network behavior, reduce the occurrence of false alarms and missed alarms, and the weighted correction mechanism further enhances the system's adaptability to long-term trend changes, making the entire anomaly detection process more intelligent and automated, and improving the stability and reliability of the system in complex network environments.

[0032] The response control module connects the root cause reasoning module and the threshold decision module, and triggers the execution of the disposal strategy when the abnormal probability exceeds the current threshold; Furthermore, the current abnormal probability value output by the model calculation module is compared with the dynamic abnormality judgment threshold to generate an abnormality judgment result; Obtain the root cause analysis results of the current abnormal event, search for matching policy numbers in the predefined disposal policy library based on the type of each key node and its impact factor, and output a list of policy numbers; For the strategy number list, the priority sorting function is performed in combination with the key node impact factors corresponding to each strategy, and an ordered processing strategy queue is output; Input the ordered processing strategy queue into the automated execution engine, and call the processing actions corresponding to the strategy numbers in sequence; The actions performed include automatically isolating high-risk devices, starting backup links or switching to redundant devices, sending SMS / email alerts, and triggering in-depth log collection tasks; Evaluate and log the effectiveness of each completed disposal strategy and generate a disposal feedback report; It should be noted that based on anomaly detection results and root cause analysis reports, combined with a prioritization mechanism and an automated execution engine, a closed-loop control system from "anomaly discovery" to "proactive resolution" is achieved. This module not only supports multiple types of emergency response actions but also dynamically adjusts the execution order of policies based on influencing factors, ensuring that resources are prioritized for the most critical nodes. The feedback recording mechanism provides valuable historical data support for subsequent policy optimization and model iteration, giving the entire system strong self-evolution capabilities and closed-loop operation and maintenance capabilities.

[0033] This embodiment also provides a computer device, which is suitable for the network-aware anomaly detection system based on big data, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the network-aware anomaly detection system based on big data proposed in the above embodiment.

[0034] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0035] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the network-aware anomaly detection system based on big data as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0036] In summary, the present invention constructs a complete network-aware anomaly detection system through standardized processing of multi-source heterogeneous data, privacy-preserving feature fusion based on information entropy and federated learning, dynamic spatiotemporal correlation graph modeling, and an adaptive threshold mechanism driven by anomaly probability distribution. The system not only improves the accuracy and real-time performance of anomaly identification, but also enhances its adaptability and robustness to complex network environments; at the same time, combined with causal reasoning and automated response mechanisms, it realizes closed-loop control of root cause analysis and policy disposal, effectively ensuring the security and stability of the network system.

[0037] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A network perception anomaly detection system based on big data, characterized by: include: Data acquisition module, feature fusion module, graph construction module, model calculation module, root cause reasoning module, threshold decision module and response control module; The data acquisition module receives network traffic data, device status data and system log data, and outputs a standardized feature set; The feature fusion module is connected to the data acquisition module, dynamically calculates the weight coefficient of each data source based on information entropy, performs privacy-preserving feature aggregation through the federated learning framework, and outputs a fused feature vector; The graph construction module is connected to the feature fusion module to maintain the network device node set and the communication edge set in real time, and update the spatiotemporal correlation graph according to the topology change event; The model calculation module is connected to the graph construction module, extracts topological features through the spatiotemporal graph convolutional network, updates the detection model based on the incremental learning mechanism, and outputs the anomaly probability value; The root cause reasoning module is connected to the model calculation module to construct a causal reasoning graph to calculate the node impact factor, and generates the root cause analysis result in combination with the security knowledge base; The threshold decision module is connected to the model calculation module to analyze the historical anomaly probability distribution characteristics and dynamically adjust the anomaly determination threshold parameters; The response control module is connected to the root cause reasoning module and the threshold decision module, and triggers the execution of the disposal strategy when the abnormal probability exceeds the current threshold.

2. The big data-based network perception anomaly detection system according to claim 1, characterized in that: The specific steps of receiving network traffic data, device status data and system log data and outputting a standardized feature set are as follows: Perform field cleaning and structured extraction on the raw network traffic data obtained by the receiving port to obtain a set of network traffic feature vectors. Perform numerical unification and missing value processing on the device operation status data obtained by the receiving port to obtain a set of device status feature vectors. Perform semantic parsing and keyword extraction on the system log data obtained by the receiving port to obtain a set of log feature vectors. Standardize the features in the network traffic feature vector set, the device status feature vector set, and the log feature vector set, and perform dimension alignment to obtain a standardized feature set; Substituting the standardized network traffic feature vector set, device status feature vector set, and log feature vector set into the above expression, we can obtain the compressed vector representation of the three types of data. The concatenation function is used to merge the three types of compressed feature vectors and output a complete standardized feature set.

3. The big data-based network perception anomaly detection system according to claim 2, characterized in that: The method dynamically calculates the weight coefficient of each data source based on information entropy, performs privacy-preserving feature aggregation through the federated learning framework, and outputs a fused feature vector. The specific steps are as follows: For the three types of standardized feature sets, their feature dimensions are extracted and the information entropy values ​​are calculated. The following information entropy function is used to obtain the information entropy value sequences of the three types of data sources. The information entropy values ​​of the three types of data sources are normalized using the weight distribution function to obtain the dynamic weight coefficient of each data source. ; The three types of data sources are deployed on different edge nodes, and each node runs a local feature extraction model; The following process is performed using the federated learning communication protocol: Each node receives the global model parameters sent by the central server; Use the local data set to update the local model parameters to obtain updated parameters; Perform differential privacy scrambling on the updated parameters to generate encrypted gradients; Upload the encrypted gradient to the federated coordination server; Perform weighted aggregation function on encrypted gradients and dynamic weights; Finally, update the global model parameters; The feature mapping function is used to project the feature space corresponding to the updated global model parameters and output the fused feature vector.

4. The big data-based network anomaly detection system according to claim 3, characterized in that: The real-time maintenance of the network device node set and the communication edge set, and updating of the spatiotemporal association graph according to the topology change event, are specifically carried out as follows: Extracting device identification fields and interface connection status from device status data; Add the device identifier to the initial network device node set, and establish a communication edge record for the device pairs that have communication behavior to form an initial communication edge set, and then use the initial communication edge set to construct the initial network graph structure; Parse the source IP and destination IP fields in the system log feature vector set and update the node connection relationship in the communication edge set; Identify changes in the current network device node set and communication edge set, and generate a list of topology change events; The spatiotemporal graph update mechanism is used to adjust the graph structure of each type of change in the topology change event list and update the spatiotemporal correlation graph. Specifically: for Each new node in the graph is added to the graph node set ; for For each old node in the graph, remove it from the graph node set Remove; for Each new edge in , add it to the graph edge set ; for For each old edge in , remove it from the graph edge set delete; At the same time, a timestamp is introduced to mark the active time period of each edge to construct a spatiotemporal correlation graph; The graph structure evolves dynamically over time to reflect real changes in network topology; Serialize and save the updated spatiotemporal correlation graph and output the final graph structure .

5. The big data-based network perception anomaly detection system according to claim 4, characterized in that: The method extracts topological features through the spatiotemporal graph convolutional network, updates the detection model based on the incremental learning mechanism, and outputs the abnormality probability value. The specific steps are as follows: Graph Structure Execute the graph adjacency matrix construction function to generate the graph adjacency matrix; The graph adjacency matrix construction function; Perform node mapping on the fused feature vector to generate a graph feature matrix; A spatiotemporal graph convolutional network is used to jointly process the graph adjacency matrix and the graph feature matrix to extract the network topology feature representation; The graph adjacency matrix and graph feature matrix Input into the multi-layer spatiotemporal graph convolutional network and perform graph convolution operation; After multi-layer propagation, the topological feature representation is output; The time aggregation function is used to perform sequence modeling on the topological features within a continuous time period to generate a spatiotemporal fusion feature vector; Execute the time aggregation function on the topological features of multiple consecutive time windows and output the spatiotemporal fusion feature vector; Adopting incremental learning mechanism to update detection model parameters online, obtain real-time optimized detection model, normalize the output of real-time detection model, and output abnormal probability value; Output abnormal probability value .

6. The big data-based network perception anomaly detection system according to claim 5, characterized in that: The specific steps of constructing a causal reasoning graph to calculate node impact factors and generating root cause analysis results in combination with a security knowledge base are as follows: Extract alarm record sets from the historical alarm database, where each alarm record contains timestamp, device ID, and alarm type information; Use the association rule mining algorithm Apriori to perform pattern recognition on alarm records, find frequent item sets, deduce the causal relationship between alarms, and output a list of potential causal relationships; Input the potential causal relationship list into the causal reasoning graph construction function to create a directed graph; Calculate the impact factor of each node in the causal reasoning graph; For detected anomalies, based on the key nodes involved, the security knowledge base query function is used to retrieve relevant background information from the pre-built security knowledge base; The impact factor of the key node Corresponding background information Input into the comprehensive analysis module to conduct multi-dimensional analysis, which considers factors such as impact, urgency and repair cost; Generate a root cause analysis report covering the main problem points, possible causes and recommended measures to assist operation and maintenance personnel in quickly locating the root cause of the problem and taking action.

7. The big data-based network perception anomaly detection system according to claim 6, characterized in that: The specific steps of analyzing the historical anomaly probability distribution characteristics and dynamically adjusting the anomaly determination threshold parameters are as follows: Extract the historical probability values ​​within a continuous time period from the abnormal probability value sequence, perform sliding collection in fixed time windows, and construct a historical abnormal probability set. ; For the historical anomaly probability set Apply the kernel density estimation method to construct the abnormal probability density function; Analyze the tail area of ​​the abnormal probability density function and set the initial abnormality judgment threshold; For continuous The initial anomaly judgment threshold sequence of each time window executes the trend change detection function and outputs the threshold adjustment signal; The initial anomaly determination threshold is weighted and modified in combination with trend change information to output a dynamic anomaly determination threshold; The threshold decision function is executed on the abnormal probability value output at the current moment and the dynamic abnormality judgment threshold.

8. The big data-based network perception anomaly detection system according to claim 7, characterized in that: When the abnormal probability exceeds the current threshold, the handling strategy is triggered to execute. The specific steps are: Compare the current abnormal probability value output by the model calculation module with the dynamic abnormality judgment threshold to generate an abnormality judgment result; Obtain the root cause analysis results of the current abnormal event, search for matching policy numbers in the predefined disposal policy library based on the type of each key node and its impact factor, and output a list of policy numbers; For the strategy number list, the priority sorting function is performed in combination with the key node impact factors corresponding to each strategy, and an ordered processing strategy queue is output; Input the ordered processing strategy queue into the automated execution engine, and call the processing actions corresponding to the strategy numbers in sequence; The actions performed include automatically isolating high-risk devices, starting backup links or switching redundant devices, sending SMS / email alerts, and triggering deep log collection tasks; Evaluate and log the effectiveness of each completed disposal strategy and generate a disposal feedback report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the network-aware anomaly detection system based on big data are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the network-aware anomaly detection system based on big data according to any one of claims 1 to 8 are implemented.

Images