Network security big data processing system and method based on artificial intelligence

Abstract

The invention relates to the technical field of network and information security, artificial intelligence and big data processing, and discloses a network security big data processing system and method based on artificial intelligence. The system comprises data acquisition and edge abstract, semantic normalization and feature bin, federal map representation learning, anomaly detection and causal interpretation, risk grading and response arrangement, model drift monitoring and adaptive retraining, and data governance and fit scale block. According to the method, efficient recognition and response to complex attack paths and cross-domain threats on the premise of privacy compliance are achieved through edge abstracting and anonymization, unified semantic normalization, federated time perception graph representation learning and interpretable anomaly detection and automatic disposal fusing multiple methods.

Description

Technical Field

[0001] This invention relates to the fields of network and information security, artificial intelligence and big data processing technology, and in particular to a network security big data processing system and method based on artificial intelligence. Background Technology

[0002] With the widespread application of cloud computing, the Internet of Things (IoT), and mobile internet, enterprise and institutional data such as security logs, traffic mirroring, endpoint telemetry, and threat intelligence are characterized by being multi-sourced, heterogeneous, large-scale, and highly time-sensitive. Existing technologies have disclosed various intrusion detection and situational awareness solutions, including centralized deep learning and distributed / federated learning approaches. For example, intelligent intrusion detection systems and methods based on federated learning typically improve the detection capability of new threats by training Long Short-Term Memory (LSTM) networks or convolutional models locally on each node and then aggregating them, while using differential privacy (DP) to mitigate the privacy and compliance risks brought about by data centralization. In recent years, the application of graph neural networks (GNNs) and knowledge graphs in cybersecurity defense has deepened, making it possible to model attack paths based on relationships and topology, and gradually integrating with strategies such as federated learning and differential privacy.

[0003] However, existing solutions generally have the following drawbacks: The difficulty in standardizing and normalizing multi-source heterogeneous data patterns limits the effectiveness of cross-domain correlation analysis. Centralized training of data brings privacy and compliance risks, and cross-organizational collaborative detection is difficult to implement; even in federated learning frameworks, there is still a lack of data pattern alignment and semantic layer unification for security scenarios. Deep models lack interpretability for anomaly detection, making it difficult to support evidence collection and tracing, compliance auditing, and automated processing orchestration. The drift in the distribution of incremental online data and model aging have not been systematically addressed, making it difficult to maintain a stable false alarm / false negative rate in the long term.

[0004] Therefore, this application discloses a network security big data processing system and method based on artificial intelligence to solve problems such as the difficulty in normalizing heterogeneous data, the difficulty in balancing privacy compliance and cross-domain collaboration, the inability to interpret anomaly detection, and the easy degradation of models during long-term operation. Summary of the Invention

[0005] Technical problems to be solved This invention aims to provide an artificial intelligence big data processing system and method for cybersecurity scenarios, solving problems such as data heterogeneity and difficulty in normalization, limited cross-domain collaboration, uninterpretable anomaly detection, and models not adapting to changes in data distribution in existing technologies, thereby achieving efficient, interpretable, and scalable threat discovery and response under compliance.

[0006] Technical solution To achieve the above objectives, the present invention provides the following technical solution: a network security big data processing system based on artificial intelligence, characterized in that it comprises: The data acquisition and edge summarization module is used to collect security-related data from the network and terminals, and generate probabilistic summaries and pattern description metadata at the edge. The semantic normalization and feature warehouse module is used to map heterogeneous events into a unified semantic structure and perform online / offline feature updates; The federated graph representation learning module is used to build dynamic security knowledge graphs in various data domains and perform local time-aware representation learning. It enables cross-domain collaboration without aggregating raw data through federated aggregation and combines differential privacy protection. The anomaly detection and causal explanation module is used to integrate multiple methods for anomaly detection and generate interpretable results containing a chain of evidence. The risk classification and response orchestration module is used to classify risks based on risk scores and trigger response scripts. The model drift monitoring and adaptive retraining module is used to monitor data distribution and concept drift and trigger incremental learning and federated retraining. The data governance and compliance module is used to record data lineage, audit evidence, and perform compliance verification.

[0007] Preferably, the data acquisition and edge summarization module uses minimum count sketch and superlogarithmic counting to summarize high cardinal digital segments and frequencies; direct identifiers are de-identified using salted hashing or tokenization; and aligned identifiers are generalized and suppressed to form equivalence classes to meet the k-anonymity requirement of a preset k value.

[0008] Preferably, the semantic normalization and feature repository module includes a self-describing pattern mapper that uses meta-learning to align source fields to a preset security event ontology and outputs an entity-relationship-event structure, supporting streaming updates of session sequences, graph topology, and temporal statistical features.

[0009] Preferably, the federated graph representation learning module employs joint training of graph attention and temporal encoding, and introduces differential privacy noise and knowledge distillation during cross-domain aggregation to improve generalization and privacy protection.

[0010] Preferably, the anomaly detection and causal explanation module integrates unsupervised density / isolated forest, graph temporal anomaly scoring, and policy rule constraints to output an evidence chain containing anomaly paths, key entities, and feature contributions.

[0011] Preferably, the model drift monitoring and adaptive retraining module monitors changes in input distribution based on the population stability index and the Kolmogorov-Smirnov test, and triggers incremental learning and a new round of federated aggregation when the threshold is exceeded.

[0012] Preferably, the federated aggregation employs a federated average weighted by sample size and supports personalized headers to adapt to statistically heterogeneous data distributions.

[0013] An artificial intelligence-based cybersecurity big data processing method, applied to the aforementioned artificial intelligence-based cybersecurity big data processing system, includes the following steps: S1 collects multi-source security data and generates probability summaries and pattern descriptions at the edge. S2 performs semantic normalization, entity relation extraction, and online feature update; S3 constructs a dynamic security knowledge graph within the domain and performs local time-aware graph representation learning; S4 performs privacy-preserving cross-domain aggregation and distillation of model parameters from various domains through federated learning; S5 integrates unsupervised, graph-time series, and rule-constrained multi-method anomaly detection and generates an interpretable chain of evidence. S6 classifies risks based on risk scores and arranges response scenarios accordingly; S7 monitors data distribution and concept drift and triggers incremental / retraining; S8 performs audit documentation and compliance checks and updates the knowledge base and rule set.

[0014] Preferably, in S4, a differential privacy mechanism is used to inject Laplacian or Gaussian noise into the uploaded gradient or parameters to limit the influence of single samples, and the differential privacy budget is tracked globally.

[0015] A computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, cause the processor to perform the aforementioned artificial intelligence-based network security big data processing method.

[0016] Beneficial effects Compared with existing technologies, this invention provides an artificial intelligence-based network security big data processing system and method, which has the following beneficial effects: 1. Unified Semantic Normalization and Transferable Analysis Capabilities: Through self-describing pattern mapping and security event ontology, multi-source heterogeneous logs are normalized to a unified "entity-relationship-event" semantic layer, enabling low-cost alignment of cross-domain and cross-vendor data and fundamentally reducing the manpower investment in feature engineering. Based on this, cross-domain knowledge can be transferred and combined with federated models to form zero / few-sample adaptation capabilities for new domains. Compared to traditional methods lacking a unified semantic layer, this invention can continuously accumulate ontology and mapping rule bases, reducing the marginal cost of accessing new sources and improving reusability.

[0017] 2. Privacy-Compliant Collaborative Modeling: Combining federated aggregation and differential privacy noise, this approach avoids raw data leaving the domain and enables collaborative training across multiple organizations and business domains, balancing detection performance with privacy protection. This mechanism supports multi-granularity collaboration of parameters, gradients, and knowledge distillation, and allows for differentiated DP budgets configured by domain to comply with respective compliance strategies. Compared to centralized training or unprotected federated solutions, this invention reduces the risk of data leakage while maintaining the generalization performance of the global model.

[0018] 3. Dynamic Knowledge Graph and Time-Aware Representation: This invention uses a dynamic security knowledge graph to characterize the time-varying nature of entities and relationships, combined with time-aware graph representation to learn and capture complex attack paths such as lateral movement and hidden channels. Compared to methods based on planar sequences or static features, this representation has a natural advantage in identifying causal chains across sessions and nodes. This invention also allows for cross-domain distillation of graph structure knowledge, enhancing the expressive power of low-resource domains.

[0019] 4. Interpretable Multi-Method Anomaly Detection: This method integrates unsupervised, graph-based temporal, and rule-constrained signals and performs confidence correction to avoid false positives / false negatives caused by single-model bias. By comparing interpretations and counterfactual data to generate an output evidence chain, it supports auditing and evidence review, and allows for backtracking and reproducible queries to assist analysts in re-verification. Compared to "black box" detection, this invention significantly improves the transparency and usability of decision-making.

[0020] 5. Continuous Adaptation and Stable Operation: Concept drift is monitored through statistical measures such as PSI / KS and feedback loops, triggering incremental learning and a new round of federated aggregation to maintain dynamic matching between the model and the environment. A retraining strategy with budget management is introduced to balance computational cost and performance recovery. In long-term operation, the system exhibits stronger tracking capabilities for new threat patterns and maintains relatively stable alarm quality.

[0021] 6. Implementation and Compliance Friendly: Edge summaries (minimum count sketches and superlog counts) and field anonymization reduce bandwidth and sensitive exposure while retaining valid signals for statistics and deduplication. Auditable metadata and hash chain traces are introduced during federated training to facilitate compliance checks and subsequent evidence preservation. The overall solution clearly defines module boundaries and data / model elements, contributing to full disclosure and successful implementation. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall system architecture of the present invention (data acquisition and edge summarization, semantic normalization and feature warehouse, federated graph learning, anomaly detection and interpretation, risk orchestration, drift monitoring, and data governance). Figure 2 This is a schematic diagram of the method flow of the present invention; Figure 3 A schematic diagram of the federated training and differential privacy aggregation process; Figure 4 A schematic diagram of modeling a dynamic security knowledge graph (entities, relations, and timestamps); Figure 5 This is a schematic diagram of the anomaly explanation output interface (chain of evidence, feature contribution, and suggested actions). Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] As described in the background section, there are shortcomings in the existing technology. In order to solve the above-mentioned technical problems, this application proposes a network security big data processing system and method based on artificial intelligence.

[0025] Example 1: Basic Deployment and Data Pipeline (See Appendix) Figure 1 , Figure 2 ) Deployment: Deploy lightweight collectors and edge summarization plugins on each data domain (data domain refers to an independent data management unit within the network or business boundary), and deploy semantic normalization / feature warehouses, federated coordinators (federated coordinators are used to schedule cross-domain training and aggregation), detection and interpretation services, SOAR (Security Orchestration Automation and Response) adapters and compliance modules on the central side.

[0026] Data Acquisition and Edge Summarization: Acquire Syslog (system log), Windows event log, EDR (Endpoint Detection and Response) telemetry, NetFlow (Network Flow Metadata) / PCAP (Packet Capture Data) summaries, IAM (Identity and Access Management) logs, database auditing, IOC (Threat Indicators), etc.; Generate a Count-Min Sketch (for frequency estimation) and HyperLogLog (for cardinality estimation) at the edge, and hash and k-anonymize sensitive fields (k-anonymization refers to generalizing / suppressing records to make any k indistinguishable records); and include a schema description (field name, type, hash mapping rules).

[0027] Semantic normalization: The source fields are aligned to a unified ontology (ontology, referring to the standardized entity / action / object / context / time quintuple in security events) using a meta-learning mapper, and the normalized events are output and written to the online feature warehouse (online feature warehouse, supporting streaming writing and window calculation); window aggregation generates count, proportion, rate and homogeneous distribution features within the sliding time window.

[0028] Knowledge graph construction: Nodes are defined as hosts, accounts, processes, files, and IP addresses / domain names; edges are defined as login, access, derivation, and connection; event timestamps and weights are stored. A time-aware encoder (used to map temporal context to vector space) is used to generate node / edge embeddings (see attached document). Figure 4 ).

[0029] Federated training: Each domain trains its own time-aware graph representation model (a representation model combining graph structure and time encoding) locally, uploading parameters or gradients in epochs. The server uses federated averaging (a common cross-client parameter weighting aggregation method) to aggregate the data, injecting differential privacy (DP) noise (e.g., Laplacian / Gaussian mechanisms) into the uploaded data to limit the influence of single samples (see attached). Figure 3 ).

[0030] Detection and Interpretation: This method integrates Isolation Forest (an unsupervised anomaly detection algorithm), time-series-graph anomaly scoring, and policy rules to output a risk rating. Interpretation includes critical paths (based on the shortest anomaly path in the graph), Top-K anomaly features (Top-K refers to the top K items ranked by contribution), and comparisons between normal samples and counterfactual suggestions (counterfactuals refer to hypothetical scenarios where the model output changes with minimal alteration) (see attached reference). Figure 5 ).

[0031] Response loop: High-risk events are automatically issued with handling scripts (isolation of hosts, blocking of connections, key reset, etc.), while medium and low-risk events are subject to manual review; the results are fed back for threshold calibration and continuous learning.

[0032] Drift and Retraining: Monitor the input distribution (PSI, population stability index / KS, Kolmogorov-Smirnov test) and false alarm rate to trigger micro-batch incremental updates or a new round of federated aggregation; the DP budget ε / δ (ε / δ, differential privacy loss / failure probability) is set according to compliance requirements.

[0033] Audit and Compliance: Retain training metadata, summaries, and hash chains (hash chains are chained, immutable logs) to record feature and parameter versions, ensuring traceability and adequacy of disclosure.

[0034] Verification Objectives and Demonstration Results: Verify the deployability and scalability of the end-to-end pipeline in a real network, demonstrating that with edge summarization and semantic normalization, data bandwidth consumption decreases while retaining sufficient detection signals, and that federated coordination and audit logging meet compliance requirements. Metrics include: bandwidth reduction percentage, normalization mapping accuracy / coverage, end-to-end processing latency, evidence chain reproducibility, and audit log integrity, etc. (see attached document). Figure 1 , Figure 2 ).

[0035] Example 2: Federated Graph Representation Learning and Communication Optimization (See Appendix) Figure 3 , Figure 4 ) In-domain model: A graph attention network (GAT) is used in conjunction with a temporal encoder. The objective function integrates self-supervised link prediction (link prediction, a self-supervised task based on graph structure) and anomaly contrastive learning (contrastive learning, a learning paradigm that brings positive samples closer and pushes negative samples further away) loss. A variational graph autoencoder (variational graph autoencoder, used for unsupervised representation) can be used as a pre-training module.

[0036] Cross-domain aggregation: Perform a federated average weighted by sample size; To mitigate the impact of statistical heterogeneity (statistical heterogeneity refers to different data distributions in different domains), the server performs knowledge distillation (knowledge distillation, using the teacher-student paradigm to transfer knowledge) to a lightweight student model, which is then distributed to each domain for incremental deployment.

[0037] Differential privacy and personalization: Gaussian / Laplacian noise is added to the upload end and pruning is performed (gradient pruning, limiting the gradient norm), and the server side tracks the DP budget; for domains with large differences in data distribution, personalized headers (personalized headers, domain-specific output layers) or hybrid personalized aggregation strategies are supported.

[0038] Communication compression: Top-K sparsity and quantization (fixed-point quantization, reducing bit width) are used to reduce bandwidth overhead, and adaptive learning rate (adaptive learning rate, dynamically adjusting step size according to feedback) is combined to achieve stable convergence.

[0039] This study validates the convergence and effectiveness of federated graph representation learning under statistical heterogeneity and privacy constraints, and demonstrates that communication compression significantly reduces bandwidth without significantly sacrificing detection performance. Metrics include: accuracy / recall changes compared to a centralized training or federated LSTM (Long Short-Term Memory) baseline, global loss convergence curve, number of bytes per communication round and total latency, and the performance impact curves for different ε values ​​(see attached diagram). Figure 3 ).

[0040] Example 3: Explainable Anomaly Detection and Chain of Evidence Generation (see attached document) Figure 5 ) Model fusion: Let there be an unsupervised score s_u (unsupervised anomaly score), a time-series-graph score s_g (graph time-series anomaly score), and a rule constraint s_r (rule matching score). The final risk R (final risk score) is obtained by weighting w_u, w_g, w_r and adjusting for confidence.

[0041] Explanation generation: SHAP (Shapley Additive Explanation) and IG (Integral Gradient) are used to output feature contributions; the shortest anomaly path (shortest anomaly path, the shortest suspicious link connecting the triggering event to the target entity) is calculated on the graph, and evidence chains and reproducible queries (reproducible queries, query statements that can be directly run in the data lake) are constructed in combination with rule templates (rule templates, domain knowledge entries).

[0042] Human-machine collaboration: Analyst feedback (feedback, manual annotation and processing results) is used for online threshold tuning and distillation relearning (distillation relearning, integrating new feedback knowledge into the student model), a closed-loop optimization system.

[0043] Verify the local fidelity and operability of the interpreted output, demonstrating that the chain of evidence can support evidence review and SOAR orchestration while shortening the analysis loop time. Metrics include: changes in risk scores triggered by feature deletion / zeroing tests (fidelity of interpretation), Top-K hit rate of the chain of evidence, changes in average analyst processing time and error handling rate, and interpretability consistency (see appendix). Figure 5 ).

[0044] Example 4: Parameter Recommendations and Deployment Scale (See Appendix) Figure 2 , Figure 3 ) Time window: Sliding window 5–15 minutes (can be adjusted according to business peaks and troughs), the chart is updated with decay weights for the past 7–30 days (decay weights, which decay exponentially over time).

[0045] DP budget: Global ε is set to 1–8 (depending on risk and regulations), δ is set to 1e-5–1e-7 (probability of failure), and sensitive domains (domains involving personal / confidential information) use smaller ε.

[0046] Training frequency: Local small batches are fine-tuned daily / hourly, and federated aggregation is performed daily / weekly; major drifts trigger temporary rotation training (temporary rotation training, temporarily increasing the number of federated rounds).

[0047] Resource recommendations: 8–32 core CPUs (Central Processing Units) / 1–2 GPUs (Graphics Processing Units) per domain, with the central side scaling linearly based on the number of participating domains.

[0048] To facilitate engineering deployment and scalability verification, this study demonstrates that the system maintains stable performance and acceptable latency under typical resource constraints and different privacy budgets. Metrics include: training time versus communication cost curves as the number of participating domains increases (near-linear or sublinear scalability), detection performance variations under different ε / δ values, and sensitivity analysis to alarm quality under varying window and attenuation parameters (see attached table). Figure 2 , Figure 3 ).

[0049] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A network security big data processing system based on artificial intelligence, characterized in that, include: The data acquisition and edge summarization module is used to collect security-related data from the network and terminals, and generate probabilistic summaries and pattern description metadata at the edge. The semantic normalization and feature warehouse module is used to map heterogeneous events into a unified semantic structure and perform online / offline feature updates; The federated graph representation learning module is used to build dynamic security knowledge graphs in various data domains and perform local time-aware representation learning. It enables cross-domain collaboration without aggregating raw data through federated aggregation and combines differential privacy protection. The anomaly detection and causal explanation module is used to integrate multiple methods for anomaly detection and generate interpretable results containing a chain of evidence. The risk classification and response orchestration module is used to classify risks based on risk scores and trigger response scripts. The model drift monitoring and adaptive retraining module is used to monitor data distribution and concept drift and trigger incremental learning and federated retraining. The data governance and compliance module is used to record data lineage, audit evidence, and perform compliance verification.

2. The network security big data processing system based on artificial intelligence according to claim 1, characterized in that: The data acquisition and edge summarization module uses minimum count sketch and superlogarithmic counting to summarize high cardinality segments and frequencies; direct identifiers are de-identified using salted hashing or tokenization; and quasi-identifiers are generalized and suppressed to form equivalence classes to meet the k-anonymity requirement of a preset k value.

3. The network security big data processing system based on artificial intelligence according to claim 1, characterized in that: The semantic normalization and feature warehouse module includes a self-describing pattern mapper that uses meta-learning to align source fields to a preset security event ontology and outputs an entity-relationship-event structure, supporting streaming updates of session sequences, graph topology, and time-series statistical features.

4. The network security big data processing system based on artificial intelligence according to claim 1, characterized in that: The federated graph representation learning module employs joint training of graph attention and temporal encoding, and introduces differential privacy noise and knowledge distillation during cross-domain aggregation to improve generalization and privacy protection.

5. The network security big data processing system based on artificial intelligence according to claim 1, characterized in that: The anomaly detection and causal explanation module integrates unsupervised density / isolation forest, graph temporal anomaly scoring, and policy rule constraints to output an evidence chain that includes anomaly paths, key entities, and feature contributions.

6. The network security big data processing system based on artificial intelligence according to claim 1, characterized in that: The model drift monitoring and adaptive retraining module monitors changes in input distribution based on the population stability index and the Kolmogorov-Smirnov test, and triggers incremental learning and a new round of federated aggregation when the threshold is exceeded.

7. The network security big data processing system based on artificial intelligence according to claim 1, characterized in that: The federated aggregation uses a federated average weighted by sample size and supports personalized headers to adapt to statistically heterogeneous data distributions.

8. A method for processing cybersecurity big data based on artificial intelligence, applied to a cybersecurity big data processing system based on artificial intelligence as described in any one of claims 1-7, characterized in that, Includes the following steps: S1 collects multi-source security data and generates probability summaries and pattern descriptions at the edge. S2 performs semantic normalization, entity relation extraction, and online feature update; S3 constructs a dynamic security knowledge graph within the domain and performs local time-aware graph representation learning; S4 performs privacy-preserving cross-domain aggregation and distillation of model parameters from various domains through federated learning; S5 integrates unsupervised, graph-time series, and rule-constrained multi-method anomaly detection and generates an interpretable chain of evidence. S6 classifies risks based on risk scores and arranges response scenarios accordingly; S7 monitors data distribution and concept drift and triggers incremental / retraining; S8 performs audit documentation and compliance checks and updates the knowledge base and rule set.

9. The method for processing network security big data based on artificial intelligence according to claim 8, characterized in that: S4 employs a differential privacy mechanism to inject Laplacian or Gaussian noise into the uploaded gradients or parameters to limit the influence of single samples, and performs global tracking of the differential privacy budget.

10. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, cause the processor to perform the artificial intelligence-based network security big data processing method according to any one of claims 8 or 9.

Images