Dynamic security capability intelligent scheduling system and method based on real-time log analysis

Abstract

The application discloses a kind of dynamic security capability intelligent scheduling system and method based on real-time log analysis, it is related to log security scheduling technical field;Real-time log acquisition module is deployed edge intelligent Agent, utilizes eBPF probe and adaptive filtering technology, guarantees the capture rate of key event under high load;Multi-dimensional feature extraction module is based on Transformer-XL architecture and dynamic attention mechanism, accurately extracts log threat semantics;Security situation awareness module constructs CNN-DBN hybrid model, realizes unknown threat early warning 30 minutes in advance;Dynamic capability scheduling module relies on digital twin technology and reinforcement learning algorithm, and the response of more than 100 kinds of heterogeneous security components is less than 50ms across domain.The application can realize second-level attack detection, automatically cooperate security component, provide visual decision support, significantly improve network security protection level, reduce operation and maintenance cost, effectively guarantee enterprise network security.

Description

Technical Field

[0001] This invention relates to the field of log security scheduling technology, and in particular to a dynamic security capability intelligent scheduling system and method based on real-time log analysis. Background Technology

[0002] Under the dual challenges of policy-driven and real-world threats to cybersecurity, enterprise cybersecurity construction faces severe tests. However, current enterprise cybersecurity protection has significant shortcomings: On the one hand, external network attack technologies iterate rapidly, and traditional passive defense models (such as single intrusion detection and virus scanning) are insufficient to cope with new and complex attacks, creating a generational gap with advanced attack technologies. This results in lagging enterprise security protection capabilities, making it impossible to effectively prevent, resist, and handle various network threats. On the other hand, enterprise internal network security management systems have vulnerabilities. The security capabilities of business systems such as document management, document receipt management, and book order management are uneven, lacking rapid perception, intelligent analysis, and accurate early warning mechanisms for the security situation. Cloud information asset management has low transparency and insufficient risk quantification, and there are management gaps at the operational level, including security assessments, policy issuance, and full lifecycle management of IT projects.

[0003] Specifically, existing protection technologies have significant limitations: in threat detection, they lack the ability to perform intelligent and in-depth analysis of network traffic, making it difficult to accurately identify malicious traffic and trace attack behavior from big data; in proactive defense, they have not built a mimicry defense system, making it impossible to proactively interfere with attackers through deception and trapping techniques to form a dynamic protection pattern of "attack and defense confrontation"; in asset and operation management, information assets are scattered and poorly managed, security systems are difficult to implement, assessment mechanisms are imperfect, and major IT projects lack full-process security supervision, resulting in long-term latent cybersecurity risks that, once they erupt, will severely impact the continuity of business and the integrity of data.

[0004] Therefore, in order to respond to policy requirements and fill the gaps in technology and management, it is urgent to build a proactive defense platform that integrates intelligent traffic analysis, mimicry defense, dynamic asset management and control, and full-process operation supervision. This platform will upgrade the security protection model from passive "post-event handling" to proactive "pre-event prevention and in-event countermeasures," comprehensively enhance the resilience and effectiveness of the enterprise's network security system, and build a solid security barrier for core business systems such as document issuance, document receipt, and book orders. Summary of the Invention

[0005] The present invention proposes a dynamic security capability intelligent scheduling system and method based on real-time log analysis to solve the problems mentioned in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a dynamic security capability intelligent scheduling system based on real-time log analysis, comprising:

[0007] Real-time log collection module: Deploys an edge agent, adopts eBPF dynamic probe and adaptive threshold filtering technology, realizes log collection through kernel bypass mechanism, and improves the data compression ratio to 20:1 by combining zero-copy transmission protocol; The module uses dynamic sampling rate adjustment algorithm to automatically switch sampling mode according to the time series decay characteristics of abnormal indicator I, and prioritizes the collection of high-risk logs when CPU load is high.

[0008] Abnormal indicators The calculation is based on a weighted average of three dimensions: CPU utilization percentage (c), network traffic mutation rate (f), and connection anomaly rate (k). The formula is as follows: Each dimension is normalized to the [0,1] interval using min-max normalization; attenuation coefficient The value is fixed at 0.3, the backtracking period n is 10 time steps, and the duration of each time step is 1 second; when the CPU load exceeds 80%, the sampling rate decays exponentially according to the cumulative value of the abnormal indicators, and only high-risk log entries that hit the threat feature library are retained, while the remaining logs are downsampled proportionally.

[0009] eBPF probes are mounted on three core kernel functions: tcp_v4_connect, sys_execve, and file_open. The collected fields include source and destination IPs, ports, process IDs, operation paths, and timestamps. Kernel bypass uses DPDK to take over the network interface card data plane to collect network logs, while storage-type system logs are read zero-copy by directly accessing NVMe devices via SPDK. Both types of data are aggregated into the ring buffer of the edge agent. Critical events are defined as log entries matching MITREATT & CKTTPs, including port scans, privilege escalation operations, abnormal external connections, and sensitive file access. During adaptive threshold filtering, critical events are forcibly retained in a whitelist and excluded from traffic filtering.

[0010] Multidimensional feature extraction module: It adopts a 12-layer Transformer-XL architecture to extract long-distance dependency features of logs, and maps log semantics to a 512-dimensional vector space through knowledge graph embedding technology; it innovatively designs a dynamic attention weighting mechanism, which performs hyperbolic tangent transformation on the hidden state vector h and the context vector c through weight matrices W, U, and V, and then generates attention weights through the softmax function, which automatically strengthens the threat-related semantic representation.

[0011] The context vector c is the global semantic aggregation vector of the previous time step, with the same dimension as the hidden state vector h, both being [sequence length, hidden layer dimension], and the hidden layer dimension is fixed at 512; the weight matrices U and V are both [512, 512] in dimension, and W is [number of attention heads, 512] in dimension, with the number of attention heads set to 8; the dynamic attention weighting mechanism dynamically adjusts the weight offset according to the threat matching degree of the current log segment. When a threat keyword is detected, the attention weight at the corresponding position is multiplied by a gain coefficient of 1.5, and the gain coefficient increases linearly with the threat level.

[0012] The security knowledge graph includes four types of entities: IP address, domain name, attack type, and vulnerability number, as well as three types of relationships: attack-vulnerability exploitation, IP-attack initiation, and domain name-IP association. The knowledge graph is embedded using the TransE algorithm, with the training dataset consisting of a public threat intelligence database and enterprise historical attack logs. The embedding dimension is uniformly 512 dimensions. Log entity extraction uses a combination of regular expression matching and named entity recognition. After extraction, the entity is mapped to the corresponding entity vector, which is then concatenated with the output features of Transformer-XL to obtain the final feature vector.

[0013] Security Situation Awareness Module: Constructs a CNN-DBN hybrid prediction model, utilizes Monte Carlo simulation algorithm, and combines the ATT-CK framework to build a dynamic attack graph; the module continuously updates the threat knowledge base through an incremental learning mechanism to achieve early warning of unknown threats;

[0014] The CNN-DBN hybrid model adopts a serial fusion structure, with the local feature vector output by the CNN serving as the input to the DBN. The CNN consists of three convolutional layers with a kernel size of 3×1 and two max pooling layers, with an output dimension of 256. The DBN consists of two restricted Boltzmann machines with 128 and 64 neurons in the hidden layers, respectively. The warning time window is set to take a 30-minute log sequence as input and output the threat probability at each time point in the next 30 minutes, with a time step of 1 minute. The model is trained using supervised pre-training and unsupervised fine-tuning.

[0015] The attack impact quantification value E is obtained by summing three indicators after normalization: asset value weight, business interruption duration, and data leakage amount, with a value range of [0, 10]; risk sensitivity coefficient Based on the enterprise's security level settings, the protection unit is set to 0.8 for Level 1 and 0.6 for Level 2. The Bayesian network includes three types of nodes: attack type, threat level, and anomaly characteristics. The conditional probability table between nodes is generated by statistical analysis of historical attack data. The number of Gibbs sampling iterations is set to 500, with the first 100 iterations discarded as a warm-up period. After the sampling converges, the posterior threat probability is output.

[0016] Dynamic Capability Scheduling Module: Based on digital twin technology, a containerized capability pool of heterogeneous security components is constructed. The scheduling strategy is generated by the reinforcement learning-Monte Carlo tree search (RL-MCTS) joint optimization algorithm. The edge-cloud state synchronization is achieved through a distributed consensus protocol. The algorithm balances the exploration and utilization of the defense strategy by superimposing exploration terms through the state-action value function Q(s,a).

[0017] The RL-MCTS joint algorithm employs deep reinforcement learning to train the state-action value function. The Q-value is used as the prior value of MCTS search to guide the tree search process; the optimal action output by MCTS in each round of search is stored in the replay pool as an experience sample for reinforcement learning, and the Q network is iteratively optimized; the 512-dimensional state vector contains 128-dimensional traffic features, 128-dimensional threat features, and 256-dimensional resource state features; the action space encodes 256 discrete actions for starting and stopping 128 security components and configuring rules, and uses one-hot vector representation.

[0018] The reinforcement learning policy network takes a 512-dimensional state vector *s* as input and outputs the probability distribution of each action; the policy calculation formula is: Where U is [256, 512], W is [action space dimension, 256], and temperature parameter The initial value is 0.5, which decays linearly with each training epoch; priority index in priority experience replay. Take 0.6, a small constant. Take 1e-6, sort the samples by TD error from largest to smallest, and select the top 20% for priority playback.

[0019] Feedback optimization module: It adopts a meta-learning-federated learning hybrid framework, and the local model adapts to new threats through a meta-gradient update mechanism; it combines differential privacy technology to achieve cross-domain data aggregation.

[0020] The meta-learning task is defined as a threat detection task with different attack types. Eight attack types are randomly selected as the meta-batch in each round of meta-training. The meta-loss function is the mean cross-entropy loss on the test set for each task. Federated learning global aggregation uses a weighted average method, with weights determined by the proportion of data volume in each node. Global parameter fusion coefficients are used. The meta-learning coefficient is set to 0.3. Set the local learning rate to 0.2. Set the value to 0.001; the model completes a round of global aggregation and meta-update every 24 hours.

[0021] Differential privacy, as supplemented by gradient compression, employs a Gaussian noise mechanism to inject noise before local gradient upload. The noise standard deviation is determined by the privacy budget. The gradient sensitivity is calculated and limited to 1.0 by the gradient clipping threshold. Gradient compression adopts the Top-k sparsity method, in which only the top 10% of gradient parameters by magnitude are retained for transmission in each round, and the remaining gradient parameters are set to zero. After the transmission is completed, the untransmitted gradients are accumulated in the local cache.

[0022] Furthermore, it also includes:

[0023] The orchestration and decision-making module constructs a security capability knowledge graph and uses a graph neural network (GNN) to solve for defense configurations through weighted activation operations of adjacency matrices and node feature matrices. It employs a multi-objective optimization algorithm to balance defense effectiveness and resource consumption. It automatically generates an explanatory decision report containing defense paths, resource consumption, and effect predictions by simulating attack chain evolution through a digital twin sandbox. The adaptive defense module uses a dynamic topology reconstruction algorithm based on real-time threat energy values ​​and employs a hot restart mechanism to reconstruct the security capability network topology within 100ms. It also automatically adjusts defense resources through a microservice elastic scaling strategy.

[0024] The security capability knowledge graph nodes include three categories: security components, protection scenarios, and resource consumption, with a node feature dimension of 64. The graph neural network adopts a 2-layer GCN model, and the adjacency matrix is ​​generated by the linkage and dependency relationships between components. Topology transformation operators are used. The threat energy value is calculated by multiplying the attack intensity and the scope of impact. In microservice elastic scaling, the SLA is defined as the threshold for processing requests per second, with a value of 10,000 requests / second and a single instance processing capacity P of 2,000 requests / second.

[0025] The SHAP value interpretation framework is designed for the CNN-DBN model of security situation awareness. It uses the KernelSHAP algorithm to calculate feature contribution and sets the sample size to 1000. Counterfactual reasoning adopts a gradient-guided search method and uses Euclidean distance as the distance metric. In the parallel ant colony search algorithm, the log space node is a single log entry, the path is a time-series log sequence, the pheromone evaporation coefficient is set to 0.1, and the number of ants is set to 32. The PBFT consensus node consists of 5 cross-domain security nodes, and the evidence data includes attack chain hash, timestamp, and node signature.

[0026] The interpretability enhancement module employs the SHAP value interpretation framework to generate a decision report by calculating the contribution of each feature to the model output; it uses counterfactual reasoning techniques to simulate changes in prediction results after altering feature values, supporting attack attribution, defense effectiveness, and risk impact explanation; the cross-domain threat hunting module utilizes a parallel ant colony search algorithm to mine attack patterns in parallel by simulating the behavior of ants releasing and sensing pheromones in heterogeneous log spaces; and it combines the PBFT consensus algorithm to construct a cross-domain threat chain graph.

[0027] Furthermore, the reinforcement learning strategy of the dynamic capability scheduling module performs a linear transformation on the 512-dimensional state vector and the safety capability action space, and generates a policy probability distribution through the hyperbolic tangent activation function and the softmax function. It adopts the priority experience replay technique, adjusts the sample replay priority according to the TD error, optimizes the parameters, and reduces the scheduling decision delay.

[0028] Furthermore, the Bayesian network inference of the security situation awareness module is based on the conditional probability product of the threat state vector and the feature vector. It utilizes Gibbs sampling technology to accelerate the calculation of posterior probability by iteratively sampling the conditional distribution of each variable when all variables are fixed.

[0029] Furthermore, the meta-gradient update mechanism of the feedback optimization module is as follows: the local model parameters are adjusted by a weighted combination of the local loss function gradient and the difference of the global meta-parameters, and gradient compression technology is used to retain only the parameters with the highest gradient magnitude for transmission.

[0030] Furthermore, the kernel bypass technology of the real-time log acquisition module is implemented through the DPDK data plane development kit, and the zero-copy transmission protocol adopts the SPDK storage performance development kit; through the adaptive batch processing mechanism, the data processing batch size is dynamically adjusted according to the current request rate, system processing capacity, and queue length, so as to control the CPU utilization rate while ensuring the data processing speed.

[0031] A method for applying the aforementioned intelligent scheduling system for dynamic security capabilities based on real-time log analysis includes:

[0032] Real-time log collection steps: Logs are collected using eBPF probes and adaptive threshold filtering technology based on historical traffic mean μ and three times the standard deviation σ. DPDK / SPDK technology is used to achieve kernel bypass and zero-copy transmission of aggregated data.

[0033] Multidimensional feature extraction steps: A 512-dimensional feature vector was constructed using the Transformer-XL architecture and dynamic attention weighting mechanism, with an F1 score of 0.96;

[0034] Security situation awareness steps: Run the CNN-DBN model and use the Monte Carlo simulation algorithm to predict the probability of threats within 30 minutes to provide early warning of unknown threats;

[0035] Dynamic capability scheduling steps: Generate scheduling strategies based on the RL-MCTS algorithm, and optimize the allocation of defense resources by superimposing exploration terms through state-action value functions;

[0036] Feedback optimization steps: The meta-learning-federated learning framework is adopted, and the model is iterated through the meta-gradient update mechanism, which is updated once every 24 hours.

[0037] Furthermore, it also includes:

[0038] The decision-making process involves: constructing a security capability knowledge graph, using graph neural networks to solve for the optimal defense configuration, and employing a multi-objective optimization algorithm to balance defense effectiveness and resource consumption; simulating the evolution of the attack chain through a digital twin sandbox to generate an explanatory decision report that includes defense paths, resource consumption, and effect predictions; and dynamically reconstructing the defense topology based on real-time threat energy values ​​and automatically adjusting defense resources through microservice elastic scaling.

[0039] Explanatory enhancement steps: The SHAP value interpretation framework is used to generate decision reports. By calculating feature contribution and counterfactual reasoning, it supports the explanation of attack attribution, defense effectiveness, and risk impact. The cross-domain threat hunting step uses a parallel ant colony search algorithm to mine attack patterns in heterogeneous log spaces. Combined with blockchain evidence storage technology, it constructs a cross-domain threat chain graph, reducing the APT attack discovery time to 2 hours.

[0040] Compared with existing technologies, the beneficial effects of this invention are:

[0041] By collecting and analyzing network log data in real time, the system can promptly detect various security threats and respond quickly. For example, when facing a DDoS attack, the system can identify attack characteristics in a short time and automatically allocate corresponding security resources for defense, effectively mitigating the impact of the attack on the enterprise network. Compared with traditional security protection technologies, this patented system can reduce attack detection time from minutes to seconds, greatly improving security response speed.

[0042] The system described in this application enables intelligent collaboration and linkage between security devices and systems, thereby improving overall security protection effectiveness. Through a dynamic capability scheduling module, the system can automatically select and configure the most suitable security components for defense based on real-time security conditions. For example, upon detecting an SQL injection attack, the system can automatically invoke security components such as WAF and IPS for collaborative defense, forming a multi-layered security protection system. This intelligent collaboration and linkage mechanism not only improves the effectiveness of security protection but also reduces the complexity and cost of security management.

[0043] The system described in this application exhibits excellent scalability and adaptability, enabling it to address ever-evolving security threats. Through a feedback optimization module, the system can continuously learn and improve its security analysis and defense capabilities. For example, the system can automatically adjust security policies and rules based on historical attack data, enhancing its ability to identify and defend against unknown threats. Furthermore, the system supports distributed deployment and horizontal scaling, adapting to the continuous expansion of enterprise network size and the evolving security needs.

[0044] The system described in this application provides visualized security situation awareness and decision support, helping security operations personnel better understand and respond to security threats. Through the security situation awareness module, the system can transform complex security data into intuitive charts and reports, enabling security operations personnel to quickly grasp the security status of the enterprise network. Simultaneously, the system also provides intelligent security decision suggestions, helping security operations personnel formulate more scientific and reasonable security strategies and measures. This visualized and intelligent security management approach significantly improves the efficiency and quality of security operations. Attached Figure Description

[0045] Figure 1 This is a schematic block diagram of the intelligent scheduling system for dynamic security capabilities based on real-time log analysis proposed in this invention.

[0046] Figure 2 This is a schematic block diagram of the intelligent scheduling method for dynamic security capabilities based on real-time log analysis proposed in this invention.

[0047] Figure 3 A performance comparison chart of different systems in attack detection;

[0048] Figure 4 This is a graph showing the resource utilization of the system under different network traffic conditions. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0051] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0052] Reference Figures 1 to 4 : A detailed implementation of a dynamic security capability intelligent scheduling system and method based on real-time log analysis

[0053] Real-time log collection module implementation method

[0054] In actual deployment, edge intelligent agents are deployed at network edge nodes such as enterprise data center entrances and exits, and branch office gateways. The agents use eBPF dynamic probe technology to directly obtain multi-source log data such as network traffic, system operation, and application operation at the kernel level, avoiding the performance loss caused by data transmission from kernel space to user space.

[0055] The system will collect log traffic data from the past 7 days and calculate the average traffic. (Unit: bytes / second, representing the average level of historical log traffic) and standard deviation (Unit: bytes / second, reflecting the dispersion of historical log traffic data; the larger the value, the more drastic the traffic fluctuation), based on the formula. Dynamically set filtering thresholds (Th: filtering threshold, in bytes per second, used to determine whether log data is noise; when real-time log traffic exceeds this threshold, the system considers it a possible anomaly) to effectively filter noisy logs generated by normal business fluctuations. For example, during a major promotional event on an e-commerce platform, this threshold setting filtered out 92% of non-threat-related logs.

[0056] When CPU load exceeds 80%, the dynamic sampling rate adjustment algorithm is activated. Where SR(t) is the sampling rate at time t (unit: times / second, representing the frequency of log collection at the current time), and SR0 is the initial sampling rate (unit: times / second, the default log sampling frequency when the system starts). Here, I(ti) is the attenuation coefficient (dimensionless, used to control the rate of attenuation of the sampling rate as the anomaly index changes; its value is typically between 0 and 1), I(ti) is the anomaly index at time ti (dimensionless, calculated by comprehensively analyzing multi-dimensional data such as CPU utilization and network traffic mutations; a larger value indicates a higher degree of system anomaly), and n is the backtracking period (representing the number of time points within the historical time window used to calculate the anomaly index). At this time, the system prioritizes collecting logs related to attack behavior. Simultaneously, it utilizes kernel bypass technology to achieve nanosecond-level log collection and leverages a zero-copy transmission protocol to increase the log data compression ratio to 20:1. Testing showed that even under extreme conditions of 10Gbps network traffic and 90% CPU load, the critical event capture rate remained at 99.99%, and CPU utilization was kept below 60%.

[0057] Implementation of Multidimensional Feature Extraction Module

[0058] A 12-layer Transformer-XL architecture with an 8-head attention mechanism is used to process the collected logs. First, the raw logs are segmented, then transformed into vector form using word embedding technology. Knowledge graph embedding technology is then used to map entities such as IP addresses and user accounts in the logs to a 512-dimensional security threat vector space.

[0059] Using a dynamic attention weighting mechanism, based on the formula Calculate the attention weights. Here, α is the attention weight vector (its dimension is the same as the log feature dimension; each element represents the importance of the corresponding feature, with larger values ​​indicating greater importance in the current analysis); W, U, and V are weight matrices (W's dimension is [number of attention heads, hidden layer dimension], U's dimension is [hidden layer dimension, hidden layer dimension], and V's dimension is [context vector dimension, hidden layer dimension], trained and optimized using a large amount of historical attack log data, used for linear transformation of the vectors); h is the hidden state vector (its dimension is [sequence length, hidden layer dimension], containing feature information of the log sequence after processing by the Transformer layer); and c is the context vector (its dimension is [context vector dimension], providing global context information related to the current processing position).

[0060] In tests on log datasets containing ransomware attacks, the module achieved an accuracy of 98% in identifying attack keywords and an F1 score of 0.96, representing improvements of 15% and 12% respectively compared to the traditional LSTM model.

[0061] Implementation of Security Situation Awareness Module

[0062] A CNN-DBN hybrid prediction model is constructed, in which the CNN part consists of 3 convolutional layers and 2 pooling layers to extract local information of log features; while DBN is used to model the dynamic evolution of threats in time series data.

[0063] Using Monte Carlo simulation algorithm Calculate the threat probability. Here, P is the probability of a threat occurring (ranging from 0 to 1; the closer the value is to 1, the higher the likelihood of the threat occurring); β is the risk sensitivity coefficient (dimensionless; enterprises can set it according to their own security needs, such as 0.8 for financial enterprises and 0.6 for internet enterprises; the larger the value, the more sensitive the enterprise is to risk); E is the quantified value of the attack impact (derived through a comprehensive assessment of factors such as potential asset losses, business interruption time, and data leakage caused by the attack; the unit is the comprehensive impact unit, and the larger the value, the greater the potential harm of the attack).

[0064] By combining the ATT&CK framework to construct a dynamic attack graph, the system automatically updates its threat knowledge base every hour. In a government agency's network, this module successfully provided an early warning of a zero-day vulnerability attack 30 minutes in advance, with a false positive rate of only 0.1%, while traditional IDS systems have a false positive rate as high as 15%.

[0065] 0-day vulnerability attack 30 minutes 0.1% none 15% APT attack 25 minutes 0.3% 5 minutes 20%

[0066] The table above compares the performance of our system and traditional IDS in two typical attack scenarios. For zero-day vulnerability attacks, traditional IDS lacks a relevant signature database and cannot issue warnings, with many normal behaviors being misidentified as attacks (false positive rate 15%). Our system, through Monte Carlo simulation algorithms that comprehensively consider risk sensitivity and attack impact, combined with dynamic attack graphs, can identify abnormal trends in advance. In APT attack scenarios, while traditional IDS can detect some attack behaviors, the high false positive rate forces security personnel to spend a significant amount of time investigating invalid alerts. Our system, with its deep feature extraction and continuous learning capabilities from logs, achieves more accurate warnings, significantly reducing security operation and maintenance costs and potential losses.

[0067] Implementation of Dynamic Capability Scheduling Module

[0068] Based on digital twin technology, over 100 heterogeneous security components, including WAF, IPS, and honeypots, are virtualized to construct a containerized security capability pool. The RL-MCTS joint optimization algorithm is employed, based on the formula... Generate scheduling strategies. Here, π represents the final security capability scheduling strategy (a set of schemes containing information such as the activation of specific security components and parameter configuration); Q(s,a) is the state-action value function (the value represents the expected long-term benefit of performing action a in state s, expressed in units of benefit), s is the current system state (a vector containing multi-dimensional information such as network traffic, threat level, and resource status), a is the executable security capability action (such as enabling a certain type of WAF, adjusting IPS interception rules, etc.); c is the exploration coefficient (dimensionless, used to balance strategy exploration and utilization, typically ranging from 0.1 to 1, with larger values ​​indicating a greater inclination to explore new strategies); N(s) is the number of times state s is accessed (unit: times, recording the frequency of the system state being experienced), and N(s,a) is the number of times the state-action pair (s,a) is accessed (unit: times, recording the number of times action a is performed in state s).

[0069] The Raft distributed consensus protocol is used to synchronize the state of edge nodes with the cloud control center. In a real-world test conducted by an internet company, in the face of a sudden DDoS attack, the system completed defense strategy scheduling within 40ms, handling 200,000 scheduling requests per second, and increasing the attack blocking rate from 70% to 95% compared to traditional solutions.

[0070] Feedback optimization module implementation method

[0071] A hybrid MAML-FL framework is used to deploy local models at various network nodes, including branches and subsidiaries. The local models are based on the formula... Perform meta-gradient update. Among them, The parameters of the local model after time t (a set of parameters containing information such as weights and biases of each layer of the model); These are the parameters of the local model at time t-1; The learning rate (dimensionless, controls the step size of parameter updates, typically ranging from 0.001 to 0.1; a larger value results in faster parameter updates, but an excessively large value may cause the model to fail to converge). The gradient of the loss function (dimensions and parameters) is calculated based on local data Di (unit: records, which is the set of security log data collected by local nodes). Consistent, indicating the direction of parameter updates); β is the global parameter fusion coefficient (dimensionless, used to adjust the degree of influence of global parameters on local model updates, with a value range of 0-1, the larger the value, the greater the influence of global parameters); These are the parameters of the global model at time t-1; The meta-learning coefficients (dimensionless, controlling the contribution of the meta-learning loss to parameter updates, with values ​​ranging from 0 to 1). The gradient of the meta-learning loss function (with the same dimension as the parameter θ).

[0072] Combined with differential privacy technology (privacy budget) =0.5 (dimensionless, used to control the strength of data privacy protection; the smaller the value, the higher the level of privacy protection) for cross-domain data aggregation, the model iterates every 24 hours. After being applied in the multi-branch network of a multinational corporation, the attack detection accuracy improved by 8% per week, and the generalization ability of unknown attacks improved by 40%.

[0073] Implementation of Intelligent Orchestration Decision and Adaptive Defense Module

[0074] The intelligent orchestration decision-making module constructs a security capability knowledge graph and utilizes graph neural networks. Solve for the optimal defense configuration. Here, G is the output of the graph neural network (a graph structure containing updated node features and weights of inter-node relationships); A is the adjacency matrix (dimensions [number of nodes, number of nodes], used to describe the connections between nodes in the graph, with elements being 0 or 1, where 1 indicates two nodes are connected); X is the node feature matrix (dimensions [number of nodes, feature dimension], containing attribute information of various security components, network nodes, and other entities); and W is the weight matrix (dimensions [feature dimension, output dimension], used to transform node features). Activation functions (such as ReLU, Sigmoid, etc., are used to introduce nonlinear factors, enabling the network to learn complex relationships).

[0075] Employing a multi-objective optimization algorithm Balancing resource input with defense effectiveness. Here, O is the final optimization objective value (dimensionless, comprehensively reflecting the achievement of multiple objectives); wi is the weight coefficient of the i-th objective (dimensionless, used to balance the importance of different objectives). =1); f i (x) represents the i-th objective function (e.g., f1(x) represents the resource utilization objective function, f2(x) represents the defense effectiveness objective function, and x is a decision variable, including information such as security component selection and deployment location). By simulating the attack chain evolution process using a digital twin sandbox, the generated strategy improved resource utilization by 40% in a simulated attack on a bank's network.

[0076] The adaptive defense module uses the formula based on the real-time threat energy value E(t). Dynamically reconstruct the defense topology. Here, G(t+1) is the defense topology at time t+1 (a graph structure describing the connection relationships of security components, traffic flow, etc.); G(t) is the defense topology at time t. This defines the topology transformation operator (a set of rules defining how to adjust the defense topology based on threat energy values); T(E(t)) is the topology transformation function generated based on the threat energy value E(t) (unit: energy unit, calculated by comprehensively considering factors such as attack intensity and impact range). This is combined with the microservice elastic scaling formula. Automatically adjust defense resources. Here, N(t) is the number of microservice instances required at time t (unit: instances); Q(t) is the request rate at time t (unit: requests / second); SLA is the Service Level Agreement (unit: requests / second, a defined service quality standard); P is the single-instance processing capacity (unit: requests / second), achieving a 95% success rate in defending against unknown vulnerabilities.

[0077] Implementation of the Explainability Enhancement and Cross-Domain Threat Hunting Module

[0078] The interpretability enhancement module uses the SHAP value interpretation framework. Generate multi-perspective decision reports. Among them, denoted as SHAP (unit: interpreted units, the numerical value represents the contribution of the feature to the model's prediction result); S is the feature subset (any subset of all feature sets N); |S| is the number of elements in subset S; |N| is the number of elements in the entire feature set N; v(S) is the output value of the model when the feature set is S (which can be the prediction probability, classification result, etc.).

[0079] Through counterfactual reasoning This system assists security personnel in attack attribution and defense effectiveness analysis. Here, CF(x) represents the counterfactual interpretation of input x (a new input similar to the original input x but capable of changing the model's output); x' is the counterfactual input; d(x,x') is the distance metric between x and x' (which can be Euclidean distance, Manhattan distance, etc., used to measure the degree of difference between the two inputs); and f(x) is the model's prediction of input x. In a certain malware attack incident, the explanation confidence reached 90%.

[0080] The cross-domain threat hunting module utilizes a parallel ant colony search algorithm. Parallel mining of attack patterns in heterogeneous log spaces. t represents the pheromone increment on path (i,j) at time t (unit: pheromone unit, used to simulate the intensity of information left by ants on the path; the larger the value, the more likely the path is to be an attack path); m is the number of ants (i.e., the number of threads for parallel search). This represents the pheromone increment left by ant k on path (i,j) at time t. By combining blockchain evidence storage technology using the PBFT consensus algorithm to construct a cross-domain threat chain graph, the attack detection time in cross-border APT attack tracking is shortened to 2 hours.

[0081] Implementation of Reinforcement Learning Strategy for Dynamic Capability Scheduling Module

[0082] A linear transformation is performed on the 512-dimensional state vector s containing information such as traffic characteristics, threat level, and resource status, and the security capability action space a, using the formula... Generate the policy probability distribution. Wherein, This is the policy probability distribution in state s based on parameter θ (a set of parameters containing information such as weights and biases of each layer of the model). The dimension is the same as the action space a, and each element represents the probability of the corresponding action being selected. Temperature is a dimensionless parameter used to adjust the randomness of the strategy; its value is usually between 0.01 and 1, and the smaller the value, the more deterministic the strategy. W, U, and V are weight matrices (optimized through training and used to perform linear transformations on the input vector).

[0083] Employing priority experience playback technology The samples are filtered and replayed. Here, pi represents the replay priority of sample i (dimensionless; a larger value indicates a higher probability that the sample will be replayed first). The TD error for sample i (unit: error unit, reflecting the difference between the predicted value and the true value); α is a small constant (unit: same as TD error unit, used to prevent priority from being 0); α is the priority exponent (dimensionless, used to adjust the sensitivity of priority to TD error, usually between 0.5 and 1), optimizing the strategy parameters to reduce scheduling decision delay to 30ms.

[0084] Implementation of Bayesian Network Inference for Security Situation Awareness Module

[0085] The Bayesian network in the security situation awareness module performs inference based on the conditional probability product of the threat state vector Et and the feature vector Ft, utilizing Gibbs sampling techniques. By iteratively sampling the conditional distribution of each variable while other variables are fixed, the calculation of the posterior probability is accelerated. Let i be the value of variable i at the t-th sampling (depending on the specific reasoning problem, the variable can be the attack type, attack time, etc.). For other variables (Representing all variables except variable i) is fixed at the (t-1)th sampling value, and is the conditional probability distribution of variable i (a probability function describing the likelihood of variable i taking a value under given conditions). Reducing the threat prediction delay to 10ms significantly improves the efficiency of unknown threat identification, achieving a recognition rate of 98%.

[0086] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A dynamic security capability intelligent scheduling system based on real-time log analysis, characterized in that, include: Real-time log collection module: Deploys an edge agent, adopts eBPF dynamic probe and adaptive threshold filtering technology, realizes log collection through kernel bypass mechanism, and improves the data compression ratio to 20:1 by combining zero-copy transmission protocol; The module uses dynamic sampling rate adjustment algorithm to automatically switch sampling mode according to the time series decay characteristics of abnormal indicator I, and prioritizes the collection of risk logs when CPU load is high. Multidimensional feature extraction module: It adopts a 12-layer Transformer-XL architecture to extract long-distance dependency features of logs, and maps log semantics to a 512-dimensional vector space through knowledge graph embedding technology; it innovatively designs a dynamic attention weighting mechanism, which performs hyperbolic tangent transformation on the hidden state vector h and the context vector c through weight matrices W, U, and V, and then generates attention weights through the softmax function, which automatically strengthens the threat-related semantic representation. Security Situation Awareness Module: Constructs a CNN-DBN hybrid prediction model, utilizes Monte Carlo simulation algorithm, and combines the ATT-CK framework to build a dynamic attack graph; the module continuously updates the threat knowledge base through an incremental learning mechanism to achieve early warning of unknown threats; Dynamic Capability Scheduling Module: Based on digital twin technology, a containerized capability pool of heterogeneous security components is constructed. The scheduling strategy is generated by the reinforcement learning-Monte Carlo tree search (RL-MCTS) joint optimization algorithm. The edge-cloud state synchronization is achieved through a distributed consensus protocol. The algorithm balances the exploration and utilization of the defense strategy by superimposing exploration terms through the state-action value function Q(s,a). Feedback optimization module: It adopts a meta-learning-federated learning hybrid framework, and the local model adapts to new threats through a meta-gradient update mechanism; it combines differential privacy technology to achieve cross-domain data aggregation.

2. The intelligent scheduling system for dynamic security capabilities based on real-time log analysis according to claim 1, characterized in that, Also includes: Orchestration and Decision Module: Constructs a security capability knowledge graph, uses a graph neural network (GNN) to solve for defense configuration through weighted activation operations of adjacency matrix and node feature matrix, and adopts a multi-objective optimization algorithm to balance defense effectiveness and resource consumption; automatically generates an explanatory decision report containing defense path, resource consumption, and effect prediction by simulating attack chain evolution through a digital twin sandbox. The adaptive defense module uses a dynamic topology reconstruction algorithm based on real-time threat energy values ​​and employs a hot restart mechanism to reconstruct the security capability network topology within 100ms. It also automatically adjusts defense resources through a microservice elastic scaling strategy.

3. The intelligent scheduling system for dynamic security capabilities based on real-time log analysis according to claim 1, characterized in that, Also includes: The interpretability enhancement module employs the SHAP value interpretation framework to generate a decision report by calculating the contribution of each feature to the model output; it uses counterfactual reasoning techniques to simulate changes in prediction results after altering feature values, supporting attack attribution, defense effectiveness, and risk impact explanation; the cross-domain threat hunting module utilizes a parallel ant colony search algorithm to mine attack patterns in parallel by simulating the behavior of ants releasing and sensing pheromones in heterogeneous log spaces; and it combines the PBFT consensus algorithm to construct a cross-domain threat chain graph.

4. The intelligent scheduling system for dynamic security capabilities based on real-time log analysis according to claim 1, characterized in that, The reinforcement learning strategy of the dynamic capability scheduling module performs a linear transformation on the 512-dimensional state vector and the safety capability action space. After passing through the hyperbolic tangent activation function, it generates a policy probability distribution through the softmax function. It adopts the priority experience replay technique, adjusts the sample replay priority according to the magnitude of the TD error, optimizes the parameters, and reduces the scheduling decision delay.

5. The intelligent scheduling system for dynamic security capabilities based on real-time log analysis according to claim 1, characterized in that, The Bayesian network inference of the security situation awareness module is based on the conditional probability product of the threat state vector and the feature vector. It uses Gibbs sampling technology to accelerate the calculation of posterior probability by iteratively sampling the conditional distribution of each variable when all variables are fixed.

6. The intelligent scheduling system for dynamic security capabilities based on real-time log analysis according to claim 1, characterized in that, The meta-gradient update mechanism of the feedback optimization module is as follows: the local model parameters are adjusted by a weighted combination of the gradient of the local loss function and the difference of the global meta-parameters, and gradient compression technology is used to retain only the parameters with the highest gradient magnitude for transmission.

7. The intelligent scheduling system for dynamic security capabilities based on real-time log analysis according to claim 1, characterized in that, The kernel bypass technology of the real-time log acquisition module is implemented through the DPDK data plane development kit, and the zero-copy transmission protocol adopts the SPDK storage performance development kit. Through the adaptive batch processing mechanism, the data processing batch size is dynamically adjusted according to the current request rate, system processing capacity, and queue length, so as to control the CPU utilization while ensuring the data processing speed.

8. A method for applying the intelligent scheduling system for dynamic security capabilities based on real-time log analysis as described in any one of claims 1-7, characterized in that, include: Real-time log collection steps: Logs are collected using eBPF probes and adaptive threshold filtering technology based on historical traffic mean μ and three times the standard deviation σ. DPDK / SPDK technology is used to achieve kernel bypass and zero-copy transmission of aggregated data. Multidimensional feature extraction steps: A 512-dimensional feature vector was constructed using the Transformer-XL architecture and dynamic attention weighting mechanism, with an F1 score of 0.96; Security situation awareness steps: Run the CNN-DBN model and use the Monte Carlo simulation algorithm to predict the probability of threats within 30 minutes to provide early warning of unknown threats; Dynamic capability scheduling steps: Generate scheduling strategies based on the RL-MCTS algorithm, and optimize the allocation of defense resources by superimposing exploration terms through state-action value functions; Feedback optimization steps: The meta-learning-federated learning framework is adopted, and the model is iterated through the meta-gradient update mechanism, which is updated once every 24 hours.

9. The intelligent scheduling method for dynamic security capabilities based on real-time log analysis according to claim 8, characterized in that, Also includes: The decision-making process involves: constructing a security capability knowledge graph, using graph neural networks to solve for the optimal defense configuration, and employing a multi-objective optimization algorithm to balance defense effectiveness and resource consumption. By simulating the evolution of the attack chain through a digital twin sandbox, an explanatory decision report is generated that includes defense paths, resource consumption, and effect predictions. The adaptive defense process dynamically reconstructs the defense topology based on real-time threat energy values ​​and automatically adjusts defense resources through microservice elastic scaling.

10. The intelligent scheduling method for dynamic security capabilities based on real-time log analysis according to claim 8, characterized in that, Also includes: Explanatory enhancement steps: The SHAP value interpretation framework is used to generate decision reports. By calculating feature contribution and counterfactual reasoning, it supports the explanation of attack attribution, defense effectiveness, and risk impact. The cross-domain threat hunting step uses a parallel ant colony search algorithm to mine attack patterns in heterogeneous log spaces. Combined with blockchain evidence storage technology, it constructs a cross-domain threat chain graph, reducing the APT attack discovery time to 2 hours.

Images