Software behavior anomaly detection system and method based on deep learning

Abstract

The invention discloses a software behavior anomaly detection system and method based on deep learning, and the system comprises a behavior monitoring module which is used for capturing a call sequence and context semantic information of a process, and extracting a multi-dimensional feature vector; the anomaly detection module is used for analyzing the multi-dimensional feature vectors through a hierarchical neural symbol network, outputting fusion features, dynamically weighting the fusion features through a gating symbol attention mechanism, and triggering a causal inference engine and a dynamic response framework based on an enhanced decision tree; the self-adaptive protection module comprises a causal inference engine and a dynamic response framework based on an enhanced decision tree, adjusts the protection intensity according to the environmental risk index, and transmits a feedback signal back to the anomaly detection module to optimize a threshold value; and the model evolution module is used for integrating an adversarial sample generator and an online incremental learning mechanism to realize the self-evolution capability. According to the method, the problem of leak detection of periodic attacks and hidden vulnerabilities is effectively solved, and meanwhile, false alarms caused by feature misjudgment are remarkably inhibited.

Description

Technical Field

[0001] This invention belongs to the field of software detection and repair technology, and specifically relates to a software behavior anomaly detection system and method based on deep learning. Background Technology

[0002] Traditional software anomaly detection technologies primarily rely on predefined rule bases and static threshold mechanisms, which suffer from three significant drawbacks: First, rule base updates lag behind the evolution of new attack methods, lacking effective detection capabilities against complex attacks such as zero-day vulnerabilities and advanced persistent threats (APTs). Second, static threshold strategies are ill-suited to dynamic operating environments, easily leading to false positives and false negatives under high-load scenarios, particularly in resource-constrained environments such as IoT edge devices. Third, traditional feature engineering methods are insufficient in capturing the semantic correlations of system call sequences, failing to effectively identify covert attack chains with long-range dependencies. While existing deep learning solutions have improved feature representation capabilities, they generally suffer from low efficiency in spatiotemporal feature fusion, lack of environmental context awareness, and reliance on offline retraining for model iteration. Furthermore, adversarial attacks easily degrade the performance of detection models, and existing defense mechanisms struggle to balance real-time performance and security requirements. Summary of the Invention

[0003] To overcome the shortcomings of traditional software anomaly detection technologies, which mainly rely on predefined rule bases and static threshold mechanisms, three significant drawbacks exist: First, rule base updates lag behind the evolution of new attack methods, lacking effective detection capabilities for complex attacks such as zero-day vulnerabilities and advanced persistent threats (APTs). Second, static threshold strategies are difficult to adapt to dynamic operating environments, easily leading to false positives and false negatives under high-load scenarios, especially in resource-constrained environments such as IoT edge devices. Third, traditional feature engineering methods are insufficient in capturing the semantic correlation of system call sequences, failing to effectively identify hidden attack chains with long-range dependencies (such as Dirty Pipe exploits). While existing deep learning solutions have improved feature representation capabilities, they generally suffer from low efficiency in spatiotemporal feature fusion, lack of environmental context awareness, and reliance on offline retraining for model iteration. Furthermore, adversarial attacks easily lead to performance degradation of detection models, and existing defense mechanisms struggle to balance real-time performance and security requirements.

[0004] The technical solution adopted by this patent application to solve its technical problem is:

[0005] The technical solution of this invention is:

[0006] In one aspect, this invention utilizes a deep learning-based software behavior anomaly detection system. Starting with a behavior monitoring module, this system captures the system call sequence and context information of a process in real time, extracting and generating standardized multi-dimensional feature vectors. These multi-dimensional feature vectors serve as the core input and are transmitted to the anomaly detection module. Upon receiving the multi-dimensional feature vectors, the anomaly detection module analyzes them using its internal hierarchical neural symbolic network. The anomaly detection module analyzes behavioral patterns through the hierarchical neural symbolic network, outputting fused features that are dynamically weighted using a gated symbolic attention mechanism. This triggers the causal inference engine and the dynamic response framework based on a reinforcement decision tree in the adaptive protection module. The protection module adjusts the protection strength according to a real-time risk index and simultaneously sends feedback signals (such as false positives / false negatives) back to the anomaly detection module to optimize the threshold. Suspicious samples and adversarial samples detected during the detection process are input to a model evolution module, which updates the detection model through online incremental learning. The evolved model is then redeployed to the anomaly detection module, forming a closed-loop self-evolving system. The system includes:

[0007] Behavior monitoring module: Real-time capture of process system call sequences and contextual semantic information, extracting multi-dimensional feature vectors;

[0008] Anomaly Detection Module: After receiving feature vectors, the anomaly detection module analyzes them through its internal hierarchical neural symbolic network. It uses a hierarchical neural symbolic network to parse the logical constraints of system calls based on a formal rule engine. It captures short / medium / long-range call dependencies through a hierarchical attention mechanism, identifies anomaly patterns with temporal dependencies using causal dilated convolution kernels, and designs a differentiable logic reasoning unit. It fuses the rule violation vector output by the symbolic layer with the neural layer features through tensor multiplication. The output fused features are dynamically weighted through a gated symbolic attention mechanism, triggering the causal inference engine of the adaptive protection module and the dynamic response framework based on a reinforced decision tree.

[0009] Adaptive protection module: Includes a causal inference engine and a dynamic response framework based on reinforced decision tree. It adjusts the protection intensity in real time according to the environmental risk index, and sends feedback signals back to the anomaly detection module to optimize the threshold.

[0010] Model Evolution Module: Integrates an adversarial example generator and an online incremental learning mechanism to enable the detection model to evolve on its own.

[0011] As a preferred embodiment of the present invention, the hierarchical neural symbolic network in the anomaly detection module includes:

[0012] Symbolic reasoning layer: Stores a predefined set of formal security rules. Each rule is composed of logical predicates. It takes system call sequences (i.e., the structured multi-dimensional feature vectors output by the behavior monitoring module) and context features as inputs and outputs a 128-dimensional rule violation vector. This rule violation vector is normalized by the Sigmoid function, and each dimension corresponds to the violation intensity value of a specific security rule.

[0013] The neural feature extraction layer employs several parallel attention heads, each with a key / value dimension of 64. A three-scale capture strategy is used: a short-term scale (0-10 steps, where each step refers to a time step in the system call sequence, with one step corresponding to one system call event), capturing bursty call patterns through sliding window local attention; a medium-term scale (10-100 steps), identifying periodic attack chains based on a hollow attention mechanism (span coefficient d=5); and a long-term scale (100+ steps), capturing the latency behavior of APT attacks using global attention. These short-term, medium-term, and long-term attention mechanisms process the same system call sequence simultaneously (in parallel). They analyze behavioral patterns from different time spans, capturing different types of security threats. Finally, the outputs of these three parallel branches are fused together to form a comprehensive feature representation containing multi-scale information.

[0014] Temporal Causal Convolution Module: Uses dilated convolution kernels with temporal causal constraints to ensure that feature extraction at each time step relies only on historical data, preventing the leakage of future information.

[0015] As a preferred embodiment of the present invention, the method for constructing multi-dimensional feature vectors by the behavior monitoring module includes:

[0016] The original system call ID is mapped to a continuous semantic vector space to capture the core call semantics. Runtime context features such as process permissions (e.g., privilege level), resource access patterns (e.g., file operation intensity), and call stack depth are extracted synchronously and fused through a bidirectional semantic modulation layer: the semantic vector serves as the main signal, and the context features are weighted, filtered, and refined through a dynamic modulation gate to generate context-enhanced semantic units.

[0017] The raw system call ID refers to a unique numerical identifier assigned by the operating system kernel to each system call. When a process needs to request a service from the operating system kernel (such as opening a file, creating a process, or network communication), it will enter the kernel through a specific instruction (such as int 0x80 or syscall) and place a numerical identifier representing the system call type (i.e., the ID) into a designated register (such as the eax register) to indicate which operation to perform. Dynamic modulation gates are attention-based gating structures. Their core function is to allow contextual features (permissions, resources, etc.) to "guide" the semantic vector of the system call, dynamically highlighting or suppressing different dimensions in the semantic vector, thereby generating an enhanced representation highly relevant to the current runtime environment. This differs from the modulation concept in communication; rather, it represents a high-level form of feature interaction in deep learning.

[0018] The resulting enhanced semantic units are structurally reshaped into a three-dimensional feature tensor: the first dimension is temporal length (sequence), the second dimension is semantic depth (vector dimension), and the third dimension is decoupled contextual feature channels (such as permission channels, resource channels, and stack depth channels). An innovative cross-channel residual mutual injection module is applied: the feature map of each contextual feature channel receives refined information from features of all other channels (processed by depthwise separable convolution) and undergoes adaptive residual fusion. This step enables dynamic interaction and collaboration of contextual information across different dimensions (such as permission changes and file access).

[0019] A gated feature evolution network is constructed to automatically evaluate the importance weight of each context channel, achieving the optimal channel combination under lightweight conditions. The input is a generated 3D feature tensor that has already undergone cross-channel interaction. By constructing a gated feature evolution network, channel importance evaluation and multi-scale feature collaboration are realized. A feature pyramid is constructed, and low-level feature maps and high-level feature maps interact through bidirectional feature guidance gates: high-level features guide low-level features to focus on key information, while changes in fine nodes feed back and correct the abstract expression of high-level features, realizing the co-evolution of feature maps and improving accuracy.

[0020] The evolved 3D feature tensor is compressed into a final feature vector through an adaptive spatiotemporal focusing layer. This layer adaptively aggregates key temporal segments and spatial dimensional information based on different samples, maximizing the preservation of discriminative features while filtering redundant noise, and outputting a robust multi-dimensional feature vector. The core objective of this layer is to intelligently compress the "3D feature tensor" containing rich spatiotemporal information obtained through the aforementioned steps into a one-dimensional "final feature vector." The term "adaptive" means that this compression process is not a simple, crude global averaging or fully connected layer, but rather dynamically focuses on the most important temporal segments and feature channels based on the characteristics of the current input sample.

[0021] As a preferred embodiment of the present invention, the implementation method of the causal inference engine includes: firstly, establishing an environmental risk index calculation model, quantifying the causal effects of dynamic factors such as system load and network status through a counterfactual reasoning framework, specifically utilizing Linux cgroups to implement precise resource intervention and calculate the difference in anomaly probability; secondly, constructing a dual threshold drift mechanism, enabling the warning threshold and blocking threshold to be dynamically adjusted with the causal risk index, and setting a causal confidence constraint condition of ΔP>0.2; then, introducing a time decay factor to handle periodic legitimate call patterns, identifying periodic behavior through an autocorrelation model and automatically reducing threshold sensitivity by applying an exponential decay formula; finally, adopting a causal reinforcement learning optimization strategy, fusing risk derivative features in the state space, defining the threshold adjustment range in the action space, and injecting a counterfactual reward correction mechanism into the reward function to achieve predictive decision-making.

[0022] As a preferred embodiment of the present invention, the adversarial example generator includes: generating semantically valid adversarial call sequences using Wasserstein GAN (an improved generative adversarial network), then designing a call semantic preservation loss function to ensure the grammatical correctness of the adversarial examples, then constructing an adversarial example knowledge base and using it for model retraining, and finally integrating a gradient masking mechanism to defend against white-box attacks.

[0023] As a preferred embodiment of the present invention, the dynamic response framework based on reinforcement decision trees implements a three-level hierarchical response mechanism. The response threshold is dynamically adjusted by the reinforcement learning model based on real-time environmental risks and historical feedback, rather than being a fixed value. The node depth of the decision tree is correlated with the response level, forming a progressive response strategy from early warning to containment.

[0024] 1. Probing Level Response (Low Confidence Anomaly): Determined by shallow nodes in the decision tree (depth ≤ 3). When an uncertain increase in behavioral entropy is detected, a slight deviation in resource access patterns is observed, and the attack chain confidence is low, a primary containment action is triggered. This level of action primarily focuses on interference and isolation with minimal resource consumption, such as deploying a virtual deception matrix to apply behavioral interference, combined with lightweight container isolation to limit the potential attack surface. The decision threshold for this level is fine-tuned online based on false positive feedback using a Q-learning algorithm.

[0025] 2. Propagation-level response (high-confidence attack): Determined by mid-level nodes in the decision tree (depth 4-6). When a clear privilege escalation attempt is detected, accompanied by persistent persistence and continuous resource abuse, proactive containment actions are triggered. This level of action aims to disrupt the attack chain, including dynamic privilege circuit breaking (immediately freezing high-risk privileges such as CAP_SYS_ADMIN) and resource access redirection (directing suspicious operations to the sandbox environment). The intensity of the action is dynamically adjusted within the state space using a reinforcement learning model.

[0026] 3. Outbreak-level response (system-level threat): Determined by deep nodes in the decision tree (depth ≥ 7). When kernel-level anomalous behavior highly matching a zero-day vulnerability is detected, and the causal threat index indicates an immediate substantial risk, system-level defense and forensic actions are triggered. These actions include memory state preservation (for post-incident forensic analysis), immediate blocking of malicious network connections, and proactive security probe injection for reverse analysis. The decision tree as a whole undergoes structural optimization every 24 hours through Monte Carlo tree search, and adversarial pruning strategies are used to remove invalid branches.

[0027] As a preferred embodiment of the present invention, the method for implementing the model evolution module includes: firstly, constructing a drift detector to identify concept drift phenomena; then, employing an elastic weight consolidation algorithm to prevent catastrophic forgetting; secondly, designing a dynamic memory bank to store key samples; and finally, supporting a hot update mechanism to achieve seamless model upgrades.

[0028] As a preferred embodiment of the present invention, the deployment architecture includes:

[0029] Lightweight edge detection unit: Deploys eBPF (a lightweight virtual machine program that runs in the Linux kernel) to achieve kernel-level monitoring;

[0030] Cloud-based analytics center: Aggregates multi-node data to optimize the global detection model;

[0031] Blockchain evidence storage module: Solidifies evidence of abnormal behavior on the blockchain.

[0032] In another aspect, this invention also provides a deep learning-based method for detecting software behavior anomalies, comprising the following steps:

[0033] Step 1: Capture the process's call sequence and contextual semantic information in real time, and extract multi-dimensional feature vectors.

[0034] Step 2: After receiving the multi-dimensional feature vector, it is analyzed through its internal hierarchical neural symbol network. The output fused features are dynamically weighted through a gated symbolic attention mechanism, triggering the causal inference engine and the dynamic response framework based on the reinforcement decision tree.

[0035] Step 3: Includes a causal inference engine and a dynamic response framework based on a reinforced decision tree, which adjusts the protection intensity in real time according to the environmental risk index, and sends feedback signals back to the anomaly detection module to optimize the threshold.

[0036] Step 4: Integrate the adversarial example generator with the online incremental learning mechanism to achieve the self-evolution capability of the detection model.

[0037] Advantages or beneficial effects of this application:

[0038] 1. This invention uses a hierarchical neural symbolic network to parse the logical constraints of system calls based on a formal rule engine. It captures short / medium / long-range call dependencies through a hierarchical attention mechanism and uses causal dilated convolution kernels to identify abnormal patterns with temporal dependencies. This effectively solves the problem of missed detection of periodic attacks and hidden vulnerabilities in traditional methods, while significantly suppressing false alarms caused by feature misjudgment.

[0039] 2. This invention employs a reinforcement learning-driven dual-threshold drift algorithm to construct an environment-adaptive risk response mechanism. By analyzing the risk index of network status in real time, the system can autonomously adjust the balance threshold of detection sensitivity. In the event of sudden traffic surges or complex attack scenarios, it can achieve millisecond-level protection strategy adjustment, breaking through the contradiction between false alarms and false negatives caused by fixed thresholds. It exhibits excellent stability, especially in high-concurrency scenarios, ensuring the continuity of critical services while maintaining high-precision attack interception.

[0040] 3. This invention integrates adversarial sample generation and elastic weight solidification technologies to construct a continuously evolving protection model. It stores novel attack features through a dynamic memory bank and, in conjunction with an online incremental learning algorithm, achieves real-time lossless updates of model parameters. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the structure of the present invention;

[0042] Figure 2 This is a flowchart illustrating the steps of the feature vector construction method of the present invention;

[0043] Figure 3 This is a schematic diagram of the system deployment architecture of the present invention. Detailed Implementation

[0044] Example

[0045] References to embodiments herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0046] In one aspect, this invention provides a deep learning-based software behavior anomaly detection system, such as... Figure 1 As shown, it includes:

[0047] The behavior monitoring module captures the system call sequence and contextual semantic information of a process in real time, extracting multi-dimensional feature vectors. Specifically, it uses eBPF programs to mount tracepoints in the Linux kernel, capturing sys_enter / sys_exit events (special instructions used for system calls). Contextual semantics include: call parameters (arg0-arg5), return values, the process's CapabilityBitmap, and call stack hash values. The system call sequence is represented as follows: , A stream of system call events that represents a process executing continuously over time.

[0048] like Figure 2 As shown, the feature vector construction methods include:

[0049] The original system call ID is mapped to a continuous semantic vector space to capture the core call semantics. Runtime context features such as process permissions (e.g., privilege level), resource access patterns (e.g., file operation intensity), and call stack depth are extracted synchronously and fused through a bidirectional semantic modulation layer: the semantic vector is used as the main signal, and the context features are weighted, filtered and refined by a dynamic modulation gate to generate context-enhanced semantic units.

[0050] The resulting sequence of enhanced semantic units is structurally reshaped into a three-dimensional feature tensor: the first dimension is temporal length (sequence), the second dimension is semantic depth (vector dimension), and the third dimension is decoupled contextual feature channels (such as permission channels, resource channels, and stack depth channels). An innovative cross-channel residual mutual injection module is applied: the feature map of each contextual feature channel receives refined information from features of all other channels (processed by depthwise separable convolution) and undergoes adaptive residual fusion. This step enables dynamic interaction and collaboration of contextual information across different dimensions (such as permission changes and file access).

[0051] A gated feature evolution network is constructed to automatically evaluate the importance weight of each context channel, achieving the optimal channel combination under lightweight conditions. The input is a generated 3D feature tensor that has already undergone cross-channel interaction. By constructing a gated feature evolution network, channel importance evaluation and multi-scale feature collaboration are realized. A feature pyramid is constructed, and low-level feature maps and high-level feature maps interact through bidirectional feature guidance gates: high-level features guide low-level features to focus on key information, while changes in fine nodes feed back and correct the abstract expression of high-level features, realizing the co-evolution of feature maps and improvement in accuracy.

[0052] The evolved 3D feature tensor is compressed into a final feature vector through an adaptive spatiotemporal focusing layer: this layer can adaptively aggregate key time segments and spatial dimension information according to different samples, maximize the preservation of discriminative features, filter redundant noise, and output a robust multi-dimensional feature vector.

[0053] Anomaly detection module: Employing a hierarchical neural symbolic network model, it uses a formal rule engine to parse the logical constraints of system calls, performs multi-scale pattern analysis on the system call sequence, and outputs anomaly confidence scores. Specifically, the model outputs... ,in A 128-dimensional rule violation degree vector is generated for the symbolic reasoning layer. The multi-scale feature tensor is the output of the neural feature extraction layer, and ⊗ represents the tensor product fusion operation. To achieve the joint representation of symbolic rules and neural features in differentiable logic reasoning units, This is a learnable weight matrix used to map the fused features to the output space. The Sigmoid activation function is used to map the output to the [0,1] interval to represent the anomaly confidence. A gated symbolic attention mechanism is also added, using a dynamic weight allocation formula. Achieve feature refinement For the rule violation query vector, The neural feature key vector enhances the semantic association capture capability of zero-day vulnerabilities, improving anomaly detection accuracy by 22%. Furthermore, a multi-scale Transformer encoder is embedded in the neural feature extraction layer, and cross-cycle attack chain patterns are extracted through a hierarchical multi-head attention mechanism (short / medium / long span of 10 / 100 / ∞ steps respectively). Combined with causal dilated convolution kernels, time dependency constraints are ensured.

[0054] Hierarchical neural symbolic networks include:

[0055] Modified implementation scheme of hierarchical symbolic neural network:

[0056] As a preferred embodiment of the present invention, the hierarchical neural symbolic network in the anomaly detection module includes:

[0057] Symbolic reasoning layer: Stores a predefined set of formal security rules. Each rule is composed of logical predicates. It takes system call sequences and context features as input and outputs a 128-dimensional rule violation vector. This vector is normalized using the Sigmoid function, with each dimension corresponding to the violation strength value of a specific security rule. Details are as follows:

[0058] This layer stores a predefined set of formal security rules (128 rules in total). Each rule consists of a combination of logical predicates and describes specific suspicious or malicious behavior patterns (such as abnormal permission operations, access to sensitive files, malicious network connections, etc.). The input to this layer is a multi-dimensional feature vector generated by the behavior monitoring module. During processing, the semantics, permissions, resources, and other contextual facts of system calls are first extracted from the input vector. Then, the facts are matched and logically reasoned against the rules in the rule base one by one to calculate the violation strength of each rule (a continuous value based on matching confidence and behavior severity). Finally, a 128-dimensional rule violation vector is output, where each dimension corresponds to the violation strength of a specific rule. This vector is normalized to the [0,1] interval using the Sigmoid function.

[0059] Neural Feature Extraction Layer: Eight parallel attention heads are used, each with a key / value dimension of 64. A three-scale capture strategy is employed: short-term scale (0-10 steps), which captures burst call patterns through local attention via a sliding window; medium-term scale (10-100 steps), which identifies periodic attack chains based on a hollow attention mechanism (span coefficient d=5); and long-term scale (100+ steps), which uses global attention to capture the latency behavior of APT attacks.

[0060] Temporal causal convolution module: It uses dilated convolution kernels with temporal causal constraints and dilation coefficients of [1,2,4] to ensure that feature extraction at each time step depends only on historical data and prevents future information leakage. Its calculation process strictly satisfies the temporal causal constraints.

[0061] Neural Symbol Fusion Layer: Differentiable logic reasoning units are designed to fuse the rule violation vector output by the symbol layer with the neural layer features through tensor product. First, the 128-dimensional rule violation vector is expanded into a three-dimensional tensor, and then element-wise multiplied with the neural feature tensor for fusion.

[0062] Gated symbolic attention mechanism: Feature refinement is achieved through dynamic weight calculation, using a formula. Calculate the gating signal, where The query vector generated for the degree of rule violation. The key vector represents the neural features. This is the dimension scaling factor;

[0063] Dynamic threshold output layer: The fused features are concatenated with environmental context metadata, the environmental features are encoded through a fully connected network, and finally the anomaly confidence in the [0,1] interval is output through the Sigmoid activation function.

[0064] In summary, the input system call sequence is fed in parallel to the symbolic inference layer and the neural feature extraction layer. The symbolic inference layer parses the logical constraints of the system calls based on a formal rule engine and generates a rule violation vector. Meanwhile, the neural feature extraction layer captures behavioral patterns across different time spans through a multi-scale attention mechanism and temporal causal convolution. The dual-path feature input is fed into the neural symbolic fusion layer, where feature fusion is achieved through tensor multiplication. After the fused features are weighted and refined by a gated symbolic attention mechanism, anomaly confidence is generated at the output layer in combination with environmental context metadata, forming a closed-loop analysis chain of "symbolic rule parsing → neural feature extraction → tensor fusion → gated refinement → environmental perception decision-making".

[0065] The adaptive protection module includes a causal inference engine and a dynamic response framework based on a reinforced decision tree. It adjusts the protection intensity in real time according to the environmental risk index. Specifically, when the CPU utilization is >80%, the warning threshold is adjusted from 0.8 to 0.85 to avoid false alarms under high load. It also connects to a federated causal forest model to achieve threshold optimization driven by federated causal reinforcement learning. It aggregates data from multiple nodes and quantifies the causal relationship between factors such as system load and network jitter and abnormal behavior based on a potential outcome framework. It dynamically generates a risk index, and each edge node shares threshold adjustment experience through federated learning. The Q-learning reward function introduces causal effect weights to achieve global optimal protection strategy collaboration, forming a closed loop of perception-decision-optimization-collaboration.

[0066] Specifically, the implementation method of the causal inference engine includes: first, establishing a high-order environmental risk index calculation model, quantifying the causal effects of dynamic factors such as system load and network status through a counterfactual reasoning framework; second, constructing a dual-threshold adaptive drift mechanism, designing a dual-threshold system with causal confidence constraints; then, introducing the enhancement processing of the time decay factor, constructing a time series autocorrelation model for periodic legal call patterns; and finally, adopting a causal reinforcement learning optimization strategy.

[0067] The formula for calculating the environmental risk index model is as follows:

[0068] , For CPU utilization fluctuations, Due to memory pressure, For example, when there is a sudden surge in abnormal network traffic. It increased from 0.2 to 0.8 within 200ms. These are the weighting coefficients, which are set by the operator according to the specific circumstances, and are generally 0.3, 0.2, and 0.5.

[0069] The dynamic adjustment rule in the dual threshold drift mechanism is as follows:

[0070] Warning threshold: ;

[0071] Blocking threshold: ;

[0072] Dynamic thresholds are based on environmental risk index Adjust the detection sensitivity to reduce the false alarm rate under high load;

[0073] The formula for the periodic treatment of the time decay factor is: Where γ=0.01 controls the decay rate to adapt to daily timed tasks, where This is a dynamic decay factor, a weight that changes over time, used to suppress the weight of historical events and enhance the decision-making influence of recent events. The initial decay factor, The decay rate coefficient, The time elapsed since the event occurred. The residual baseline value is typically set to 0.1; the core function of the dynamic attenuation factor is to act as a modulation coefficient to adaptively adjust the warning threshold. and blocking threshold This allows the system to intelligently handle periodic behavior and reduce false alarms.

[0074] The Q-learning update formula is used in reinforcement learning optimization: ,in This represents the current system environment status. The state of taking defensive actions. For state-action value function, The rate at which new knowledge overwrites old knowledge. For immediate rewards for environmental feedback, This is the decay factor for future rewards. The new state after the action is performed. This represents the optimal expected return under the new conditions.

[0075] The dynamic response framework implements a three-tiered response: the probing level response is implemented through nodes with a decision tree depth ≤ 3. When the behavior entropy value is in the range [2.2, 3.0], the resource mutation rate is < 250%, and the attack chain integrity is < 45%, a virtual deception matrix deployment and lightweight container isolation are triggered, and the splitting threshold is dynamically optimized based on Q-learning; the diffusion level response is executed by decision nodes with a depth of 4-6. Under the conditions that the privilege breach strength is > 0.7, the persistence index is ≥ 65%, and the resource abuse duration is > 120 seconds, dynamic privilege circuit breaking (freezing CAP_SY) is initiated. The framework employs S_ADMIN and resource redirection (sandbox file hijacking), while simultaneously regulating action intensity through a reinforcement learning state space (including the derivative of permission breach). Burst-level responses are activated by decision nodes with a depth ≥7. When zero-day vulnerability matching rate >90%, kernel anomaly density >5 / minute, and causal threat index >0.85, quantum memory preservation (block hashing for evidence), network tunneling blocking (0-RTT connection destruction), and reverse penetration probe injection are executed. The decision tree evolves its branch structure every 24 hours through Monte Carlo search and uses adversarial pruning to remove compromised branches. The framework achieves microsecond-level action execution through kernel-level eBPF, and the decision threshold is dynamically adjusted through temporal GRU prediction, forming a closed-loop evolutionary system.

[0076] Model evolution module: Integrates adversarial example generator and online incremental learning mechanism to achieve self-evolution capability of the detection model. Specifically, during online learning, new samples are processed through a buffer queue (capacity...). The training process involves adversarial meta-learning and quantum noise injection. The generator is trained based on the meta-learning framework to quickly adapt to new attack patterns and generate adversarial samples with cross-vulnerability generalization capabilities. Quantum random circuits are embedded in the model inference stage to inject controllable noise into gradient calculation, thereby disrupting the reversibility of white-box attacks while ensuring over 89% normal detection accuracy. Sample uncertainty is calculated based on Bayesian neural networks, and high-uncertainty samples are prioritized for online training, reducing the amount of labeling by 70% and forming an active defense closed loop.

[0077] The adversarial example generator includes: using Wasserstein GAN to generate semantically valid adversarial call sequences, then designing a call semantics-preserving loss function to ensure the grammatical correctness of adversarial examples, then building an adversarial example knowledge base and using it for model retraining, and finally integrating a gradient masking mechanism to defend against white-box attacks.

[0078] The model evolution module implementation method includes: first, constructing a drift detector to identify concept drift phenomena; then, using an elastic weight consolidation algorithm to prevent catastrophic forgetting; second, designing a dynamic memory bank to store key samples; and finally, supporting a hot update mechanism to achieve seamless model upgrades.

[0079] like Figure 3As shown, the system deployment architecture includes:

[0080] Lightweight edge detection unit: Deploy eBPF program to achieve kernel-level monitoring;

[0081] Cloud-based analytics center: Aggregates multi-node data to optimize the global detection model;

[0082] Blockchain evidence storage module: Solidifies evidence of abnormal behavior on the blockchain.

[0083] In summary, it should be noted that this system is based on a deep learning-based software behavior anomaly detection system. Through a multi-layered collaborative architecture, it achieves full-cycle security control. The behavior monitoring module utilizes eBPF technology to capture system call sequences and contextual semantics (including parameters, process permissions, and call stack hashes) in real-time within the kernel, constructing a multi-dimensional feature tensor. The anomaly detection module employs a spatiotemporal hybrid neural network, using parallel dilated CNNs to capture long-range call patterns (such as dependencies spanning 50+ steps), bidirectional LSTM to parse temporal contextual relationships, and a cross-attention mechanism to dynamically fuse spatiotemporal features, outputting anomaly confidence. The adaptive protection module introduces a dynamic threshold engine, adjusting dual threshold boundaries in real-time through a risk index model (integrating factors such as CPU load, memory pressure, and network anomalies), and employing reinforcement learning optimization strategies. The model evolution module integrates an adversarial example generator and an online incremental learning mechanism, generating semantically legitimate adversarial sequences through a Wasserstein GAN adversarial network, and continuously optimizing the model using an elastic weight consolidation algorithm to ensure adaptive protection against new attack patterns.

[0084] In another aspect, this invention also provides a deep learning-based method for detecting software behavior anomalies, comprising the following steps:

[0085] Step 1: Capture the process's call sequence and contextual semantic information in real time, and extract multi-dimensional feature vectors.

[0086] Step 2: After receiving the multi-dimensional feature vector, it is analyzed through its internal hierarchical neural symbol network. The output fused features are dynamically weighted through a gated symbolic attention mechanism, triggering the causal inference engine and the dynamic response framework based on the reinforcement decision tree.

[0087] Step 3: Includes a causal inference engine and a dynamic response framework based on a reinforced decision tree, which adjusts the protection intensity in real time according to the environmental risk index, and sends feedback signals back to the anomaly detection module to optimize the threshold.

[0088] Step 4: Integrate the adversarial example generator with the online incremental learning mechanism to achieve the self-evolution capability of the detection model.

[0089] The above embodiments are provided for those skilled in the art to implement or use the present invention. Those skilled in the art can make various modifications or changes to the above embodiments without departing from the spirit of the present invention. Therefore, the scope of protection of the present invention is not limited to the above embodiments, but should be the maximum scope that conforms to the innovative features mentioned in the claims.

[0090] Experimental Design and Result Analysis:

[0091] Experimental Objective: To verify the advantages of the causal inference engine in reducing false positive rates and improving attack interception efficiency. Experimental Environment:

[0092] Hardware: Edge nodes (4-core CPU / 8GB RAM), cloud analytics center;

[0093] Dataset: Includes normal system calls (CIC-IDS2017) and attack samples (Dirty Pipe exploit, DDoS attack).

[0094] Baseline comparison: Traditional static threshold scheme.

[0095] Experimental scenario:

[0096] High-load scenarios: Inject CPU stress testing tool (stress-ng) to achieve CPU utilization of 85%;

[0097] Hybrid attack scenario: Simultaneously launching a DDoS attack (SYN Flood) and a covert vulnerability exploit (Dirty Pipe).

[0098] Table 1

[0099]

[0100] The experimental results are shown in Table 1. As can be seen from Table 1, the dynamic threshold scheme reduces the false alarm rate by 63.6% under high load, proving that its environmental adaptability is significantly better than that of the static threshold; the false negative rate for hybrid attacks is reduced by 73.3%, thanks to the synergy of spatiotemporal hybrid neural network and multi-level response strategy; the response latency is optimized to the millisecond level, meeting the real-time protection requirements.

[0101] In adversarial sample attack tests, after integrating the gradient masking mechanism, the success rate of bypassing adversarial samples decreased from 35% in traditional solutions to 7%, proving its effectiveness in defending against white-box attacks.

Claims

1. A deep learning-based software behavior anomaly detection system, characterized by, The application relates to a software behavior monitoring system, which comprises the following modules: a behavior monitoring module: real-time capture of the calling sequence of a process and context semantic information, extraction of a multi-dimensional feature vector; an anomaly detection module: after receiving the multi-dimensional feature vector, analysis is performed through a hierarchical neural symbolic network in the anomaly detection module, output of fusion features through dynamic weighting of a gated symbolic attention mechanism, triggering of a causal inference engine and a dynamic response framework based on a reinforcement decision tree of an adaptive protection module; the adaptive protection module: the adaptive protection module comprises the causal inference engine and the dynamic response framework based on the reinforcement decision tree, and the protection strength is adjusted in real time according to an environmental risk index, and feedback signals are fed back to the anomaly detection module to optimize the threshold value; a model evolution module: integration of an adversarial sample generator and an online incremental learning mechanism, and self-evolution ability of the detection model is realized. 2.The deep learning based software behavior anomaly detection system of claim 1, wherein, The behavior monitoring module extracts a multi-dimensional feature vector, and the specific implementation is as follows: the software calling ID is mapped to a continuous semantic vector space, core calling semantics are captured, process permissions, resource access modes and calling stack depth runtime context features are synchronously extracted, fusion is performed through a bidirectional semantic modulation layer: the semantic vector is used as a main signal, the context features are weighted, screened and refined through a dynamic modulation gate, and a context-enhanced semantic unit is generated; the context-enhanced semantic unit is structured and reshaped into a three-dimensional feature tensor: the first dimension is the time sequence length, the second dimension is the semantic depth, and the third dimension is a decoupled context feature channel, a cross-channel residual interaction module is applied: the feature map of each context feature channel receives refined information from all other channel features processed through depth separable convolution, and adaptive residual fusion is performed; a gated feature evolution network is constructed, the importance weight of each context channel is automatically evaluated, the optimal channel combination under light weight is realized, the three-dimensional feature tensor which has undergone cross-channel interaction is input, the channel importance evaluation and multi-scale feature cooperation are realized through the construction of the gated feature evolution network, a feature pyramid is constructed, low-level feature maps and high-level feature maps interact through bidirectional feature guide gates: high-level features guide low-level features to focus on key information, and changes in nodes in turn feed back and correct the abstract expression of high-level features; the three-dimensional feature tensor which has undergone gated feature evolution is compressed into a final feature vector through an adaptive spatio-temporal focusing layer: the layer can adaptively aggregate key time segments and spatial dimension information according to different samples, retain discriminative features, filter redundant noise, and output robust multi-dimensional feature vectors. 3.The deep learning based software behavior anomaly detection system of claim 1, wherein, After receiving the multi-dimensional feature vector, the anomaly detection module analyzes through a hierarchical neural symbolic network in the anomaly detection module, a hierarchical neural symbolic network is adopted, a formal rule engine is used to analyze the logical constraints of the calling, a hierarchical attention mechanism is used to capture short / medium / long-range calling dependencies, a causal inflation convolution kernel is used to identify abnormal patterns with time sequence dependence, a differentiable logic reasoning unit is designed, a rule violation degree vector output by a symbolic layer is multiplied with a neural layer feature to realize fusion, fusion features are dynamically weighted through a gated symbolic attention mechanism, and a causal inference engine and a dynamic response framework based on a reinforcement decision tree of an adaptive protection module are triggered. 4.The deep learning based software behavior anomaly detection system of claim 3, wherein, The hierarchical neural symbolic network in the anomaly detection module is specifically implemented as follows: Symbolic reasoning layer: store a set of pre-defined formal security rules, each rule is composed of a logical predicate combination, input multi-dimensional feature vector and context feature, output rule violation degree vector, the rule violation degree vector is normalized by Sigmoid function, each dimension corresponds to the violation intensity value of the security rule; Neural feature extraction layer: adopt several parallel attention heads, adopt a three-scale capture strategy: short-term scale, time steps in sequence are 0-10 steps, capture burst invocation mode through sliding window local attention; medium sequence time steps are 10-100 steps, identify periodic attack chain based on a hollow attention mechanism; Long-term scale, time steps in sequence are above 100, use global attention to capture the incubation behavior of APT attacks; Time causal convolution module: use an expanded convolution kernel with time causal constraints to ensure that feature extraction at each time step only depends on historical data. 5.The deep learning based software behavior anomaly detection system of claim 1, wherein, The implementation of the causal inference engine includes: first, establish an environmental risk index calculation model, quantify the causal effect of dynamic factors through the counterfactual reasoning framework, specifically use Linux cgroups to implement resource intervention and calculate the abnormal probability difference; second, build a double-threshold drift mechanism, dynamically adjust the warning threshold and blocking threshold according to the causal risk index, and set the causal confidence constraint condition ΔP>0.2; then introduce a time decay factor to handle periodic legal invocation patterns, identify periodic behavior through an autocorrelation model and apply an exponential decay formula to automatically reduce threshold sensitivity; finally, use a causal reinforcement learning optimization strategy to fuse risk derivative features in the state space, define threshold adjustment amplitude in the action space, and inject a counterfactual reward correction mechanism in the reward function to realize predictive decision-making. 6.The deep learning based software behavior anomaly detection system of claim 1, wherein, The dynamic response framework based on reinforcement decision tree realizes a three-level hierarchical response mechanism, and the response threshold is dynamically adjusted by the reinforcement learning model according to the real-time environmental risk and historical feedback, which is not a fixed value; the node depth of the decision tree is associated with the response level, forming a progressive response strategy from warning to suppression: Exploration level response: determined by the shallow nodes of the decision tree with a depth less than or equal to 3; when detecting that the behavior entropy value is uncertain and rising, the resource access pattern deviates, and the attack chain confidence is low and the threshold, trigger the primary containment action; this level of action mainly interferes and isolates with the lowest resource consumption, and combines lightweight container isolation to limit the potential attack surface; the decision threshold of this level is optimized online through the Q-learning algorithm according to the false alarm feedback; Diffusion level response: determined by the middle layer nodes of the decision tree with a depth of 4 to 6; when detecting clear privilege escalation attempts, signs of persistent residence, and persistent resource abuse behavior, trigger active containment actions; this level of action aims to interrupt the attack chain, including dynamic privilege fusing and resource access redirection; The execution intensity of the action is dynamically regulated in the state space by the reinforcement learning model; The explosion level response is determined by a deep node of the decision tree with a depth greater than or equal to 7; when kernel-level abnormal behavior matching a zero-day vulnerability is detected, and the causal threat index indicates that there is an immediate substantial risk, system-level defense and forensic actions are triggered; this level of action includes memory state preservation, immediate blocking of malicious network connections, and active security probe injection for reverse analysis; the whole decision tree is optimized in structure by Monte Carlo tree search every 24 hours, and an adversarial pruning strategy is used to remove ineffective branches. 7.The deep learning based software behavior anomaly detection system of claim 1, wherein, The adversarial sample generator includes: using an improved generative adversarial network Wasserstein GAN to generate semantic legal adversarial call sequences, then designing call semantic preservation loss functions to ensure the grammatical correctness of adversarial samples, then constructing an adversarial sample knowledge base and using it for model retraining, and finally integrating gradient mask mechanism to defend against white-box attacks. 8.The deep learning based software behavior anomaly detection system of claim 1, wherein, The model evolution module includes: first, constructing a drift detector to identify concept drift phenomena, then using an elastic weight consolidation algorithm to prevent catastrophic forgetting, second, designing a dynamic memory bank to store key samples, and finally supporting a hot update mechanism to realize seamless model upgrade.

9. A software behavior anomaly detection method based on deep learning, used for implementing the software behavior anomaly detection system according to any one of claims 1 to 8, characterized in that, The method comprises the following steps: Step 1: capture the call sequence and context semantic information of the process in real time, and extract a multi-dimensional feature vector; Step 2: after receiving the multi-dimensional feature vector, analyze it through the internal hierarchical neural symbolic network, output the fused features through the gated symbolic attention mechanism for dynamic weighting, trigger the causal inference engine and the dynamic response framework based on the reinforcement decision tree; Step 3: includes the causal inference engine and the dynamic response framework based on the reinforcement decision tree, adjusts the protection strength in real time according to the environmental risk index, and at the same time, feeds back the feedback signal to the anomaly detection module to optimize the threshold; Step 4: integrate the adversarial sample generator and the online incremental learning mechanism to realize the self-evolution ability of the detection model.

Images