Software behavior anomaly detection system and method based on deep learning

Abstract

The application discloses a software behavior anomaly detection system and method based on deep learning, which comprises a behavior monitoring module, an anomaly detection module, an adaptive protection module and a model evolution module.The behavior monitoring module is used for capturing the calling sequence and context semantic information of a process and extracting a multi-dimensional feature vector.The anomaly detection module is used for analyzing the multi-dimensional feature vector through a hierarchical neural symbolic network, outputting a fusion feature through dynamic weighting of a gated symbolic attention mechanism, triggering a causal inference engine and a dynamic response framework based on a reinforcement decision tree.The adaptive protection module comprises the causal inference engine and the dynamic response framework based on the reinforcement decision tree, adjusts the protection strength according to an environmental risk index, and feeds back a feedback signal to the anomaly detection module to optimize a threshold.The model evolution module is used for integrating an adversarial sample generator and an online incremental learning mechanism to realize self-evolution capability.The application effectively solves the problem of missed detection of periodic attacks and hidden vulnerabilities, and significantly suppresses false positives caused by feature misjudgment.

Description

Technical Field

[0001] This invention belongs to the field of software detection and repair technology, and specifically relates to a software behavior anomaly detection system and method based on deep learning. Background Technology

[0002] Traditional software anomaly detection technologies primarily rely on predefined rule bases and static threshold mechanisms, which suffer from three significant drawbacks: First, rule base updates lag behind the evolution of new attack methods, lacking effective detection capabilities against complex attacks such as zero-day vulnerabilities and advanced persistent threats (APTs). Second, static threshold strategies are ill-suited to dynamic operating environments, easily leading to false positives and false negatives under high-load scenarios, particularly in resource-constrained environments such as IoT edge devices. Third, traditional feature engineering methods are insufficient in capturing the semantic correlations of system call sequences, failing to effectively identify covert attack chains with long-range dependencies. While existing deep learning solutions have improved feature representation capabilities, they generally suffer from low efficiency in spatiotemporal feature fusion, lack of environmental context awareness, and reliance on offline retraining for model iteration. Furthermore, adversarial attacks easily degrade the performance of detection models, and existing defense mechanisms struggle to balance real-time performance and security requirements. Summary of the Invention

[0003] To overcome the shortcomings of traditional software anomaly detection technologies, which mainly rely on predefined rule bases and static threshold mechanisms, three significant drawbacks exist: First, rule base updates lag behind the evolution of new attack methods, lacking effective detection capabilities for complex attacks such as zero-day vulnerabilities and advanced persistent threats (APTs). Second, static threshold strategies are difficult to adapt to dynamic operating environments, easily leading to false positives and false negatives under high-load scenarios, especially in resource-constrained environments such as IoT edge devices. Third, traditional feature engineering methods are insufficient in capturing the semantic correlation of system call sequences, failing to effectively identify hidden attack chains with long-range dependencies (such as Dirty Pipe exploits). While existing deep learning solutions have improved feature representation capabilities, they generally suffer from low efficiency in spatiotemporal feature fusion, lack of environmental context awareness, and reliance on offline retraining for model iteration. Furthermore, adversarial attacks easily lead to performance degradation of detection models, and existing defense mechanisms struggle to balance real-time performance and security requirements.

[0004] The technical solution adopted by this patent application to solve its technical problem is:

[0005] The technical solution of this invention is:

[0006] In one aspect, this invention utilizes a deep learning-based software behavior anomaly detection system. Starting with a behavior monitoring module, this system captures the system call sequence and context information of a process in real time, extracting and generating standardized multi-dimensional feature vectors. These multi-dimensional feature vectors serve as the core input and are transmitted to the anomaly detection module. Upon receiving the multi-dimensional feature vectors, the anomaly detection module analyzes them using its internal hierarchical neural symbolic network. The anomaly detection module analyzes behavioral patterns through the hierarchical neural symbolic network, outputting fused features that are dynamically weighted using a gated symbolic attention mechanism. This triggers the causal inference engine and the dynamic response framework based on a reinforcement decision tree in the adaptive protection module. The protection module adjusts the protection strength according to a real-time risk index and simultaneously sends feedback signals (such as false positives / false negatives) back to the anomaly detection module to optimize the threshold. Suspicious samples and adversarial samples detected during the detection process are input to a model evolution module, which updates the detection model through online incremental learning. The evolved model is then redeployed to the anomaly detection module, forming a closed-loop self-evolving system. The system includes:

[0007] Behavior monitoring module: Real-time capture of process system call sequences and contextual semantic information, extracting multi-dimensional feature vectors;

[0008] Anomaly Detection Module: After receiving feature vectors, the anomaly detection module analyzes them through its internal hierarchical neural symbolic network. It uses a hierarchical neural symbolic network to parse the logical constraints of system calls based on a formal rule engine. It captures short / medium / long-range call dependencies through a hierarchical attention mechanism, identifies anomaly patterns with temporal dependencies using causal dilated convolution kernels, and designs a differentiable logic reasoning unit. It fuses the rule violation vector output by the symbolic layer with the neural layer features through tensor multiplication. The output fused features are dynamically weighted through a gated symbolic attention mechanism, triggering the causal inference engine of the adaptive protection module and the dynamic response framework based on a reinforced decision tree.

[0009] Adaptive protection module: Includes a causal inference engine and a dynamic response framework based on reinforced decision tree. It adjusts the protection intensity in real time according to the environmental risk index, and sends feedback signals back to the anomaly detection module to optimize the threshold.

[0010] Model Evolution Module: Integrates an adversarial example generator and an online incremental learning mechanism to enable the detection model to evolve on its own.

[0011] As a preferred embodiment of the present invention, the hierarchical neural symbolic network in the anomaly detection module includes:

[0012] Symbolic reasoning layer: Stores a predefined set of formal security rules. Each rule is composed of logical predicates. It takes system call sequences (i.e., the structured multi-dimensional feature vectors output by the behavior monitoring module) and context features as inputs and outputs a 128-dimensional rule violation vector. This rule violation vector is normalized by the Sigmoid function, and each dimension corresponds to the violation intensity value of a specific security rule.

[0013] The neural feature extraction layer employs several parallel attention heads, each with a key / value dimension of 64. A three-scale capture strategy is used: a short-term scale (0-10 steps, where each step refers to a time step in the system call sequence, with one step corresponding to one system call event), capturing bursty call patterns through sliding window local attention; a medium-term scale (10-100 steps), identifying periodic attack chains based on a hollow attention mechanism (span coefficient d=5); and a long-term scale (100+ steps), capturing the latency behavior of APT attacks using global attention. These short-term, medium-term, and long-term attention mechanisms process the same system call sequence simultaneously (in parallel). They analyze behavioral patterns from different time spans, capturing different types of security threats. Finally, the outputs of these three parallel branches are fused together to form a comprehensive feature representation containing multi-scale information.

[0014] Temporal Causal Convolution Module: Uses dilated convolution kernels with temporal causal constraints to ensure that feature extraction at each time step relies only on historical data, preventing the leakage of future information.

[0015] As a preferred embodiment of the present invention, the method for constructing multi-dimensional feature vectors by the behavior monitoring module includes:

[0016] The original system call ID is mapped to a continuous semantic vector space to capture the core call semantics. Runtime context features such as process permissions (e.g., privilege level), resource access patterns (e.g., file operation intensity), and call stack depth are extracted synchronously and fused through a bidirectional semantic modulation layer: the semantic vector serves as the main signal, and the context features are weighted, filtered, and refined through a dynamic modulation gate to generate context-enhanced semantic units.

[0017] The raw system call ID refers to a unique numerical identifier assigned by the operating system kernel to each system call. When a process needs to request a service from the operating system kernel (such as opening a file, creating a process, or network communication), it will enter the kernel through a specific instruction (such as int 0x80 or syscall) and place a numerical identifier representing the system call type (i.e., the ID) into a designated register (such as the eax register) to indicate which operation to perform. Dynamic modulation gates are attention-based gating structures. Their core function is to allow contextual features (permissions, resources, etc.) to "guide" the semantic vector of the system call, dynamically highlighting or suppressing different dimensions in the semantic vector, thereby generating an enhanced representation highly relevant to the current runtime environment. This differs from the modulation concept in communication; rather, it represents a high-level form of feature interaction in deep learning.

[0018] The resulting enhanced semantic units are structurally reshaped into a three-dimensional feature tensor: the first dimension is temporal length (sequence), the second dimension is semantic depth (vector dimension), and the third dimension is decoupled contextual feature channels (such as permission channels, resource channels, and stack depth channels). An innovative cross-channel residual mutual injection module is applied: the feature map of each contextual feature channel receives refined information from features of all other channels (processed by depthwise separable convolution) and undergoes adaptive residual fusion. This step enables dynamic interaction and collaboration of contextual information across different dimensions (such as permission changes and file access).

[0019] A gated feature evolution network is constructed to automatically evaluate the importance weight of each context channel, achieving the optimal channel combination under lightweight conditions. The input is a generated 3D feature tensor that has already undergone cross-channel interaction. By constructing a gated feature evolution network, channel importance evaluation and multi-scale feature collaboration are realized. A feature pyramid is constructed, and low-level feature maps and high-level feature maps interact through bidirectional feature guidance gates: high-level features guide low-level features to focus on key information, while changes in fine nodes feed back and correct the abstract expression of high-level features, realizing the co-evolution of feature maps and improving accuracy.

[0020] The evolved 3D feature tensor is compressed into a final feature vector through an adaptive spatiotemporal focusing layer. This layer adaptively aggregates key temporal segments and spatial dimensional information based on different samples, maximizing the preservation of discriminative features while filtering redundant noise, and outputting a robust multi-dimensional feature vector. The core objective of this layer is to intelligently compress the "3D feature tensor" containing rich spatiotemporal information obtained through the aforementioned steps into a one-dimensional "final feature vector." The term "adaptive" means that this compression process is not a simple, crude global averaging or fully connected layer, but rather dynamically focuses on the most important temporal segments and feature channels based on the characteristics of the current input sample.

[0021] As a preferred embodiment of the present invention, the implementation method of the causal inference engine includes: firstly, establishing an environmental risk index calculation model, quantifying the causal effects of dynamic factors such as system load and network status through a counterfactual reasoning framework, specifically utilizing Linux cgroups to implement precise resource intervention and calculate the difference in anomaly probability; secondly, constructing a dual threshold drift mechanism, enabling the warning threshold and blocking threshold to be dynamically adjusted with the causal risk index, and setting a causal confidence constraint condition of ΔP>0.2; then, introducing a time decay factor to handle periodic legitimate call patterns, identifying periodic behavior through an autocorrelation model and automatically reducing threshold sensitivity by applying an exponential decay formula; finally, adopting a causal reinforcement learning optimization strategy, fusing risk derivative features in the state space, defining the threshold adjustment range in the action space, and injecting a counterfactual reward correction mechanism into the reward function to achieve predictive decision-making.

[0022] As a preferred embodiment of the present invention, the adversarial example generator includes: generating semantically valid adversarial call sequences using Wasserstein GAN (an improved generative adversarial network), then designing a call semantic preservation loss function to ensure the grammatical correctness of the adversarial examples, then constructing an adversarial example knowledge base and using it for model retraining, and finally integrating a gradient masking mechanism to defend against white-box attacks.

[0023] As a preferred embodiment of the present invention, the dynamic response framework based on reinforcement decision trees implements a three-level hierarchical response mechanism. The response threshold is dynamically adjusted by the reinforcement learning model based on real-time environmental risks and historical feedback, rather than being a fixed value. The node depth of the decision tree is correlated with the response level, forming a progressive response strategy from early warning to containment.

[0024] 1. Probing Level Response (Low Confidence Anomaly): Determined by shallow nodes in the decision tree (depth ≤ 3). When an uncertain increase in behavioral entropy is detected, a slight deviation in resource access patterns is observed, and the attack chain confidence is low, a primary containment action is triggered. This level of action primarily focuses on interference and isolation with minimal resource consumption, such as deploying a virtual deception matrix to apply behavioral interference, combined with lightweight container isolation to limit the potential attack surface. The decision threshold for this level is fine-tuned online based on false positive feedback using a Q-learning algorithm.

[0025] 2. Propagation-level response (high-confidence attack): Determined by mid-level nodes in the decision tree (depth 4-6). When a clear privilege escalation attempt is detected, accompanied by persistent persistence and continuous resource abuse, proactive containment actions are triggered. This level of action aims to disrupt the attack chain, including dynamic privilege circuit breaking (immediately freezing high-risk privileges such as CAP_SYS_ADMIN) and resource access redirection (directing suspicious operations to the sandbox environment). The intensity of the action is dynamically adjusted within the state space using a reinforcement learning model.

[0026] 3. Outbreak-level response (system-level threat): Determined by deep nodes in the decision tree (depth ≥ 7). When kernel-level anomalous behavior highly matching a zero-day vulnerability is detected, and the causal threat index indicates an immediate substantial risk, system-level defense and forensic actions are triggered. These actions include memory state preservation (for post-incident forensic analysis), immediate blocking of malicious network connections, and proactive security probe injection for reverse analysis. The decision tree as a whole undergoes structural optimization every 24 hours through Monte Carlo tree search, and adversarial pruning strategies are used to remove invalid branches.

[0027] As a preferred embodiment of the present invention, the method for implementing the model evolution module includes: firstly, constructing a drift detector to identify concept drift phenomena; secondly, employing an elastic weight consolidation algorithm to prevent catastrophic forgetting; thirdly, designing a dynamic memory bank to store key samples; and finally, supporting a hot update mechanism to achieve seamless model upgrades.

[0028] As a preferred embodiment of the present invention, the deployment architecture includes:

[0029] Lightweight edge detection unit: Deploys eBPF (a lightweight virtual machine program that runs in the Linux kernel) to achieve kernel-level monitoring;

[0030] Cloud-based analytics center: Aggregates multi-node data to optimize the global detection model;

[0031] Blockchain evidence storage module: Solidifies evidence of abnormal behavior on the blockchain.

[0032] In another aspect, this invention also provides a deep learning-based method for detecting software behavior anomalies, comprising the following steps:

[0033] Step 1: Capture the process's call sequence and contextual semantic information in real time, and extract multi-dimensional feature vectors.

[0034] Step 2: After receiving the multi-dimensional feature vector, it is analyzed through its internal hierarchical neural symbol network. The output fused features are dynamically weighted through a gated symbolic attention mechanism, triggering the causal inference engine and the dynamic response framework based on the reinforcement decision tree.

[0035] Step 3: Includes a causal inference engine and a dynamic response framework based on a reinforced decision tree, which adjusts the protection intensity in real time according to the environmental risk index, and sends feedback signals back to the anomaly detection module to optimize the threshold.

[0036] Step 4: Integrate the adversarial example generator with the online incremental learning mechanism to achieve the self-evolution capability of the detection model.

[0037] Advantages or beneficial effects of this application:

[0038] 1. This invention uses a hierarchical neural symbolic network to parse the logical constraints of system calls based on a formal rule engine. It captures short / medium / long-range call dependencies through a hierarchical attention mechanism and uses causal dilated convolution kernels to identify abnormal patterns with temporal dependencies. This effectively solves the problem of missed detection of periodic attacks and hidden vulnerabilities in traditional methods, while significantly suppressing false alarms caused by feature misjudgment.

[0039] 2. This invention employs a reinforcement learning-driven dual-threshold drift algorithm to construct an environment-adaptive risk response mechanism. By analyzing the risk index of network status in real time, the system can autonomously adjust the balance threshold of detection sensitivity. In the event of sudden traffic surges or complex attack scenarios, it can achieve millisecond-level protection strategy adjustment, breaking through the contradiction between false alarms and false negatives caused by fixed thresholds. It exhibits excellent stability, especially in high-concurrency scenarios, ensuring the continuity of critical services while maintaining high-precision attack interception.

[0040] 3. This invention integrates adversarial sample generation and elastic weight solidification technologies to construct a continuously evolving protection model. It stores novel attack features through a dynamic memory bank and, in conjunction with an online incremental learning algorithm, achieves real-time lossless updates of model parameters. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the structure of the present invention;

[0042] Figure 2 This is a flowchart illustrating the steps of the feature vector construction method of the present invention;

[0043] Figure 3 This is a schematic diagram of the system deployment architecture of the present invention. Detailed Implementation

[0044] Example

[0045] References to embodiments herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0046] In one aspect, this invention provides a deep learning-based software behavior anomaly detection system, such as... Figure 1 As shown, it includes:

[0047] The behavior monitoring module captures the system call sequence and contextual semantic information of a process in real time, extracting multi-dimensional feature vectors. Specifically, it uses eBPF programs to mount tracepoints in the Linux kernel, capturing sys_enter / sys_exit events (special instructions used for system calls). Contextual semantics include: call parameters (arg0-arg5), return values, the process's CapabilityBitmap, and call stack hash values. The system call sequence is represented as follows: , A stream of system call events that represents a process executing continuously over time.

[0048] like Figure 2 As shown, the feature vector construction methods include:

[0049] The original system call ID is mapped to a continuous semantic vector space to capture the core call semantics. Runtime context features such as process permissions (e.g., privilege level), resource access patterns (e.g., file operation intensity), and call stack depth are extracted synchronously and fused through a bidirectional semantic modulation layer: the semantic vector is used as the main signal, and the context features are weighted, filtered and refined by a dynamic modulation gate to generate context-enhanced semantic units.

[0050] The resulting sequence of enhanced semantic units is structurally reshaped into a three-dimensional feature tensor: the first dimension is temporal length (sequence), the second dimension is semantic depth (vector dimension), and the third dimension is decoupled contextual feature channels (such as permission channels, resource channels, and stack depth channels). An innovative cross-channel residual mutual injection module is applied: the feature map of each contextual feature channel receives refined information from features of all other channels (processed by depthwise separable convolution) and undergoes adaptive residual fusion. This step enables dynamic interaction and collaboration of contextual information across different dimensions (such as permission changes and file access).

[0051] A gated feature evolution network is constructed to automatically evaluate the importance weight of each context channel, achieving the optimal channel combination under lightweight conditions. The input is a generated 3D feature tensor that has already undergone cross-channel interaction. By constructing a gated feature evolution network, channel importance evaluation and multi-scale feature collaboration are realized. A feature pyramid is constructed, and low-level feature maps and high-level feature maps interact through bidirectional feature guidance gates: high-level features guide low-level features to focus on key information, while changes in fine nodes feed back and correct the abstract expression of high-level features, realizing the co-evolution of feature maps and improvement in accuracy.

[0052] The evolved 3D feature tensor is compressed into a final feature vector through an adaptive spatiotemporal focusing layer: this layer can adaptively aggregate key time segments and spatial dimension information according to different samples, maximize the preservation of discriminative features, filter redundant noise, and output a robust multi-dimensional feature vector.

[0053] Anomaly detection module: Employing a hierarchical neural symbolic network model, it uses a formal rule engine to parse the logical constraints of system calls, performs multi-scale pattern analysis on the system call sequence, and outputs anomaly confidence scores. Specifically, the model outputs... ,in A 128-dimensional rule violation degree vector is generated for the symbolic reasoning layer. The multi-scale feature tensor is the output of the neural feature extraction layer, and ⊗ represents the tensor product fusion operation. To achieve the joint representation of symbolic rules and neural features in differentiable logic reasoning units, This is a learnable weight matrix used to map the fused features to the output space. The Sigmoid activation function maps the output to the [0,1] interval to represent anomaly confidence. A gated symbolic attention mechanism is also added, using a dynamic weight allocation formula. Achieve feature refinement For the rule violation query vector, The neural feature key vector enhances the semantic association capture capability of zero-day vulnerabilities, improving anomaly detection accuracy by 22%. Furthermore, a multi-scale Transformer encoder is embedded in the neural feature extraction layer, and cross-cycle attack chain patterns are extracted through a hierarchical multi-head attention mechanism (short / medium / long span of 10 / 100 / ∞ steps respectively). Combined with causal dilated convolution kernels, time dependency constraints are ensured.

[0054] Hierarchical neural symbolic networks include:

[0055] Modified implementation scheme of hierarchical neural symbolic network:

[0056] As a preferred embodiment of the present invention, the hierarchical neural symbolic network in the anomaly detection module includes:

[0057] Symbolic reasoning layer: Stores a predefined set of formal security rules. Each rule is composed of logical predicates. It takes system call sequences and context features as input and outputs a 128-dimensional rule violation vector. This vector is normalized using the Sigmoid function, with each dimension corresponding to the violation strength value of a specific security rule. Details are as follows:

[0058] This layer stores a predefined set of formal security rules (128 rules in total). Each rule consists of a combination of logical predicates and describes specific suspicious or malicious behavior patterns (such as abnormal permission operations, access to sensitive files, malicious network connections, etc.). The input to this layer is a multi-dimensional feature vector generated by the behavior monitoring module. During processing, the semantics, permissions, resources, and other contextual facts of system calls are first extracted from the input vector. Then, the facts are matched and logically reasoned against the rules in the rule base one by one to calculate the violation strength of each rule (a continuous value based on matching confidence and behavior severity). Finally, a 128-dimensional rule violation vector is output, where each dimension corresponds to the violation strength of a specific rule. This vector is normalized to the [0,1] interval using the Sigmoid function.

[0059] Neural Feature Extraction Layer: Eight parallel attention heads are used, each with a key / value dimension of 64. A three-scale capture strategy is employed: short-term scale (0-10 steps), which captures burst call patterns through local attention via a sliding window; medium-term scale (10-100 steps), which identifies periodic attack chains based on a hollow attention mechanism (span coefficient d=5); and long-term scale (100+ steps), which uses global attention to capture the latency behavior of APT attacks.

[0060] Temporal causal convolution module: Uses dilated convolution kernels with temporal causal constraints, with dilation coefficients of [1,2,4], to ensure that feature extraction at each time step relies only on historical data, preventing future information leakage. Its calculation process strictly satisfies the temporal causal constraints.

[0061] Neural Symbol Fusion Layer: Differentiable logic reasoning unit is designed to fuse the rule violation degree vector output by the symbol layer with the neural layer features through tensor product. First, the 128-dimensional rule violation degree vector is expanded into a three-dimensional tensor, and then element-wise multiplied and fused with the neural feature tensor.

[0062] Gated symbolic attention mechanism: Feature refinement is achieved through dynamic weight calculation, using a formula. Calculate the gating signal, where The query vector generated for the degree of rule violation. The key vector represents the neural features. This is the dimension scaling factor;

[0063] Dynamic threshold output layer: The fused features are concatenated with environmental context metadata, the environmental features are encoded through a fully connected network, and finally the anomaly confidence in the [0,1] interval is output through the Sigmoid activation function.

[0064] In summary, the input system call sequence is fed in parallel to the symbolic inference layer and the neural feature extraction layer. The symbolic inference layer parses the logical constraints of the system calls based on a formal rule engine and generates a rule violation vector. Meanwhile, the neural feature extraction layer captures behavioral patterns across different time spans through a multi-scale attention mechanism and temporal causal convolution. The dual-path feature input is fed into the neural symbolic fusion layer, where feature fusion is achieved through tensor multiplication. After the fused features are weighted and refined by a gated symbolic attention mechanism, anomaly confidence is generated at the output layer in combination with environmental context metadata, forming a closed-loop analysis chain of "symbolic rule parsing → neural feature extraction → tensor fusion → gated refinement → environmental perception decision-making".

[0065] The adaptive protection module includes a causal inference engine and a dynamic response framework based on a reinforced decision tree. It adjusts the protection intensity in real time according to the environmental risk index. Specifically, when the CPU utilization is >80%, the warning threshold is adjusted from 0.8 to 0.85 to avoid false alarms under high load. It also connects to a federated causal forest model to achieve threshold optimization driven by federated causal reinforcement learning. It aggregates data from multiple nodes and quantifies the causal relationship between factors such as system load and network jitter and abnormal behavior based on a potential outcome framework. It dynamically generates a risk index, and each edge node shares threshold adjustment experience through federated learning. The Q-learning reward function introduces causal effect weights to achieve global optimal protection strategy collaboration, forming a closed loop of perception-decision-optimization-collaboration.

[0066] Specifically, the implementation method of the causal inference engine includes: first, establishing a high-order environmental risk index calculation model, quantifying the causal effects of dynamic factors such as system load and network status through a counterfactual reasoning framework; second, constructing a dual-threshold adaptive drift mechanism, designing a dual-threshold system with causal confidence constraints; then, introducing the enhancement processing of the time decay factor, constructing a time series autocorrelation model for periodic legal call patterns; and finally, adopting a causal reinforcement learning optimization strategy.

[0067] The formula for calculating the environmental risk index model is as follows:

[0068] , For CPU utilization fluctuations, Due to memory pressure, For example, when there is a sudden surge in abnormal network traffic. It increased from 0.2 to 0.8 within 200ms. These are the weighting coefficients, which are set by the operator according to the specific circumstances, and are generally 0.3, 0.2, and 0.5.

[0069] The dynamic adjustment rule in the dual threshold drift mechanism is as follows:

[0070] Warning threshold: ;

[0071] Blocking threshold: ;

[0072] Dynamic thresholds are based on environmental risk index Adjust the detection sensitivity to reduce the false alarm rate under high load;

[0073] The formula for the periodic treatment of the time decay factor is: Where γ=0.01 controls the decay rate to adapt to daily timed tasks, where This is a dynamic decay factor, a weight that changes over time, used to suppress the weight of historical events and enhance the decision-making influence of recent events. The initial decay factor, The decay rate coefficient, The time elapsed since the event occurred. The residual baseline value is typically set to 0.1; the core function of the dynamic attenuation factor is to act as a modulation coefficient to adaptively adjust the warning threshold. and blocking threshold This allows the system to intelligently handle periodic behavior and reduce false alarms.

[0074] The Q-learning update formula is used in reinforcement learning optimization:

[0075] ,in This represents the current system environment status. The state of taking defensive actions. For state-action value function, The rate at which new knowledge overwrites old knowledge. For immediate rewards for environmental feedback, This is the decay factor for future rewards. The new state after the action is performed. This represents the optimal expected return under the new conditions.

[0076] The dynamic response framework implements a three-tiered response: the probing level response is implemented through nodes with a decision tree depth ≤ 3. When the behavior entropy value is in the range [2.2, 3.0], the resource mutation rate is < 250%, and the attack chain integrity is < 45%, a virtual deception matrix deployment and lightweight container isolation are triggered, and the splitting threshold is dynamically optimized based on Q-learning; the diffusion level response is executed by decision nodes with a depth of 4-6. Under the conditions that the privilege breach strength is > 0.7, the persistence index is ≥ 65%, and the resource abuse duration is > 120 seconds, dynamic privilege circuit breaking (freezing CAP_SY) is initiated. The framework employs S_ADMIN and resource redirection (sandbox file hijacking), while simultaneously regulating action intensity through a reinforcement learning state space (including the derivative of permission breach). Burst-level responses are activated by decision nodes with a depth ≥7. When zero-day vulnerability matching rate >90%, kernel anomaly density >5 / minute, and causal threat index >0.85, quantum memory preservation (block hashing for evidence), network tunneling blocking (0-RTT connection destruction), and reverse penetration probe injection are executed. The decision tree evolves its branch structure every 24 hours through Monte Carlo search and uses adversarial pruning to remove compromised branches. The framework achieves microsecond-level action execution through kernel-level eBPF, and the decision threshold is dynamically adjusted through temporal GRU prediction, forming a closed-loop evolutionary system.

[0077] Model evolution module: Integrates adversarial example generator and online incremental learning mechanism to achieve self-evolution capability of the detection model. Specifically, during online learning, new samples are processed through a buffer queue (capacity...). The training process involves adversarial meta-learning and quantum noise injection. The generator is trained based on the meta-learning framework to quickly adapt to new attack patterns and generate adversarial samples with cross-vulnerability generalization capabilities. Quantum random circuits are embedded in the model inference stage to inject controllable noise into gradient calculation, thereby disrupting the reversibility of white-box attacks while ensuring over 89% normal detection accuracy. Sample uncertainty is calculated based on Bayesian neural networks, and high-uncertainty samples are prioritized for online training, reducing the amount of labeling by 70% and forming an active defense closed loop.

[0078] The adversarial example generator includes: using Wasserstein GAN to generate semantically valid adversarial call sequences, then designing a call semantics-preserving loss function to ensure the grammatical correctness of adversarial examples, then building an adversarial example knowledge base and using it for model retraining, and finally integrating a gradient masking mechanism to defend against white-box attacks.

[0079] The model evolution module implementation method includes: first, constructing a drift detector to identify concept drift phenomena; then, using an elastic weight consolidation algorithm to prevent catastrophic forgetting; second, designing a dynamic memory bank to store key samples; and finally, supporting a hot update mechanism to achieve seamless model upgrades.

[0080] like Figure 3As shown, the system deployment architecture includes:

[0081] Lightweight edge detection unit: Deploy eBPF program to achieve kernel-level monitoring;

[0082] Cloud-based analytics center: Aggregates multi-node data to optimize the global detection model;

[0083] Blockchain evidence storage module: Solidifies evidence of abnormal behavior on the blockchain.

[0084] In summary, it should be noted that this system is based on a deep learning-based software behavior anomaly detection system. Through a multi-layered collaborative architecture, it achieves full-cycle security control. The behavior monitoring module utilizes eBPF technology to capture system call sequences and contextual semantics (including parameters, process permissions, and call stack hashes) in real-time within the kernel, constructing a multi-dimensional feature tensor. The anomaly detection module employs a spatiotemporal hybrid neural network, using parallel dilated CNNs to capture long-range call patterns (such as dependencies spanning 50+ steps), bidirectional LSTM to parse temporal contextual relationships, and a cross-attention mechanism to dynamically fuse spatiotemporal features, outputting anomaly confidence. The adaptive protection module introduces a dynamic threshold engine, adjusting dual threshold boundaries in real-time through a risk index model (integrating factors such as CPU load, memory pressure, and network anomalies), and employing reinforcement learning optimization strategies. The model evolution module integrates an adversarial example generator and an online incremental learning mechanism, generating semantically legitimate adversarial sequences through a Wasserstein GAN adversarial network, and continuously optimizing the model using an elastic weight consolidation algorithm to ensure adaptive protection against new attack patterns.

[0085] In another aspect, this invention also provides a deep learning-based method for detecting software behavior anomalies, comprising the following steps:

[0086] Step 1: Capture the process's call sequence and contextual semantic information in real time, and extract multi-dimensional feature vectors.

[0087] Step 2: After receiving the multi-dimensional feature vector, it is analyzed through its internal hierarchical neural symbol network. The output fused features are dynamically weighted through a gated symbolic attention mechanism, triggering the causal inference engine and the dynamic response framework based on the reinforcement decision tree.

[0088] Step 3: Includes a causal inference engine and a dynamic response framework based on a reinforced decision tree, which adjusts the protection intensity in real time according to the environmental risk index, and sends feedback signals back to the anomaly detection module to optimize the threshold.

[0089] Step 4: Integrate the adversarial example generator with the online incremental learning mechanism to achieve the self-evolution capability of the detection model.

[0090] The above embodiments are provided for those skilled in the art to implement or use the present invention. Those skilled in the art can make various modifications or changes to the above embodiments without departing from the spirit of the present invention. Therefore, the scope of protection of the present invention is not limited to the above embodiments, but should be the maximum scope that conforms to the innovative features mentioned in the claims.

[0091] Experimental Design and Result Analysis:

[0092] Experimental Objective: To verify the advantages of the causal inference engine in reducing false positive rates and improving attack interception efficiency. Experimental Environment:

[0093] Hardware: Edge nodes (4-core CPU / 8GB RAM), cloud analytics center;

[0094] Dataset: Includes normal system calls (CIC-IDS2017) and attack samples (Dirty Pipe exploit, DDoS attack).

[0095] Baseline comparison: Traditional static threshold scheme.

[0096] Experimental scenario:

[0097] High-load scenarios: Inject CPU stress testing tool (stress-ng) to achieve CPU utilization of 85%;

[0098] Hybrid attack scenario: Simultaneously launching a DDoS attack (SYN Flood) and a covert vulnerability exploit (Dirty Pipe).

[0099] Table 1

[0100]

[0101] The experimental results are shown in Table 1. As can be seen from Table 1, the dynamic threshold scheme reduces the false alarm rate by 63.6% under high load, proving that its environmental adaptability is significantly better than that of the static threshold; the false negative rate for hybrid attacks is reduced by 73.3%, thanks to the synergy of spatiotemporal hybrid neural network and multi-level response strategy; the response latency is optimized to the millisecond level, meeting the real-time protection requirements.

[0102] In adversarial sample attack tests, after integrating the gradient masking mechanism, the success rate of bypassing adversarial samples decreased from 35% in traditional solutions to 7%, proving its effectiveness in defending against white-box attacks.

Claims

1. A deep learning-based software behavior anomaly detection system, characterized in that, Includes the following modules: Behavior monitoring module: Captures process call sequences and contextual semantic information in real time, and extracts multi-dimensional feature vectors; Anomaly detection module: After receiving multi-dimensional feature vectors, it analyzes them through its internal hierarchical neural symbol network and outputs fused features that are dynamically weighted through a gated symbolic attention mechanism to trigger the causal inference engine of the adaptive protection module and the dynamic response framework based on a reinforcement decision tree. The anomaly detection module receives multi-dimensional feature vectors and analyzes them through its internal hierarchical neural symbolic network. It uses a hierarchical neural symbolic network to parse the logical constraints of calls based on a formal rule engine, captures short / medium / long-range call dependencies through a hierarchical attention mechanism, identifies anomaly patterns with temporal dependencies using causal dilated convolution kernels, and designs a differentiable logic reasoning unit to fuse the rule violation vector output by the symbolic layer with the neural layer features through tensor multiplication. The output fused features are dynamically weighted through a gated symbolic attention mechanism, triggering the causal inference engine of the adaptive protection module and the dynamic response framework based on a reinforced decision tree. The hierarchical neural symbolic network in the anomaly detection module is specifically implemented as follows: Symbolic reasoning layer: Stores a predefined set of formal security rules. Each rule is composed of logical predicates. It takes a multi-dimensional feature vector and context features as input and outputs a rule violation vector. The rule violation vector is normalized by the Sigmoid function. Each dimension corresponds to the violation strength value of the security rule. Neural feature extraction layer: Several parallel attention heads are used, employing a three-scale capture strategy: short-term scale, with time steps of 0-10 steps in the sequence, capturing sudden call patterns through sliding window local attention; medium-term scale, with time steps of 10-100 steps in the sequence, identifying periodic attack chains based on the hollow attention mechanism; On a long-term scale, with time steps exceeding 100 in the sequence, global attention is used to capture the latency behavior of APT attacks. Temporal causal convolution module: Uses dilated convolution kernels with temporal causal constraints to ensure that feature extraction at each time step depends only on historical data; Adaptive protection module: Includes a causal inference engine and a dynamic response framework based on reinforced decision tree. It adjusts the protection intensity in real time according to the environmental risk index, and sends feedback signals back to the anomaly detection module to optimize the threshold. Model Evolution Module: Integrates an adversarial example generator and an online incremental learning mechanism to enable the detection model to evolve on its own.

2. The software behavior anomaly detection system based on deep learning according to claim 1, characterized in that, The behavior monitoring module extracts multi-dimensional feature vectors as follows: The software call ID is mapped to a continuous semantic vector space to capture the core call semantics. Simultaneously, runtime context features such as process permissions, resource access patterns, and call stack depth are extracted and fused through a bidirectional semantic modulation layer: the semantic vector serves as the main signal, and the context features are weighted, filtered, and refined through a dynamic modulation gate to generate context-enhanced semantic units. The context-enhanced semantic unit is structurally reshaped into a three-dimensional feature tensor: the first dimension is the temporal length, the second dimension is the semantic depth, and the third dimension is the decoupled context feature channel. A cross-channel residual mutual injection module is applied: the feature map of each context feature channel receives the refined information from all other channel features after depthwise separable convolution processing, and performs adaptive residual fusion. A gated feature evolution network is constructed to automatically evaluate the importance weight of each context channel and achieve the optimal channel combination under lightweight conditions. The input is a three-dimensional feature tensor that has already undergone cross-channel interaction. By constructing a gated feature evolution network, channel importance evaluation and multi-scale feature collaboration are realized. A feature pyramid is constructed, and the low-level feature map and the high-level feature map interact through bidirectional feature guidance gates: the high-level features guide the low-level features to focus on key information, while the changes in the nodes feed back and correct the abstract expression of the high-level features. The 3D feature tensor, which has undergone gated feature evolution, is compressed into the final feature vector through an adaptive spatiotemporal focusing layer. This layer can adaptively aggregate key time segments and spatial dimension information according to different samples, retain discriminative features, filter redundant noise, and output a robust multi-dimensional feature vector.

3. The deep learning-based software behavior anomaly detection system according to claim 1, characterized in that, The implementation of the causal inference engine includes: first, establishing an environmental risk index calculation model, quantifying the causal effects of dynamic factors through a counterfactual reasoning framework, specifically utilizing Linux cgroups to implement resource intervention and calculate the difference in anomaly probabilities; second, constructing a dual-threshold drift mechanism to dynamically adjust the warning threshold and blocking threshold with the causal risk index, and setting a causal confidence constraint condition of ΔP>0.2; then, introducing a time decay factor to handle periodic legitimate call patterns, identifying periodic behavior through an autocorrelation model and automatically reducing threshold sensitivity by applying an exponential decay formula; finally, adopting a causal reinforcement learning optimization strategy, fusing risk derivative features in the state space, defining the threshold adjustment range in the action space, and injecting a counterfactual reward correction mechanism into the reward function to achieve predictive decision-making.

4. The deep learning-based software behavior anomaly detection system according to claim 1, characterized in that, The dynamic response framework based on reinforcement decision trees implements a three-level hierarchical response mechanism. The response threshold is dynamically adjusted by the reinforcement learning model based on real-time environmental risk and historical feedback, and is not a fixed value. The node depth of the decision tree is correlated with the response level, forming a progressive response strategy from early warning to containment. Probing level response: determined by shallow nodes of a decision tree with a depth of 3 or less; when a behavioral entropy value shows an uncertain increase, a deviation in resource access patterns is detected, and the attack chain confidence is low than the threshold, a primary containment action is triggered; this level of action mainly focuses on interference and isolation with minimal resource consumption, combined with lightweight container isolation to limit the potential attack surface; the judgment threshold for this level is optimized online based on false alarm feedback using the Q-learning algorithm; Propagation-level response: Determined by mid-level nodes in a decision tree with a depth of 4 to 6; when a clear attempt to escalate privileges is detected, accompanied by persistent residency and continuous resource abuse, an active containment action is triggered; this level of action aims to disrupt the attack chain, including dynamic privilege escalation and resource access redirection; The intensity of action execution is dynamically adjusted in the state space through a reinforcement learning model; Explosive response: Determined by deep nodes of a decision tree with a depth of 7 or greater; when kernel-level anomalous behavior matching a zero-day vulnerability is detected, and the causal threat index indicates an immediate substantial risk, system-level defense and forensic actions are triggered; these actions include memory state preservation, immediate blocking of malicious network connections, and proactive security probe injection for reverse analysis; the entire decision tree is structurally optimized every 24 hours through Monte Carlo tree search, and adversarial pruning strategies are used to remove invalid branches.

5. The software behavior anomaly detection system based on deep learning according to claim 1, characterized in that, The adversarial example generator includes: generating semantically legal adversarial call sequences using an improved generative adversarial network Wasserstein GAN; designing a call semantic preservation loss function to ensure the grammatical correctness of adversarial examples; constructing an adversarial example knowledge base for model retraining; and finally integrating a gradient masking mechanism to defend against white-box attacks.

6. The deep learning-based software behavior anomaly detection system according to claim 1, characterized in that, The model evolution module includes: first, constructing a drift detector to identify concept drift; then, using an elastic weight consolidation algorithm to prevent catastrophic forgetting; second, designing a dynamic memory bank to store key samples; and finally, supporting a hot update mechanism to achieve seamless model upgrades.

7. A deep learning-based method for detecting software behavior anomalies, used to implement the software behavior anomaly detection system according to any one of claims 1 to 6, characterized in that, Includes the following steps: Step 1: Capture the process's call sequence and contextual semantic information in real time, and extract multi-dimensional feature vectors; Step 2: After receiving the multi-dimensional feature vector, it is analyzed through its internal hierarchical neural symbol network. The output fused features are dynamically weighted through a gated symbolic attention mechanism, triggering the causal inference engine and the dynamic response framework based on the reinforcement decision tree. Step 3: Includes a causal inference engine and a dynamic response framework based on a reinforced decision tree, which adjusts the protection intensity in real time according to the environmental risk index, and sends feedback signals back to the anomaly detection module to optimize the threshold. Step 4: Integrate the adversarial example generator with the online incremental learning mechanism to achieve the self-evolution capability of the detection model.

Images