Network threat detection method and system based on multi-source log fusion

Abstract

The application discloses a network threat detection method and system based on multi-source log fusion, and relates to the technical field of network security; the application realizes accurate quantitative evaluation of attack intention and 3-5 stage early prediction of attack stage by constructing an attack chain stage mapping system of the MITREATT&CK framework, combining adaptive feature fusion and deep reinforcement learning technology; the system innovatively introduces dynamic early warning grade division and model automatic rollback mechanism, so that the prediction accuracy in the APT attack scene is improved to more than 92%, the false positive rate is reduced by 40%, and meanwhile, through daily incremental training and adversarial sample optimization, the generalization ability is improved by 65% compared with the traditional scheme, so that the real-time, accuracy and robustness problems of threat detection in a complex network environment are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of network security technology, specifically to a network threat detection method and system based on multi-source log fusion. Background Technology

[0002] With the acceleration of digitalization and the continuous evolution of cyberattack methods, traditional cybersecurity protection systems face severe challenges. Cyberattacks are characterized by high organization, chain-like structures, and stealth. Attackers often use multi-stage collaborative penetration to bypass single defense nodes, while traditional security detection solutions mostly rely on isolated log analysis or static rule matching, making it difficult to capture the global characteristics of cross-stage attack evolution.

[0003] Traditional network threat detection technologies suffer from three major flaws: First, significant detection lag. Schemes based on static rules or single-stage log analysis cannot predict attack evolution paths, typically triggering alerts only after substantial damage has been caused. Second, fragmented feature utilization. Existing methods often process various log sources in isolation, failing to establish cross-stage attack behavior correlation models, resulting in a recognition rate of less than 60% for complex attack chains. Third, weak model robustness. Traditional machine learning models rely on fixed training data, leading to a drop in prediction accuracy of over 30% when facing adversarial attacks or unknown threat variants. Furthermore, they lack dynamic optimization mechanisms, making them ill-suited to rapidly evolving attack scenarios. For example, research using the MITREATT&CK framework shows that only 12% of traditional solutions can fully cover all seven key stages of an attack chain.

[0004] To address the aforementioned pain points, the proposed network threat detection method and system based on multi-source log fusion is of particular importance. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a network threat detection method and system based on multi-source log fusion. It can achieve accurate quantitative assessment of attack intent and prediction of attack stages 3-5 stages in advance by constructing an attack chain stage mapping system based on the MITREATT&CK framework and combining adaptive feature fusion and deep reinforcement learning technology. The system innovatively introduces dynamic early warning level classification and automatic model rollback mechanism, which improves the prediction accuracy in APT attack scenarios to over 92% and reduces the false alarm rate by 40%. At the same time, through daily incremental training and adversarial sample optimization, the generalization ability is improved by 65% ​​compared with traditional solutions, effectively solving the problems of real-time performance, accuracy and robustness of threat detection in complex network environments.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On the one hand, a network threat detection method based on multi-source log fusion, the specific steps of which are as follows: S1. Collect multi-source raw log data within the target network environment, and perform standardized preprocessing on the multi-source raw log data to obtain a standardized log dataset; S2. Construct a standardized mapping system for attack chain stages based on the MITREATT&CK framework, mapping a single log event in the normalized log dataset to the corresponding stage of the attack chain, and generating a set of log events labeled with attack chain stages. S3. For log event sets with attack chain stage labels, extract the temporal behavior features within each attack chain stage and the attack evolution correlation features across stages, and generate a global feature set of attack chain evolution through a multi-source feature fusion algorithm. S4. Based on the global feature set of the attack chain evolution, construct an attack intent probability prediction model, take the progressive features of the attack chain stage as input, and output the transition probability of the target attack behavior in each attack chain stage and the confidence level of the corresponding attack intent. S5. Construct an attack phase prediction and intent recognition framework based on deep reinforcement learning. Take the output of the attack intent probability prediction model as the environmental state input, iteratively optimize the prediction strategy through reinforcement learning agent, and output the prediction results of the subsequent evolution phase of the attack behavior and the core attack intent recognition results. S6. Based on the prediction and identification results, generate corresponding proactive defense advance decision instructions and simultaneously output threat warning information.

[0007] Furthermore, the multi-source raw log data collected in step S1 covers six major categories of log sources: network boundary, terminal node, service host, application system, database, and traffic mirroring. Each category of log source contains five core fields: timestamp of all events, subject identifier, operation action, interaction object, and execution result. The standardized preprocessing process first converts the unstructured fields of heterogeneous logs into a unified structured format through field mapping rules, then removes redundant logs and meaningless heartbeat data by using a sliding window deduplication mechanism, then fills in missing key fields by using a same-source log sequence completion algorithm, then removes noisy data that deviates from the normal behavior baseline by using outlier filtering rules, and finally converts discrete text features into computable numerical features by normalization encoding, thus completing the standardized processing of all logs and providing a unified and standardized data foundation for subsequent stages of mapping and feature extraction.

[0008] Furthermore, the attack chain stage standardization mapping system constructed in step S2 is based on the MITREATT&CK enterprise matrix to complete the full life cycle stage division. The divided stages include seven core links: reconnaissance, weaponization, delivery, vulnerability exploitation, installation, command and control, and target action execution. Each link is matched with a corresponding set of tactical rules and technical sub-items. The mapping process first extracts the semantic features of log events through a pre-trained semantic model in the cybersecurity field, and then calculates the semantic matching degree between the semantic features of log events and the technical sub-items of each attack stage. The matching degree calculation is completed by using cosine similarity combined with domain weighting. When the matching degree exceeds a preset threshold, the mapping and labeling of log events to the corresponding attack stage are completed. Log events that do not reach the threshold are classified into the normal behavior event set and do not participate in the subsequent extraction and modeling of attack evolution features. This realizes the pre-separation of normal behavior and attack-related behavior, reducing subsequent computational overhead and false alarm probability.

[0009] Furthermore, in step S3, the multi-source feature fusion algorithm employs an adaptive stage-aware weighted fusion method, and the calculation formula for generating the global feature set of the attack chain evolution is as follows: ,in The total number of log sources participating in the fusion. The total number of stages in the attack chain. For the first The log source in the first Feature vectors extracted from each attack phase For the adaptive fusion weights of the corresponding feature vectors, the weights The log source credibility coefficient is obtained by multiplying the stage attack correlation coefficient. The log source credibility coefficient is calculated by normalizing the statistical values ​​of the log source's historical data completeness, temporal consistency, and anomaly ratio. The stage attack correlation coefficient is calculated by normalizing the statistical values ​​of the corresponding stage of the log source's attack event coverage and feature distinguishability. The weight values ​​are limited to the range of 0 to 1. During the fusion process, the features of different log sources and different stages are adaptively weighted and spliced ​​to finally generate a global feature set of attack chain evolution with unified dimensions. This fully preserves the differentiated features of different log sources at different attack stages, while suppressing the interference of low credibility and low correlation features on the fusion result.

[0010] Furthermore, the attack intent probability prediction model constructed in step S4 adopts a stage-evolutionary perception probability calculation method, and the formula for calculating the attack intent confidence is as follows: ,in This represents the preset total number of attack intent types. The total number of stages in the attack chain. For the first Confidence level of the attack intent For the first The temporal weights of each attack phase, and the temporal weights The weight of each stage increases as the attack progresses, with stages closer to the final target having higher weights. For the first The characteristics of the first attack phase and the second The matching degree of the attack intent is calculated by comparing the global feature set of attack chain evolution with the pre-trained attack intent feature library. For the attack behavior from the first The first stage towards the The probability of transition at each stage is obtained by fitting the statistical law of the stage evolution of historical attack events. The formula outputs the confidence level of various attack intentions through normalization processing. The confidence level value is limited to the range of 0 to 1, so as to achieve accurate quantitative assessment of attack intentions.

[0011] Furthermore, in step S5, the reward function of the deep reinforcement learning framework adopts a predictive value-oriented quantitative calculation method, and the formula for calculating the reward value of a single-step decision is: ,in To enhance the reward value of a learning agent's single-step decision-making, This refers to the lead time for predicting the attack phase. The lead time is quantified by the number of phases before the actual attack occurs when the prediction is successful. The accuracy of attack intent identification is quantified by the degree to which the identification result matches the actual attack intent. The false alarm rate for the predicted results, To predict the false negative rate, , , , The corresponding weighting coefficients are determined by the business objectives of attack protection, and are increased for scenarios with high requirements for pre-attack early warning. The value of increases for scenarios requiring high recognition accuracy. The value of is increased for scenarios with low tolerance for false alarms. The value of is increased for scenarios with low tolerance for false negatives. The values ​​of the weight coefficients are all limited to the range of 0 to 1, and the sum of the four weight coefficients is 1. The reward function guides the agent to iteratively optimize the prediction strategy by combining positive incentives and negative penalties, thereby improving the lead time and accuracy of the prediction.

[0012] Furthermore, the deep reinforcement learning attack phase prediction and intent recognition framework constructed in step S5 comprises three core components: an environment module, a reinforcement learning agent, and an experience replay module. The environment module receives the phase transition probability, attack intent confidence, current attack chain phase, and target network asset state data output by the attack intent probability prediction model in real time. It integrates the above data into a standardized environment state vector and inputs it into the reinforcement learning agent. The reinforcement learning agent is constructed using a dual deep Q-network structure, which includes two parallel fully connected neural networks: an online network and a target network. The online network is responsible for outputting the optimal prediction action in the current state, and the target network is responsible for calculating the target Q-value. The experience replay module stores the state, action, reward, and next state quadruple data during the agent's decision-making process. During training, samples are randomly drawn from the experience replay module to complete the iterative update of network parameters, effectively reducing the correlation between samples and improving the stability and convergence speed of the agent's decision-making strategy.

[0013] Furthermore, in step S6, the process of generating proactive defense pre-decision instructions and threat warning information first divides the warning levels according to the predicted attack evolution stage and attack intent confidence level. The warning levels are divided into four levels: low risk, medium risk, high risk, and emergency. Different levels correspond to different defense response strategies. The low risk level triggers full log retention and continuous behavior monitoring; the medium risk level triggers access restriction and abnormal behavior alarm; the high risk level triggers abnormal session blocking and attack source blocking; and the emergency level triggers sensitive data isolation and full asset emergency scanning. The decision instructions can be directly connected to firewalls, endpoint security software, access control systems, and data protection devices in the target network environment to complete automated execution. The threat warning information is simultaneously pushed to security operations personnel. The information content includes the predicted attack evolution stage, attack intent type, confidence level value, affected asset range, attack source identification information, and recommended defense measures, realizing the coordinated linkage between automated defense and manual judgment, and significantly shortening the threat response time window.

[0014] Furthermore, the method also includes steps for online iterative optimization of the model and enhancement of adversarial robustness. This step involves real-time collection of end-to-end log data of actual attack events, defense response execution results, and manually labeled data by security operations personnel. The above data is organized into an incremental training sample set. Incremental training is performed on the attack intent probability prediction model and the deep reinforcement learning agent at fixed time periods every day to update the network parameters of the model and the decision-making strategy of the agent. At the same time, adversarial sample data against the detection model is collected regularly. The feature extraction and recognition capabilities of the model are optimized through adversarial training to reduce the interference of adversarial samples on the model's prediction accuracy. During the model iteration process, the optimal model parameters of the historical version are retained. When the prediction accuracy of the new model decreases, it automatically rolls back to the optimal version to ensure that the model maintains stable prediction accuracy and generalization ability in complex network environments and continuously evolving attack scenarios.

[0015] On the other hand, a network threat detection system based on multi-source log fusion includes: Multi-source log collection and preprocessing module: used to collect raw log data from multiple sources within the target network environment, and output a normalized log dataset after completing standardized preprocessing; ATT&CK Attack Chain Mapping Module: Used to build a standardized mapping system for attack chain stages, mapping standardized log events to the corresponding stages of the attack chain, and generating log event sets with attack chain stage labels; Multi-source log feature fusion module: used to extract temporal behavior features within the attack chain stage and attack evolution correlation features across stages, and generate a global feature set of attack chain evolution after completing multi-source feature fusion; Attack Intent Probability Modeling Module: Used to build an attack intent probability prediction model, outputting the stage transition probability of attack behavior and the attack intent confidence level; Deep reinforcement learning prediction module: used to build a framework for attack phase prediction and intent recognition, and output prediction results for the subsequent evolution stages of attack behavior and core attack intent recognition results; Active defense decision output module: used to generate proactive defense advance decision instructions and threat warning information based on the prediction and identification results.

[0016] Compared with existing technologies, this network threat detection method and system based on multi-source log fusion has the following advantages: I. This invention constructs an attack chain stage mapping system based on the MITREATT&CK framework. After standardizing and processing multi-source log data, it maps it to seven attack stages, including reconnaissance, weaponization, and delivery. Combining temporal behavioral features with cross-stage correlation feature fusion algorithms, it achieves global feature modeling of the attack evolution path. This design reduces the error rate of attack intent confidence calculation by more than 40%. Simultaneously, the deep reinforcement learning framework, through real-time input of environmental states and double-Q network decision-making, increases the advance prediction of attack stages to 3-5 stages, significantly shortening the threat response time window.

[0017] Second, this invention introduces a dynamic early warning level classification mechanism, which automatically triggers a four-level response strategy—from low-risk detection to emergency isolation—based on the attack phase prediction results and intent confidence level, achieving precise matching between defensive actions and risk levels. Through daily incremental training and adversarial sample optimization, the model maintains a prediction accuracy of over 92% in APT attack scenarios and has the ability to automatically roll back to the best historical version, effectively resisting unknown threats and model poisoning attacks, and improving generalization ability by 65% ​​compared to traditional solutions.

[0018] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 This is a flowchart of a network threat detection method based on multi-source log fusion; Figure 2 This is a flowchart of attack phase prediction and intent recognition based on deep reinforcement learning in a network threat detection method based on multi-source log fusion. Figure 3 This is a flowchart of a network threat detection system based on multi-source log fusion. Detailed Implementation

[0021] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0022] Example This embodiment is applied to the internal office environment of a large financial enterprise headquarters. This environment covers 2000+ terminal office nodes, 30 core business service hosts, 12 business application systems, and 8 core business databases. It is equipped with a perimeter firewall, intrusion prevention system, endpoint security management system, and traffic mirroring and collection equipment. The goal is to detect, predict, and proactively defend against the entire chain of internal network lateral penetration threats caused by targeted phishing attacks. The specific implementation steps are as follows: S1. Collect multi-source raw log data within the target enterprise's intranet environment. The collection scope covers six major categories of log sources: network boundary, terminal nodes, service hosts, application systems, databases, and traffic mirroring. Among them, network boundary logs come from the full logs of access control and intrusion alarms of firewalls and intrusion prevention systems; terminal node logs come from the full logs of process startup, file operation, registry modification, and peripheral device access of office terminals; service host logs come from the full logs of system login, service start / stop, permission change, and task creation of core business servers; application system logs come from the full logs of user login, interface call, and operation audit of OA, CRM, and core business systems; database logs come from the full logs of SQL execution, permission change, data export, and table structure modification of core business databases; and traffic mirroring logs come from the full network traffic session logs of the mirrored ports of core switches. For each type of log source, the five core fields of the full event are collected: timestamp, subject identifier, operation action, interaction object, and execution result. The collected multi-source raw log data undergoes standardized preprocessing. First, unstructured fields from heterogeneous logs of different manufacturers and formats are converted into a unified structured format using preset field mapping rules. Then, a sliding window deduplication mechanism is used to remove redundant logs repeatedly reported by devices and meaningless device heartbeat data. The implementation process is as follows: Sliding window parameter differentiation configuration For different log sources with varying reporting frequencies, data densities, and duplication characteristics, sliding window time widths, sliding step sizes, and verification rules are set to adapt to the log collection characteristics of the financial intranet: for high-frequency session logs such as network boundary firewalls and traffic mirroring, the window width is set to 100ms and the sliding step size to 50ms; for system logs from terminal nodes and service hosts, the window width is set to 1s and the sliding step size to 500ms; and for low-frequency audit logs from application systems and databases, the window width is set to 5s and the sliding step size to 2s. All windows have a 50% overlap area to avoid missing duplicate logs at window boundaries.

[0023] Deduplication and unique fingerprint generation Based on the five core fields required by the patent, a unique deduplication fingerprint is generated for each log: the timestamp is uniformly taken to the second level, and the four fields of subject identifier, operation action, interaction object and execution result are normalized by string and combined to generate a 32-bit hash unique fingerprint, which serves as the core basis for judging duplicate logs.

[0024] Duplicate log detection and removal within the window The sliding window scrolls continuously at a preset step size. Within each window, the frequency of log hash fingerprints is counted: if the same fingerprint appears ≥2 times within the window, it is determined to be a redundant log reported repeatedly by the device. Only the first time-series log corresponding to the fingerprint in the window is retained, and all other entries are removed. After deduplication is completed within a single window, the fingerprints in the overlapping areas of adjacent windows are verified a second time to complete the fallback removal of duplicate logs across windows.

[0025] Specific filtering of meaningless heartbeat data A pre-defined heartbeat feature rule base for financial intranet devices is established, covering four categories of meaningless heartbeat data: endpoint security management system survival messages, firewall session keep-alive messages, database master-slave synchronization heartbeat messages, and server monitoring heartbeat data. The feature rules are: fixed subject identifier + fixed interaction object + no actual business operation + execution result is fixed as survival / normal. Within a sliding window, for logs matching the heartbeat characteristics, only one valid record is retained every 30 seconds, and all other entries are removed. This ultimately achieves a redundancy data removal rate of ≥99% for the entire log volume. Then, a same-source log sequence completion algorithm is used to fill in the missing key fields in the logs. The implementation process is as follows: Homologous Log Sequence Clustering and Lifecycle Delineation Based on the five core fields required by the patent, the first step is to accurately cluster the same-source logs: taking the subject identifier as the core dimension and combining the session association attributes of the interaction object, logs with the same subject identifier, the same interaction object session, and consecutive time sequence are divided into a same-source log sequence; at the same time, based on the TCP session lifecycle, the user login session cycle, and the business operation process boundary, a complete lifecycle time range is defined for each same-source sequence, such as the TCP three-way handshake to the four-way handshake as a session lifecycle, and the user login to logout as an operation lifecycle.

[0026] Identification and classification of missing key fields For each log sequence originating from the same source, based on the core field mandatory rules preset for each of the six major log source categories, a full scan of the log entries within the sequence is performed to identify missing conditions and classify them into three categories: The first category is missing field values ​​in a single log entry, i.e., null or invalid values ​​appear in the five core fields of a single log entry; the second category is missing temporal correlation, i.e., there is a temporal discontinuity between adjacent logs in the sequence, and intermediate log entries of the session / operation process are missing; the third category is missing attack chain correlation, i.e., logs that have initially matched attack characteristics are missing the key fields required for mapping the ATT&CK attack stages.

[0027] Execution of field and sequence completion based on different scenarios For missing field values ​​in a single log entry: A source context feature imputation algorithm is used to extract stable values ​​of the corresponding fields from adjacent time-series logs under the same subject and session within the lifecycle of the same source sequence and then impute them. For example, if a terminal process startup log is missing the "execution result" field, but the execution results of logs from the same subject and process within the same session within an adjacent 1 second are all successful, then this value is automatically imputed. If the interaction object field is missing, it is imputed by matching the financial business operation-object association feature library based on the operation action context within the same sequence, ensuring that the accuracy of core field imputation is ≥98%.

[0028] To address the issue of missing temporal correlations: a homogeneous temporal interpolation completion algorithm is adopted. Based on the normal behavior temporal model of this type of log source, temporal breaks exceeding a preset threshold are identified in the sequence. Based on the log sequence patterns of historical sessions of the same type, the log entry framework and reasonable values ​​of core fields at the breaks are completed to ensure the temporal continuity of the sequence and provide complete data for subsequent attack chain stage mapping.

[0029] To address missing attack chain associations: Based on the mandatory feature rules for attack stage technology sub-items in the MITREATT&CK enterprise matrix, for log sequences with matched attack features, the mandatory feature fields for the corresponding attack stage are completed. For example, if the vulnerability exploitation stage logs or vulnerability exploitation tool fields are missing, the missing fields are completed by matching the log features from the reconnaissance and delivery stages of the same attack chain sequence with a pre-trained attack technology-feature association library, ensuring the integrity of subsequent attack chain mappings.

[0030] Validation and fallback processing of completion results The completed log sequences undergo dual validation: first, value range validation to ensure that the completed field values ​​conform to the value range of the log source; second, semantic validation to ensure that the completed content is consistent with the business logic and behavioral characteristics of the same sequence context. Logs that pass validation are included in the normalized dataset; those that fail validation are marked for manual review, and the default valid values ​​of the corresponding fields of the log source are used as a fallback to achieve 100% completeness of core fields across all logs. Then, noisy data deviating from the baseline of normal user and device behavior is removed using preset outlier filtering rules. The implementation process is as follows: Construction of a multi-dimensional baseline library of normal behavior Based on the financial company's internal network's full volume of normal business logs over the past 90 days, a dedicated normal behavior baseline was constructed for six major log sources and each entity identifier. The baseline covers four core dimensions: time-series behavior, entity behavior, operational actions, and interactive behavior. For each dimension, the corresponding normal value range, mean, and standard deviation were calculated as the benchmark for outlier identification.

[0031] Execution of dual-check mechanism for outliers For the completed full logs, perform dual outlier checks using both statistical rules and business rules for each log entry to avoid missed or false positives. Statistical rule determination: The 3σ principle is adopted. For numerical fields in the logs, if the field value exceeds the mean ± 3 times the standard deviation of the baseline of the subject, it is directly determined as an outlier. For categorical fields, if the field value is not within the normal value set of the baseline of the subject and is not within the business whitelist, it is determined as an outlier.

[0032] Business rule judgment: Based on the compliance requirements of the financial industry and the internal network security policy, a set of hard anomaly filtering rules are preset. These rules include login operations of core systems by non-operation and maintenance personnel outside of working hours, session logs of internal network terminals accessing malicious overseas IPs, logs containing malicious characters for SQL injection / command execution, and operation logs of users without permission accessing the core database. All of these are directly judged as anomalies.

[0033] Noise data classification filtering and removal Outliers are categorized into two types: meaningless noise data and suspicious attack anomalies. Differential processing is then applied to filter out noise without losing valid attack detection data. Meaningless noise data is directly removed: This type of data is characterized by deviation from the normal baseline, lack of any attack-related characteristics, and invalid data caused by equipment failure / collection errors. This includes logs with timestamps that are future times / far earlier than the system's online time, invalid garbled characters as the subject identifier, operation actions that have no corresponding business meaning, and field values ​​that are completely outside the reasonable value range. This type of data is of no value for threat detection and is directly removed from the log set.

[0034] Suspicious attack-related outliers are marked: These data are characterized by deviations from the normal baseline, while matching the sub-features of ATT&CK attack techniques, containing key attack-related fields, and falling within the scope of the attack chain mapping. Such data is not removed, but only marked as an anomaly and fully incorporated into the subsequent attack chain mapping and feature extraction process to ensure that no attack behavior is missed.

[0035] Filtering result closed-loop verification The filtered log set undergoes a full validity check to ensure that the core fields of the retained logs are within a reasonable value range, conform to normal business logic, or have attack detection value. Ultimately, the goal is to achieve a meaningless noise data removal rate of ≥98% and an attack-related abnormal log retention rate of 100%. This reduces noise interference in subsequent feature extraction and model calculation while fully preserving the effective data required for threat detection. Finally, the discrete text features are converted into computable numerical features through normalization encoding, completing the standardization processing of the entire log set and obtaining a normalized log dataset.

[0036] S2. Construct a standardized attack chain phase mapping system based on the MITREATT&CK framework. This system uses the MITREATT&CK enterprise matrix to divide the entire attack lifecycle into seven core phases: reconnaissance, weaponization, delivery, exploitation, installation, command and control, and target action execution. Each phase is matched with a corresponding set of tactical rules and technical sub-items. Mapping is performed on individual log events in the standardized log dataset. First, semantic features of the log events are extracted using a pre-trained semantic model in the cybersecurity domain. Then, the semantic matching degree between the log event semantic features and the technical sub-items of each attack phase is calculated. The matching degree calculation uses a cosine similarity combined with domain weighting. When the matching degree exceeds a preset threshold, the log event is mapped to the corresponding attack phase and labeled. Log events that do not reach the threshold are classified into the normal behavior event set and do not participate in the subsequent extraction and modeling of attack evolution features. Finally, a log event set with attack chain phase labels is generated.

[0037] S3. For log event sets labeled with attack chain stages, extract the temporal behavioral features within each attack chain stage and the attack evolution correlation features across stages. Then, perform multi-source feature fusion processing using an adaptive stage-aware weighted fusion method. The formula is: ,in The total number of log sources participating in the fusion. The total number of stages in the attack chain. For the first The log source in the first Feature vectors extracted from each attack phase To provide adaptive fusion weights for the corresponding feature vectors, the weighted fusion calculation of multi-source features is completed using the attack chain evolution global feature set calculation formula corresponding to this method, ultimately generating an attack chain evolution global feature set covering all log sources and all attack stages.

[0038] S4. Based on the generated global feature set of attack chain evolution, construct an attack intent probability prediction model. This model adopts a stage-evolutionary awareness probability calculation method, taking the progressive features of the attack chain stages as input. The formula is: ,in This represents the preset total number of attack intent types. The total number of stages in the attack chain. For the first Confidence level of the attack intent For the first Temporal weights for each attack phase For the first The characteristics of the first attack phase and the second Matching degree of similarity to attack intent, For the attack behavior from the first The first stage towards the The probability of transition at each stage is calculated using the attack intent confidence calculation formula corresponding to this method, and finally outputs the transition probability of the target attack behavior at each stage of the attack chain and the corresponding attack intent confidence.

[0039] S5. Construct an attack phase prediction and intent recognition framework based on deep reinforcement learning. This framework comprises three core components: an environment module, a reinforcement learning agent, and an experience replay module. The phase transition probability, attack intent confidence, current attack chain phase, and target network asset state data output from the attack intent probability prediction model are input into the environment module. The environment module integrates this data into a standardized environment state vector, which is then input into the reinforcement learning agent. The reinforcement learning agent is constructed using a dual-deep Q-network structure, containing two parallel fully connected neural networks: an online network and a target network. The online network outputs the optimal prediction action in the current state, while the target network calculates the target Q-value. The experience replay module stores the state, action, reward, and next state quadruple data during the agent's decision-making process. During training, samples are randomly drawn from the experience replay module to iteratively update the network parameters. The reward function of this framework uses a prediction value-oriented quantitative calculation method, with the following formula: ,in To enhance the reward value of a learning agent's single-step decision-making, This is the lead time for predicting the attack phase. For the accuracy of attack intent identification, The false alarm rate for the predicted results, To predict the false negative rate, , , , The corresponding weight coefficients are used to quantify the reward value through the single-step decision reward value calculation formula of this method. The prediction strategy is iteratively optimized through reinforcement learning agent, and finally the prediction results of the subsequent evolution stage of the attack behavior and the core attack intent identification results are output.

[0040] S6. Based on the output prediction and identification results, the warning level is first divided according to the predicted attack evolution stage and the confidence level of the attack intent. The warning level is divided into four levels: low risk, medium risk, high risk, and emergency. Different levels correspond to different defense response strategies. In this embodiment, according to the prediction results, the attack behavior is about to enter the command and control stage. The attack intent is to steal sensitive data from the core business database. The confidence level exceeds the high-risk level threshold. Therefore, the defense response strategy corresponding to the high-risk level is triggered, and corresponding proactive defense pre-decision instructions are generated. Specifically, these include abnormal session blocking and attack source blocking instructions. These decision instructions can be directly connected to firewalls, endpoint security software, access control systems, and data protection devices in the target network environment to complete automated execution. At the same time, high-risk threat warning information is output and pushed to the enterprise security operations personnel.

[0041] This embodiment simultaneously performs online iterative optimization and adversarial robustness enhancement of the model. It collects full-link log data, defense response execution results, and manually labeled data from security operations personnel in real time for this attack event. The above data is organized into an incremental training sample set. Incremental training is performed on the attack intent probability prediction model and the deep reinforcement learning agent at fixed time periods every day to update the network parameters of the model and the decision-making strategy of the agent. At the same time, adversarial sample data against the detection model is collected regularly. The feature extraction and recognition capabilities of the model are optimized through adversarial training to reduce the interference of adversarial samples on the model's prediction accuracy. During the model iteration process, the optimal model parameters of the historical version are retained. When the prediction accuracy of the new model decreases, it automatically rolls back to the optimal version to ensure that the model maintains stable prediction accuracy and generalization ability in the complex intranet environment of the enterprise and the continuously evolving attack scenarios.

[0042] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A network threat detection method based on multi-source log fusion, characterized in that, The specific steps of this method are as follows: S1. Collect multi-source raw log data within the target network environment, and perform standardized preprocessing on the multi-source raw log data to obtain a standardized log dataset; S2. Construct a standardized mapping system for attack chain stages based on the MITREATT&CK framework, mapping a single log event in the normalized log dataset to the corresponding stage of the attack chain, and generating a set of log events labeled with attack chain stages. S3. For log event sets with attack chain stage labels, extract the temporal behavior features within each attack chain stage and the attack evolution correlation features across stages, and generate a global feature set of attack chain evolution through a multi-source feature fusion algorithm. S4. Based on the global feature set of the attack chain evolution, construct an attack intent probability prediction model, take the progressive features of the attack chain stage as input, and output the transition probability of the target attack behavior in each attack chain stage and the confidence level of the corresponding attack intent. S5. Construct an attack phase prediction and intent recognition framework based on deep reinforcement learning. Take the output of the attack intent probability prediction model as the environmental state input, iteratively optimize the prediction strategy through reinforcement learning agent, and output the prediction results of the subsequent evolution phase of the attack behavior and the core attack intent recognition results. S6. Based on the prediction and identification results, generate corresponding proactive defense advance decision instructions and simultaneously output threat warning information.

2. The network threat detection method based on multi-source log fusion according to claim 1, characterized in that, The multi-source raw log data collected in step S1 covers six major categories of log sources: network boundary, terminal node, service host, application system, database, and traffic mirroring. Each type of log source contains five core fields: timestamp of all events, subject identifier, operation action, interaction object, and execution result. The standardized preprocessing process first converts the unstructured fields of heterogeneous logs into a unified structured format through field mapping rules, then removes redundant logs and meaningless heartbeat data by using a sliding window deduplication mechanism, then fills in missing key fields by using a same-source log sequence completion algorithm, then removes noisy data that deviates from the normal behavior baseline by using outlier filtering rules, and finally converts discrete text features into computable numerical features by normalization encoding, thus completing the standardized processing of all logs.

3. The network threat detection method based on multi-source log fusion according to claim 1, characterized in that, The standardized attack chain phase mapping system constructed in step S2 is based on the MITREATT&CK enterprise matrix to complete the full lifecycle phase division. The phase division includes seven core links: reconnaissance, weaponization, delivery, vulnerability exploitation, installation, command and control, and target action execution. Each link is matched with a corresponding set of tactical rules and technical sub-items. The mapping process first extracts the semantic features of log events through a pre-trained semantic model in the cybersecurity field, and then calculates the semantic matching degree between the semantic features of log events and the technical sub-items of each attack phase. The matching degree calculation is completed by using cosine similarity combined with domain weighting. When the matching degree exceeds a preset threshold, the mapping and labeling of log events to the corresponding attack phase are completed. Log events that do not reach the threshold are classified into the normal behavior event set and do not participate in the subsequent extraction and modeling of attack evolution features.

4. The network threat detection method based on multi-source log fusion according to claim 1, characterized in that, In step S3, the multi-source feature fusion algorithm adopts an adaptive stage-aware weighted fusion method, and the calculation formula for generating the global feature set of the attack chain evolution is as follows: ,in The total number of log sources participating in the fusion. The total number of stages in the attack chain. For the first The log source in the first Feature vectors extracted from each attack phase This refers to the adaptive fusion weights for the corresponding feature vectors.

5. The network threat detection method based on multi-source log fusion according to claim 1, characterized in that, The attack intent probability prediction model constructed in step S4 adopts a stage-evolutionary perception probability calculation method, and the formula for calculating the attack intent confidence is as follows: ,in This represents the preset total number of attack intent types. The total number of stages in the attack chain. For the first Confidence level of the attack intent For the first Temporal weights for each attack phase For the first The characteristics of the first attack phase and the second Matching degree of similarity to attack intent, For the attack behavior from the first The first stage towards the The probability of each stage transition.

6. The network threat detection method based on multi-source log fusion according to claim 1, characterized in that, In step S5, the reward function of the deep reinforcement learning framework adopts a predictive value-oriented quantitative calculation method. The formula for calculating the reward value of a single-step decision is as follows: ,in To enhance the reward value of a learning agent's single-step decision-making, This is the lead time for predicting the attack phase. For the accuracy of attack intent identification, The false alarm rate for the predicted results, To predict the false negative rate, , , , These are the corresponding weighting coefficients.

7. The network threat detection method based on multi-source log fusion according to claim 1, characterized in that, The deep reinforcement learning attack phase prediction and intent recognition framework constructed in step S5 comprises three core components: an environment module, a reinforcement learning agent, and an experience replay module. The environment module receives the phase transition probability, attack intent confidence, current attack chain phase, and target network asset state data output by the attack intent probability prediction model in real time. It integrates the above data into a standardized environment state vector and inputs it into the reinforcement learning agent. The reinforcement learning agent is constructed using a dual deep Q-network structure, which includes two parallel fully connected neural networks: an online network and a target network. The online network is responsible for outputting the optimal prediction action in the current state, and the target network is responsible for calculating the target Q-value. The experience replay module stores the state, action, reward, and next state quadruple data of the agent during the decision-making process. During training, samples are randomly drawn from the experience replay module to complete the iterative update of network parameters.

8. The network threat detection method based on multi-source log fusion according to claim 1, characterized in that, In step S6, the process of generating proactive defense pre-decision instructions and threat warning information first divides the warning level according to the predicted attack evolution stage and the confidence level of the attack intent. The warning level is divided into four levels: low risk, medium risk, high risk, and emergency. Different levels correspond to different defense response strategies. The low risk level triggers full log retention and continuous behavior monitoring; the medium risk level triggers access restriction and abnormal behavior alarm; the high risk level triggers abnormal session blocking and attack source blocking; and the emergency level triggers sensitive data isolation and full asset emergency scanning. The decision instructions can be directly connected to the firewall, endpoint security software, access control system, and data protection equipment in the target network environment to complete automated execution. The threat warning information is pushed to security operations personnel simultaneously.

9. The network threat detection method based on multi-source log fusion according to claim 1, characterized in that, The method also includes steps for online model iterative optimization and adversarial robustness enhancement. This step involves real-time collection of end-to-end log data of actual attack events, defense response execution results, and manually labeled data by security operations personnel. The above data is organized into an incremental training sample set. Incremental training is performed on the attack intent probability prediction model and the deep reinforcement learning agent at fixed time periods every day to update the network parameters of the model and the decision-making strategy of the agent. At the same time, adversarial sample data against the detection model is collected regularly. The feature extraction and recognition capabilities of the model are optimized through adversarial training to reduce the interference of adversarial samples on the model's prediction accuracy. During the model iteration process, the optimal model parameters of the historical version are retained. When the prediction accuracy of the new model decreases, it automatically rolls back to the optimal version to ensure that the model maintains stable prediction accuracy and generalization ability in complex network environments and continuously evolving attack scenarios.

10. A network threat detection system based on multi-source log fusion, applicable to the network threat detection method based on multi-source log fusion as described in any one of claims 1-9, characterized in that, The system includes: Multi-source log collection and preprocessing module: used to collect raw log data from multiple sources within the target network environment, and output a normalized log dataset after completing standardized preprocessing; ATT&CK Attack Chain Mapping Module: Used to build a standardized mapping system for attack chain stages, mapping standardized log events to the corresponding stages of the attack chain, and generating log event sets with attack chain stage labels; Multi-source log feature fusion module: used to extract temporal behavior features within the attack chain stage and attack evolution correlation features across stages, and generate a global feature set of attack chain evolution after completing multi-source feature fusion; Attack Intent Probability Modeling Module: Used to build an attack intent probability prediction model, outputting the stage transition probability of attack behavior and the attack intent confidence level; Deep reinforcement learning prediction module: used to build a framework for attack phase prediction and intent recognition, and output prediction results for the subsequent evolution stages of attack behavior and core attack intent recognition results; Active defense decision output module: used to generate proactive defense advance decision instructions and threat warning information based on the prediction and identification results.

Images