A network security log intelligent analysis method and device

Abstract

The application discloses a network security log intelligent analysis method and device, and relates to the technical field of network security, which comprises the following steps: receiving original log data of each security device in real time based on a pre-configured transmission protocol, performing a preprocessing operation on the original log data to obtain standard log data; performing a matching operation on the standard log data based on an operator and a multi-field combination logic to determine candidate log data from the standard log data; determining the malicious probability score of each candidate log based on a pre-trained discriminative model, and determining candidate log data greater than or equal to a preset score threshold as target log data; and inputting the full-field information and context information of the target log data into a pre-trained generative model to output the analysis result of the target log data. The application can reduce the proportion of redundant alarms and false alarms and realize accurate analysis of security logs based on the dual-engine processing of the discriminative model and the generative model.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and more particularly to the field of security log analysis technology, specifically to a method and apparatus for intelligent analysis of network security logs. Background Technology

[0002] In current cybersecurity operations, various security devices generate massive amounts of log data during continuous operation. This log data is typically characterized by its enormous volume, complex data types, significant redundancy in alert information, and high reliance on manual analysis experience. In practice, log formats from different device manufacturers vary considerably, and data standards are inconsistent, making direct correlation analysis between logs difficult. Furthermore, a large number of duplicate or low-value alerts mask genuine security threats, further increasing the analytical burden on security operations personnel and reducing overall response efficiency.

[0003] To address the aforementioned issues, existing technologies primarily employ the following approaches: First, rule-based log analysis methods, which filter logs and identify alerts through predefined keyword matching and threshold judgment. However, this method relies on human experience to construct rules, resulting in poor flexibility and difficulty in handling complex attack scenarios. Second, traditional machine learning-based methods, which use classification or clustering models to detect anomalies in logs. While this improves automation to some extent, it typically relies on feature engineering and lacks a deep understanding of log semantics. Third, centralized analysis is conducted using SIEM or SOC platforms, where alerts are manually assessed and event correlation analysis is performed. Although this approach possesses certain comprehensive analytical capabilities, it remains largely manual, resulting in low efficiency and difficulty in achieving real-time and accurate security threat identification in massive log environments.

[0004] Currently, no effective solution has been proposed to address the aforementioned problems. Therefore, there is an urgent need to develop a new intelligent analysis method for network security logs. Summary of the Invention

[0005] To address at least one of the technical problems raised in the background section, this application provides a method and apparatus for intelligent analysis of network security logs, which effectively improves the results of intelligent analysis of network security logs.

[0006] In a first aspect, embodiments of the present invention provide a method for receiving raw log data from various security devices in real time based on a pre-configured transmission protocol, and performing preprocessing operations on the raw log data to obtain standard log data; Based on the logic of combining operators and multiple fields, a matching operation is performed on the standard log data to determine candidate log data. Based on a pre-trained discriminative model, the malicious probability score of each candidate log is determined, and candidate log data that is greater than or equal to the preset score threshold is determined as target log data. Using the full field information and context information of the target log data as input, and based on a pre-trained generative model, the analysis results of the target log data are output.

[0007] In some optional embodiments of this example, the preprocessing operation on the raw log data to obtain standard log data includes: Remove null values ​​and duplicate data from the original log data to obtain the first log data; Based on a pre-configured rule repository, the first log data is parsed to extract its structured fields, thus obtaining the second log data. Redundant fields are removed from the second log data, and the remaining fields are normalized across devices to obtain the standard log data.

[0008] In some optional embodiments of this example, the step of configuring the rule repository includes: A parsing rule metadata table is established with equipment manufacturer, product model and log version number as three-level indexes, and the parsing rule metadata table is stored in the rule warehouse in a first preset format. Each parsing rule consists of a matching anchor, a parsing mode identifier, a field mapping template and a log version number. The rule repository is monitored by a rule hot-loading daemon. When a change in the log version number in the rule repository is detected, the newly added log data is subjected to syntax verification and sample replay testing. If the verification passes, the parsing rule metadata table is updated based on the newly added log data.

[0009] In some optional embodiments of this example, the step of removing redundant fields from the second log data and normalizing the remaining fields includes: Configure a field whitelist based on the input characteristics of the generative model and the discriminative model; Based on the field whitelist, redundant fields in the second log data are removed, wherein the remaining fields include timestamp, source IP address, destination IP address, and event severity level; For the remaining fields, those that are semantically equivalent but have inconsistent field names and value formats are uniformly mapped to standard field names. Different values ​​corresponding to the severity level of the event are uniformly mapped to a risk quantification score range; The timestamps are uniformly converted into a second preset format, and the source IP address and the target IP address are subjected to demasking and address restoration operations.

[0010] In some optional embodiments of this example, the step of performing a matching operation on the standard log data based on operator and multi-field combination logic to determine candidate log data from the standard log data includes: Based on preset filtering rules, a rule syntax tree is constructed, wherein the nodes of the rule syntax tree are triples consisting of fields, operators, and values. The standard log data is traversed from top to bottom through the rule syntax tree to determine the current hit status of the standard log data in the rule syntax tree; Based on the current hit status and matching priority, candidate log data is determined from the standard log data.

[0011] In some optional embodiments of this example, determining the current hit status of the standard log data in the rule syntax tree includes: The matching method is determined based on the type of the operator in the currently traversed node; Determine the matching termination condition based on the logical type of the currently traversed node; Based on the matching method and the matching termination condition, the current hit status of the standard log data in the rule syntax tree is determined.

[0012] In some optional embodiments of this example, the preset filtering rules include whitelist allow rules, abnormal list interception rules, and rules to be analyzed by intelligent channels. The matching priority is that whitelist allow rules take precedence over abnormal list interception rules, and abnormal list interception rules are preceded by rules to be analyzed by intelligent channels. The step of determining candidate log data from the standard log data based on the current hit situation and matching priority includes: Step A1: Determine whether the standard log data matches the whitelist allow rule. If it matches, discard the standard log data; if it does not match, proceed to step A2. Step A2: Determine whether the standard log data matches the anomaly list interception rule. If it matches, trigger an alarm operation; if it does not match, proceed to step A3. Step A3: Determine whether the standard log data matches the intelligent channel analysis rule. If it matches, determine that the standard log data is the log data to be intelligently analyzed; if it does not match, determine that the standard log data is unmatched log data. The candidate log data consists of the log data to be intelligently analyzed and the log data without matching.

[0013] In some optional embodiments of this example, determining the malicious probability score of each candidate log based on a pre-trained discriminative model includes: The candidate log data is input into the pre-trained discriminative model to generate a high-dimensional semantic vector; Density clustering algorithm is used to perform density clustering on the vector set composed of the high-dimensional semantic vectors; Outliers and anomalous clusters are extracted from the clustering results. The candidate log data corresponding to the extracted outliers and anomalous clusters are used as samples to be labeled, and a fine-tuning training set is constructed by combining the results of manual labeling. The discriminative model is adjusted using the fine-tuning training set, and the adjusted discriminative model is deployed to the streaming computing engine to perform streaming inference on the candidate log data that comes in in real time, and output the malicious probability score corresponding to each candidate log data.

[0014] Secondly, embodiments of the present invention also provide a network security log intelligent analysis device, comprising: The data preprocessing module is configured to receive raw log data from each security device in real time based on a pre-configured transmission protocol, and perform preprocessing operations on the raw log data to obtain standard log data. The candidate log data determination module is configured to perform a matching operation on the standard log data based on the combination logic of operators and multiple fields, and determine candidate log data from the standard log data; The target log data determination module is configured to determine the malicious probability score of each candidate log based on a pre-trained discriminative model, and determine the candidate log data that is greater than or equal to the preset score threshold as the target log data. The intelligent analysis module is configured to take the full field information and context information of the target log data as input, and output the analysis results of the target log data based on a pre-trained generative model.

[0015] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the network security log intelligent analysis method described in the first aspect.

[0016] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the network security log intelligent analysis method described in the first aspect.

[0017] Fifthly, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the network security log intelligent analysis method described in the first aspect.

[0018] This invention provides a method and apparatus for intelligent analysis of network security logs. By preprocessing raw log data from various security devices, it can solve the problem of inconsistent log formats from different security devices, which makes unified analysis difficult. Based on a dual-engine processing of discriminative and generative models, on the one hand, the discriminative model reduces the proportion of redundant alarms and false alarms; on the other hand, using the full-field information and context information of the target log data as input, the generative model realizes semantic understanding and attack behavior identification of security logs, thereby solving the problem that traditional methods cannot perform deep semantic analysis. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 One of the flowcharts of the intelligent network security log analysis method provided in the embodiments of this application; Figure 2 The second flowchart illustrates the intelligent analysis method for network security logs provided in this application embodiment. Figure 3 The third flowchart illustrates the intelligent analysis method for network security logs provided in this application embodiment; Figure 4 The fourth flowchart illustrates the intelligent analysis method for network security logs provided in this application embodiment. Figure 5 The fifth flowchart illustrating the intelligent network security log analysis method provided in this application embodiment; Figure 6 A flowchart illustrating the intelligent analysis method for network security logs provided in this application is shown in Figure 6. Figure 7 The seventh flowchart illustrating the intelligent analysis method for network security logs provided in this application embodiment; Figure 8 This is a schematic diagram of the intelligent network security log analysis device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the intelligent network security log analysis system provided in the embodiments of this application; Figure 10 This is a block diagram of an electronic device used to implement embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0021] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0022] The acquisition, transmission, storage, use, and processing of data in this application comply with relevant national laws and regulations. It should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0023] After in-depth analysis of existing technologies, the following shortcomings were found in current security log analysis: (1) The problem of alarm noise is quite prominent. Since most security devices adopt rule-based or threshold-based detection methods, a large number of duplicate alarms and low-value alarms are easily generated in actual operation. There are many false alarms, which cause the key information with real security risks to be submerged in a large number of invalid alarms, increasing the difficulty of analysis. (2) Existing methods generally lack the ability to understand the semantics of log content. In particular, rule matching and traditional machine learning methods usually only process fields or features, and it is difficult to make an overall judgment on log behavior in combination with contextual information. (3) The current security log analysis process is highly dependent on human experience. In alarm screening, event judgment and correlation analysis, security personnel still need to analyze and process each item one by one, which is not only inefficient, but also difficult to ensure the consistency of analysis results. (4) Since different security devices have differences in log format, field structure and other aspects, it is difficult to process and effectively correlate multi-source logs in a unified manner, which limits the ability to identify complex attack behaviors. (5) Existing systems usually lack effective optimization mechanisms, and the results generated during the analysis process are difficult to be used in reverse to adjust the model or rules, making it difficult for the overall system capability to continuously improve with the accumulation of data.

[0024] To address at least one of the aforementioned technical problems, this application proposes an intelligent analysis method for network security logs, such as... Figure 1 As shown, it includes: Step S1: Receive raw log data from each security device in real time based on a pre-configured transmission protocol, and perform preprocessing operations on the raw log data to obtain standard log data; Step S2: Based on the combination logic of operators and multiple fields, perform a matching operation on the standard log data to determine candidate log data from the standard log data; Step S3: Based on the pre-trained discriminative model, determine the malicious probability score of each candidate log, and determine the candidate log data that is greater than or equal to the preset score threshold as the target log data; Step S4: Using the full field information and context information of the target log data as input, and based on the pre-trained generative model, output the analysis results of the target log data.

[0025] This application provides an intelligent analysis method for network security logs. By preprocessing the raw log data from various security devices, it can solve the problem of inconsistent log formats from different security devices, which makes unified analysis difficult. Based on a dual-engine processing of discriminative and generative models, on the one hand, the discriminative model reduces the proportion of redundant alarms and false alarms; on the other hand, using the full field information and context information of the target log data as input, the generative model realizes semantic understanding and attack behavior identification of security logs, thereby solving the problem that traditional methods cannot perform deep semantic analysis.

[0026] The following is a detailed description of steps S1-S4: Step S1: Receive raw log data from each security device in real time based on a pre-configured transmission protocol, and perform preprocessing operations on the raw log data to obtain standard log data.

[0027] In this embodiment, the pre-configured transmission protocol is UDP / TCP, and the security devices are firewalls, intrusion detection systems, intrusion prevention systems, host security systems, and database auditing systems, etc. This application does not limit these.

[0028] It should be understood that raw log data is often inconsistent in format and complex in content, thus requiring preprocessing. Preprocessing includes data cleaning operations such as removing null values ​​and duplicate data, as well as parsing the log content to extract key information such as timestamps, source IP addresses, destination IP addresses, port numbers, and protocol types. Simultaneously, format standardization processing is performed on logs from different sources, and time alignment can also be performed to ensure consistency in subsequent analysis.

[0029] In some optional embodiments of this example, such as Figure 2As shown, the raw log data in step S1 undergoes preprocessing to obtain standard log data, including: Step S11: Remove null values ​​and duplicate data from the original log data to obtain the first log data.

[0030] In this step, removing null values ​​and duplicate data from the original log data is a routine data cleaning operation, which will not be described in detail here.

[0031] Step S12: Based on the pre-configured rule warehouse, perform field parsing on the first log data, extract the structured fields of the first log data, and obtain the second log data.

[0032] In this step, structured fields may include, for example, event type, source / destination IP, user, command, file path, etc. This application does not impose any restrictions on these fields.

[0033] In this step, the rule repository is a dynamically updated and hot-updated rule library, such as... Figure 3 As shown, the steps to configure this rule repository include: Step S121: Establish a parsing rule metadata table with device manufacturer, product model and log version number as three-level indexes, and store the parsing rule metadata table in the rule repository in a first preset format (e.g. YAML / JSON format). Each parsing rule consists of a matching anchor, parsing mode identifier, field mapping template and log version number.

[0034] Step S122: Monitor the rule repository based on the rule hot-loading daemon process. When a change in the log version number in the rule repository is detected, perform syntax verification and sample replay test on the newly added log data.

[0035] Step S123: If the verification passes, update the parsing rule metadata table based on the newly added log data.

[0036] Specifically, the system maintains a parsing rule metadata table with a three-level index of "device manufacturer - product model - log version number". Each parsing rule consists of four core parts: first, a matching anchor point, used to locate the log to be parsed, specifically including the facility / priority field in the Syslog message header, the first token of the log message, or a fixed prefix string; second, a parsing pattern identifier, specifically including any one of four methods: regular expression, JSON path extraction, key-value pair splitting, and fixed position truncation; third, a field mapping template, used to standardize the original log fields into a unified data model; and fourth, a version number, used to identify the rule's iteration status.

[0037] Furthermore, the parsing rules are stored in a separate rule repository in YAML or JSON format. The system has a rule hot-loading daemon process, completely decoupled from the main rule engine process, used to continuously monitor changes in the file status of the rule repository. When the system connects to a new device or the device manufacturer upgrades its log format, security operations personnel do not need to restart any service processes; they only need to submit the newly added or modified rule entries to the rule repository on the management end. Upon detecting an incrementing version number in the rule repository, the hot-loading daemon process immediately triggers the rule verification and loading process.

[0038] Preferably, the verification process specifically includes two stages: syntax verification and sample replay testing. In the syntax verification stage, the daemon process performs targeted checks based on the parsing pattern specified in the current rule entry—regular expression pattern checks the compilation validity of the regular expression; JSON pattern checks the reachability of the JSON path expression; key-value pair pattern checks the consistency of the delimiter definition; and fixed-position pattern checks the non-negativity and reasonableness of the truncation offset length and field length. In the sample replay testing stage, the daemon process applies the candidate rule to the latest N original logs cached in the shadow queue for simulated parsing and calculates the success rate of extracting key fields; only when the success rate reaches a preset threshold is the current rule deemed to have passed verification.

[0039] Furthermore, for candidate rules that pass full verification, the daemon process uses a double-buffered lock-free switching mechanism to atomically update the rule routing table in memory with the new rule, thus completing the replacement of the old and new rules in an atomic manner.

[0040] During this process, the raw log collection and parsing pipeline remains continuously running without interruption or pause, thus ensuring the integrity and continuity of the log data stream. Through this mechanism, the present invention achieves online, second-level hot updates of parsing rules, effectively avoiding the risk of log loss or processing interruption caused by restarting the collection process due to rule changes in traditional solutions.

[0041] Step S13: Remove redundant fields from the second log data and perform cross-device normalization on the remaining fields to obtain the standard log data.

[0042] It should be noted that this application does not simply transmit all fields, but adopts a two-layer strategy of "whitelist retention + semantic normalization mapping" to process the second log data and obtain the final standard log data.

[0043] like Figure 4 As shown, step S13 further includes: Step S131: Configure a field whitelist based on the input characteristics of the generative model and the discriminative model.

[0044] Step S132: Based on the field whitelist, remove redundant fields from the second log data, wherein the remaining fields include timestamp, source IP address, destination IP address, and event severity level; Specifically, in this embodiment, a field whitelist is pre-configured based on the input feature list required by the downstream discriminative and generative models. The whitelist includes at least the following core fields: timestamp, device identifier, source IP address, destination IP address, source port, destination port, transport protocol, event type, event severity level, operating user, source hostname, destination hostname, application process name, command / URL / file path, and original message body. Vendor-specific fields, debugging fields used only for troubleshooting, and non-informational fields with constant values ​​are directly removed during the structuring process.

[0045] Step S133: For the remaining fields that are semantically equivalent but have inconsistent field names and value formats, map them uniformly to standard field names.

[0046] In this step, for fields whose field names and value formats are semantically equivalent but whose representations are inconsistent, cross-device normalization mapping is performed. For example, field names that represent the same semantics in logs from different vendors, such as "src", "source", "SIP", and "from-ip", are uniformly mapped to the standard field name src_ip.

[0047] Step S134: Map the different values ​​corresponding to the severity level of the event to a unified risk quantification score range.

[0048] For example, the values ​​of "high", "severe", "3", "critical" and other values ​​representing different levels of severity in different log sources are uniformly mapped to a preset five-level risk quantification score (e.g., mapped to the set {1, 2, 3, 4, 5}).

[0049] Step S135: Convert the timestamp into a second preset format, and perform masking and address restoration operations on the source IP address and the target IP address.

[0050] In this step, all timestamps are uniformly converted to the standard UTC millisecond epoch format; the source IP address and the destination IP address are demasked and the IPv4-mapped-IPv6 format is restored to the original IPv4 address.

[0051] After the above two-layer strategy processing, logs from heterogeneous devices from different manufacturers are fully aligned in four dimensions: field set, field naming, value encoding, and time base, forming a unified structured log object that can be directly consumed by downstream AI models.

[0052] Step S2: Based on the combination logic of operators and multiple fields, perform a matching operation on the standard log data to determine candidate log data from the standard log data.

[0053] In this embodiment, standard log data is initially screened based on preset filtering rules to filter out invalid and low-value logs, reduce alarm noise, alleviate the computational burden on the subsequent AI engine analysis, and improve overall analysis efficiency. For example, the normalized logs are input into the rule engine, which performs high-speed matching based on multi-field combination logic (AND / OR) and operators (=, !=, contains, like, etc.) to filter known harmless operations and high-frequency whitelist traffic, and outputs a candidate log set for in-depth analysis.

[0054] In some optional embodiments of this example, such as Figure 5 As shown, step S2 further includes: Step S21: Construct a rule syntax tree based on preset filtering rules, wherein the nodes of the rule syntax tree are triples consisting of fields, operators and values.

[0055] In this embodiment, all filtering rules described in a domain-specific language (DSL) from the rule base are first loaded into memory. After lexical analysis and syntax parsing, each rule is parsed into an atomic conditional abstract syntax tree (AST) consisting of "field-operator-value" triples, and then grouped and managed according to static and dynamic rules. Static rules are compiled all at once during system startup, generating a hash index structure based on a field dictionary; dynamic rules are incrementally compiled upon receiving update instructions from the human-machine collaborative feedback module, and the memory rules are updated using atomic replacement to ensure uninterrupted matching service during rule updates.

[0056] For each normalized log entering the engine, the system pre-builds its field dictionary and inverted index of field values. For high-frequency filtering fields, including event severity level, device type, protocol type, source IP network segment, and destination IP network segment, a bitmap index is maintained in memory, so that subsequent rule matching can complete the field value hit judgment in O(1) or O(k) time complexity.

[0057] Step S22: Traverse the rule syntax tree from top to bottom using the standard log data to determine the current hit status of the standard log data in the rule syntax tree.

[0058] For each candidate log entry, the system traverses its corresponding rule AST from top to bottom. In some optional methods of this embodiment, such as... Figure 6 As shown, step S22, determining the current hit status of the standard log data in the rule syntax tree, includes: Step S221: Determine the matching method based on the type of the operator of the currently traversed node.

[0059] In this embodiment, atomic conditions are dispatched to the corresponding matchers for matching based on the operator type. The matchers include at least: exact matching (=, !=), substring matching (contains), wildcard matching (like), regular expression matching, IP segment inclusion matching (such as in CIDR), numerical range matching, and set membership matching.

[0060] Step S222: Determine the matching termination condition based on the logical type of the currently traversed node.

[0061] In this embodiment, for logical nodes composed of AND, OR, and NOT, the system adopts a short-circuit evaluation strategy: if any sub-condition under an AND node is not met, the branch is terminated immediately; if any sub-condition under an OR node is met, the branch is terminated immediately, thereby reducing unnecessary matching calculation overhead.

[0062] Step S223: Based on the matching method and the matching termination condition, determine the current hit status of the standard log data in the rule syntax tree.

[0063] Based on the matching method and matching termination condition determined above, the current hit status of standard log data in the rule syntax tree is thus determined.

[0064] Step S23: Based on the current hit status and matching priority, determine candidate log data from the standard log data.

[0065] It should be noted that the preset filtering rules include whitelist allow rules, abnormal list interception rules, and rules to be analyzed by intelligent channels. The matching priority is that whitelist allow rules take precedence over abnormal list interception rules, and abnormal list interception rules are preceded by rules to be analyzed by intelligent channels.

[0066] The rules in the rule base are divided into three categories based on the action taken: whitelist allow rules (whitelist, corresponding to known harmless operations), abnormal list interception rules (abnormal list, corresponding to confirmed malicious behavior, requiring direct alerts), and rules awaiting intelligent channel analysis (requiring in-depth analysis by the AI ​​engine). Matching decisions follow the order: whitelist allow rules take precedence over abnormal list interception rules over rules awaiting intelligent channel analysis. When a log matches an allow rule, such as a confirmed high-frequency whitelist scenario like a regular login by an internal operations IP within the maintenance window, business heartbeat traffic, or backup task traffic, the system discards the log and removes it from further processing. When a log matches an interception rule, the system directly outputs a high-priority alert. When a log does not match any rules or only matches rules awaiting intelligent channel analysis, it is added to the candidate log set for subsequent AI engine processing.

[0067] In some optional embodiments of this example, such as Figure 7 As shown, step S23 further includes: Step A1: Determine whether the standard log data matches the whitelist allow rule. If it matches, discard the standard log data; if it does not match, proceed to step A2. Step A2: Determine whether the standard log data matches the anomaly list interception rule. If it matches, trigger an alarm operation; if it does not match, proceed to step A3. Step A3: Determine whether the standard log data matches the intelligent channel analysis rule. If it matches, determine that the standard log data is the log data to be intelligently analyzed; if it does not match, determine that the standard log data is unmatched log data. The candidate log data consists of the log data to be intelligently analyzed and the log data without matching.

[0068] Furthermore, it should be noted that for candidate logs output to the AI ​​channel, the system performs sliding window counting within a fixed time window based on a four-tuple consisting of (rule ID, source IP, destination IP, and event fingerprint). The default time window is 60 seconds, but dynamic adjustment is supported. When the frequency of logs with the same four-tuple appearing within the time window exceeds a preset threshold, the system aggregates multiple identical logs within that window into a single aggregated event, attaching counting metadata to avoid duplicate pushes of the same alarms and prevents wasted computing power for the downstream AI engine.

[0069] The rule engine calculates operational metrics for each rule at preset intervals (every 5 minutes by default, configurable). These metrics include at least rule hit rate, average matching latency, and false positive rate. The false positive rate is calculated based on the proportion of false alarm samples returned from the human-machine collaboration feedback module. These metrics are periodically reported to the human-machine collaboration feedback module for maintenance personnel to reference in adding, deleting, modifying, and optimizing the rule base.

[0070] Step S3: Based on the pre-trained discriminative model, determine the malicious probability score of each candidate log, and identify the candidate log data that is greater than or equal to the preset score threshold as the target log data.

[0071] In this embodiment, the candidate log data is input into the pre-trained discriminative model to generate high-dimensional semantic vectors; a density clustering algorithm is used to perform density clustering on the vector set composed of the high-dimensional semantic vectors; outliers and anomalous clusters are extracted from the clustering results, and the candidate log data corresponding to the extracted outliers and anomalous clusters are used as samples to be labeled, and a fine-tuning training set is constructed by combining the manual labeling results; the parameters of the discriminative model are adjusted using the fine-tuning training set, and the adjusted discriminative model is deployed to a streaming computing engine to perform streaming inference on the candidate log data flowing in in real time, and output the malicious probability score corresponding to each candidate log data.

[0072] Among them, the pre-trained discriminative models are, for example, SecBert, RoBERTa, DeBERTa, or other pre-trained language models fine-tuned for the security domain; the density clustering algorithms are HDBSCAN, DBSCAN, OPTICS, or anomaly detection algorithms based on Isolation Forest.

[0073] In this embodiment, the long-distance semantic dependencies of the attack payload are captured by SecBert's bidirectional self-attention mechanism. Combined with HDBSCAN's sensitivity to sparse regions, the problem of traditional methods missing deformation attacks and unknown threats is effectively solved, achieving an initial feature compression rate of over 75% and significantly reducing the computational load.

[0074] Step S4: Using the full field information and context information of the target log data as input, and based on the pre-trained generative model, output the analysis results of the target log data.

[0075] In this embodiment, a high-score alarm (target log data) and its full-field information are received, triggering the intelligent agent workflow: automatically calling related log queries, asset information queries, CTI / IOC reputation interfaces, and PCAP packet parsing tools to complete the session context; concatenating the completed multidimensional data with the expert experience base (judgment logic, handling strategies, and historical cases) into a structured Prompt, which is then input into a generative large language model; the model performs attack intent reasoning, threat level assessment, and handling suggestion generation, outputting a natural language judgment report containing threat characterization, scope of impact, and confidence score.

[0076] In this embodiment, the full-field information includes the following: (1) All structured fields retained after preprocessing in step S1 include at least: timestamp, device identifier, source IP address, destination IP address, source port, destination port, transport protocol, event type, event level, operating user, source hostname, destination hostname, application process name, URL / command / file path, and original message body.

[0077] (2) The discriminative engine outputs feature fields, including the semantic vector, cluster number, outlier score, malicious probability score, cluster density and cluster center distance obtained by HDBSCAN density clustering, and similarity score between the current log and the historical known attack templates.

[0078] (3) Context aggregation field, within the sliding time window (5 minutes before and after by default, and supports dynamic configuration), the number of other log entries that share the same source IP, destination IP or user identifier as the alarm (target log data); the time interval between the most recent alarm of the same type and the current alarm; and the historical alarm frequency and risk score accumulation value of the source IP or destination IP in the past 24 hours.

[0079] (4) Analyze the tracking fields, including at least the globally unique event ID generated in the system for this alarm, the rule ID hit upstream, the cluster ID triggered, and the data acquisition pipeline identifier, for subsequent feedback and audit tracing.

[0080] The above four types of field sets are uniformly encapsulated into JSON structured objects, which serve as the complete input load for the intelligent agent, thereby ensuring that the generative model can obtain sufficient observational evidence during the inference phase.

[0081] The aforementioned "correlated logs" refer to a collection of logs that, in terms of time, space, or behavioral semantics, meet at least one of the following correlation conditions with the current high-score alert, and thus may belong to the same attack chain or different stages of the same attack session: (1) Five-tuple association: Within a preset time window, logs that share the source IP address, destination IP address, source port, destination port, and any key field or combination of the transport protocol with the alarm log. For example, scan logs initiated by the same source IP address to different ports of the same destination IP address within a short period of time.

[0082] (2) Session identifier association: Logs that share the same TCP session ID, HTTP session ID, login session ID, transaction ID or process PID and other session identifier fields can restore a complete communication interaction process.

[0083] (3) Asset topology association, logs from the same asset network segment, the same business domain, the same host or the same tenant, and in time sequence constitute a progressive relationship of attack stages such as reconnaissance, delivery, utilization, installation and action.

[0084] (4) IOC / feature association: logs that hit the same IOC indicator (such as the same malicious domain name, the same malicious file hash, the same C2 IP address, the same attack tool signature string) or the same TTP (tactical, technical and process) feature tag.

[0085] (5) Behavioral chain association: Logs that form directed dependencies on user behavior sequences or host call chains. For example, a kill chain formed in chronological order among Web access logs, WAF alarms, host login logs, process creation logs, and file operation logs.

[0086] Logs that meet any of the above association conditions are considered associated logs and included in the context completion scope.

[0087] Next, the steps for completing the session context will be introduced: Step C1: Orchestration of context query tasks.

[0088] Specifically, after receiving a high-score alarm and its full-field information, the analytical agent automatically generates a multi-dimensional context query task list based on the alarm field content and the aforementioned association judgment rules. Each query task in the task list explicitly specifies its corresponding data source, query time range, query key, and maximum number of returned records. The query time range defaults to looking back 24 hours and extending forward 30 minutes, and can be dynamically adjusted according to the alarm level. Query keys include, but are not limited to, source IP address, destination IP address, user identifier, domain name, file hash, and session ID.

[0089] Step C2: Concurrent fetching of heterogeneous data from multiple sources.

[0090] The analysis agent concurrently executes data query operations across multiple dimensions via tool calling interfaces: First, call the log retrieval interface to pull relevant logs from the unified log storage (including Elasticsearch or ClickHouse) based on the association conditions; Second, call the asset information query interface to connect to the CMDB or asset ledger system, and obtain the business system to which the asset belongs, the person in charge of the asset, the asset importance level, the external open ports and services, and historical vulnerability information based on the IP address or hostname; Third, call the CTI / IOC reputation query interface to connect with the internal threat intelligence platform or third-party threat intelligence source to perform reputation queries on the source IP address, destination IP address, domain name, URL and file hash, and return whether the entity has hit the known malicious IOC, the threat group to which it belongs and the tags of related attack activities; Fourth, call the PCAP packet parsing tool to pull the original network packets on the traffic storage platform according to the five-tuple and time range, and parse and extract TLS handshake information (including certificate SNI and JA3 fingerprint), DNS query and response records, HTTP request headers and request bodies, and deep network evidence such as suspicious shellcode or command fragments that may exist in the payload. When necessary, the analytical agent can also call the SOAR script interface to link with the EDR system to obtain terminal evidence such as the host side's process tree, file landing records, and registry changes.

[0091] Step C3: Evidence cleaning and field alignment.

[0092] Specifically, for all results retrieved from the aforementioned multi-source heterogeneous data, deduplication, field name normalization mapping, and timestamp alignment operations are uniformly performed. Simultaneously, noisy evidence unrelated to the current alarm is removed, and missing fields are placed with null values ​​and the reason for the missing value is recorded, ultimately forming a structurally consistent set of evidence objects.

[0093] Step C4: Concatenation of session-level evidence and annotation of the attack phase.

[0094] Specifically, the original alerts, associated logs, asset intelligence, IOC reputation query results, network packet parsing results, and endpoint evidence are concatenated in chronological order. Based on this, and combined with pre-defined attack phase determination rules from an expert experience base, each piece of evidence is labeled with its attack phase tag and credibility weight. These attack phase determination rules include at least: mapping port scanning to the reconnaissance phase, vulnerability exploitation login to the delivery phase, command execution to the exploitation phase, privilege escalation to the installation phase, lateral movement to the action phase, and data transmission to the goal achievement phase, thereby constructing a complete attack session context object.

[0095] Step C5: Context integrity verification.

[0096] Specifically, the complete attack session context is verified to meet the minimum evidence set requirements for generative model inference. The minimum evidence set requirements include at least: the alarm itself, source and destination asset information, and at least N related logs within the same time window. If key evidence is missing, such as inability to retrieve asset information or expired PCAP, the missing items and their specific impact on the credibility of the judgment must be explicitly marked in the context for reference by the generative model during inference, preventing the model from making a definitive misjudgment due to insufficient evidence.

[0097] The attack session context, completed through the above steps, is combined with the judgment logic, handling strategies, and historical similar cases stored in the expert experience base to form a structured prompt. This prompt is then input into a generative large language model, which performs subsequent attack intent reasoning, threat level assessment, and handling suggestions.

[0098] In this embodiment, the Chengshi AI engine does not generate text in isolation. Instead, it obtains multi-source heterogeneous data through active tool calling and injects expert experience as a constraint into the reasoning process. This fundamentally suppresses the "illusion" of large models and enables the output report to have verifiable attack chain logic and operable handling guidelines.

[0099] Furthermore, this application displays the assessment report and related evidence chain through a front-end interface, allowing security analysts to confirm, correct, or reject the report. The system converts the manual correction results into structured feedback data, which includes at least: positive and negative sample labels (i.e., log samples labeled as malicious or benign), rule correction suggestions (i.e., instructions for adding, deleting, or modifying existing rules), and confidence scores.

[0100] Feedback data is automatically entered into the incremental training queue, and the following iterative update process is triggered periodically according to a preset cycle: Incremental fine-tuning training of the SecBert discriminative model is triggered, and the model parameters are updated using newly labeled samples. Simultaneously, the prompt word strategy update of the generative model is triggered, and the decision logic and output format constraints in the inference prompt template are adjusted based on feedback.

[0101] Through the above process, the system forms a continuous closed loop of "reasoning → human feedback → model update", realizing the continuous optimization of the discriminative model and the generative model in the business operation process.

[0102] In this embodiment, the closed-loop feedback mechanism transforms implicit human experience into explicit training data, enabling the model to continuously evolve with the progress of offensive and defensive confrontations, avoiding the technical bottleneck of "deployment leading to obsolescence", and significantly extending the effective life cycle of the system.

[0103] This application constructs a dual-engine processing architecture that combines a discriminative model and a generative model. The discriminative model classifies, filters, and prioritizes input logs, while the generative large language model performs semantic analysis and security event reasoning on the filtered logs, achieving a layered processing mechanism. Secondly, an alarm denoising mechanism based on the discriminative model is designed. This mechanism trains and models log features, aggregates duplicate alarms, identifies abnormal patterns, and outputs alarm confidence scores, achieving alarm filtering and ranking. Thirdly, a log semantic analysis mechanism based on the generative large language model is constructed. By inputting log content into the generative model and combining it with a preset prompt template, a structured analysis including attack type, attack path, and handling suggestions is generated. The analysis results are presented in two parts. First, a multi-source log association analysis mechanism is proposed. By constructing a relationship graph between logs, multiple logs are associated and inferred based on time series, IP association, and behavioral characteristics to reconstruct the attack chain. Second, a human-machine collaborative closed-loop optimization mechanism is designed. By introducing human feedback data, the model output results are labeled and corrected, and the feedback results are used for model retraining or rule updates, thus forming a continuous optimization mechanism. The front end displays the judgment report and evidence chain, which security analysts confirm, correct, or reject. The system converts the human correction results into structured feedback data (including positive and negative sample labels and rule correction suggestions). The feedback data automatically enters the incremental training queue, periodically triggering the fine-tuning and updating of the prompt word strategy of the SecBert discriminative model and the generative model, completing the closed-loop iteration.

[0104] This invention also provides a network security log intelligent analysis device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of a network security log intelligent analysis method, the implementation of this device can refer to the implementation applicable to network security log intelligent analysis; repeated details will not be elaborated further.

[0105] like Figure 8 As shown, the intelligent network security log analysis device includes: The data preprocessing module 10 is configured to receive raw log data from each security device in real time based on a pre-configured transmission protocol, and perform preprocessing operations on the raw log data to obtain standard log data. The candidate log data determination module 20 is configured to perform a matching operation on the standard log data based on the combination logic of operators and multiple fields, and determine candidate log data from the standard log data; The target log data determination module 30 is configured to determine the malicious probability score of each candidate log based on a pre-trained discriminative model, and determine the candidate log data that is greater than or equal to the preset score threshold as the target log data. The intelligent analysis module 40 is configured to take the full field information and context information of the target log data as input, and output the analysis results of the target log data based on a pre-trained generative model.

[0106] In some optional embodiments of this example, the preprocessing operation on the raw log data to obtain standard log data includes: Remove null values ​​and duplicate data from the original log data to obtain the first log data; Based on a pre-configured rule repository, the first log data is parsed to extract its structured fields, thus obtaining the second log data. Redundant fields are removed from the second log data, and the remaining fields are normalized across devices to obtain the standard log data.

[0107] In some optional embodiments of this example, the step of configuring the rule repository includes: A parsing rule metadata table is established with equipment manufacturer, product model and log version number as three-level indexes, and the parsing rule metadata table is stored in the rule warehouse in a first preset format. Each parsing rule consists of a matching anchor, a parsing mode identifier, a field mapping template and a log version number. The rule repository is monitored by a rule hot-loading daemon. When a change in the log version number in the rule repository is detected, the newly added log data is subjected to syntax verification and sample replay testing. If the verification passes, the parsing rule metadata table is updated based on the newly added log data.

[0108] In some optional embodiments of this example, the step of removing redundant fields from the second log data and normalizing the remaining fields includes: Configure a field whitelist based on the input characteristics of the generative model and the discriminative model; Based on the field whitelist, redundant fields in the second log data are removed, wherein the remaining fields include timestamp, source IP address, destination IP address, and event severity level; For the remaining fields, those that are semantically equivalent but have inconsistent field names and value formats are uniformly mapped to standard field names. Different values ​​corresponding to the severity level of the event are uniformly mapped to a risk quantification score range; The timestamps are uniformly converted into a second preset format, and the source IP address and the target IP address are subjected to demasking and address restoration operations.

[0109] In some optional embodiments of this example, the step of performing a matching operation on the standard log data based on operator and multi-field combination logic to determine candidate log data from the standard log data includes: Based on preset filtering rules, a rule syntax tree is constructed, wherein the nodes of the rule syntax tree are triples consisting of fields, operators, and values. The standard log data is traversed from top to bottom through the rule syntax tree to determine the current hit status of the standard log data in the rule syntax tree; Based on the current hit status and matching priority, candidate log data is determined from the standard log data.

[0110] In some optional embodiments of this example, determining the current hit status of the standard log data in the rule syntax tree includes: The matching method is determined based on the type of the operator in the currently traversed node; Determine the matching termination condition based on the logical type of the currently traversed node; Based on the matching method and the matching termination condition, the current hit status of the standard log data in the rule syntax tree is determined.

[0111] In some optional embodiments of this example, the preset filtering rules include whitelist allow rules, abnormal list interception rules, and rules to be analyzed by intelligent channels. The matching priority is that whitelist allow rules take precedence over abnormal list interception rules, and abnormal list interception rules are preceded by rules to be analyzed by intelligent channels. The step of determining candidate log data from the standard log data based on the current hit situation and matching priority includes: Step A1: Determine whether the standard log data matches the whitelist allow rule. If it matches, discard the standard log data; if it does not match, proceed to step A2. Step A2: Determine whether the standard log data matches the anomaly list interception rule. If it matches, trigger an alarm operation; if it does not match, proceed to step A3. Step A3: Determine whether the standard log data matches the intelligent channel analysis rule. If it matches, determine that the standard log data is the log data to be intelligently analyzed; if it does not match, determine that the standard log data is unmatched log data. The candidate log data consists of the log data to be intelligently analyzed and the log data without matching.

[0112] In some optional embodiments of this example, determining the malicious probability score of each candidate log based on a pre-trained discriminative model includes: The candidate log data is input into the pre-trained discriminative model to generate a high-dimensional semantic vector; Density clustering algorithm is used to perform density clustering on the vector set composed of the high-dimensional semantic vectors; Outliers and anomalous clusters are extracted from the clustering results. The candidate log data corresponding to the extracted outliers and anomalous clusters are used as samples to be labeled, and a fine-tuning training set is constructed by combining the results of manual labeling. The discriminative model is adjusted using the fine-tuning training set, and the adjusted discriminative model is deployed to the streaming computing engine to perform streaming inference on the candidate log data that flows in in real time, and output the malicious probability score corresponding to each candidate log data.

[0113] like Figure 9 As shown in the illustration, this application also provides a network security log intelligent analysis system. The system adopts a modular design, including a data acquisition and preprocessing module, a log feature extraction module, a rule filtering engine module, a discriminative AI alarm noise reduction engine, a generative AI alarm analysis engine, a multi-source log correlation analysis module, and a human-machine collaborative feedback module. The modules are connected via data interfaces and sequentially complete log acquisition, processing, analysis, and feedback according to a preset data processing flow, thus forming a complete closed-loop processing chain.

[0114] During the data flow process, the raw logs are first processed by the data acquisition and preprocessing module, and after standardization, they are transmitted to the log feature extraction module. After being filtered by the rule engine, they are further filtered by the discriminative AI alarm noise reduction engine. The retained logs are correlated in the multi-source log correlation analysis module, and the correlation results are comprehensively analyzed and judged by the generative AI alarm judgment engine. Finally, the security event analysis results are output, and continuous optimization is achieved through the human-machine collaborative feedback module.

[0115] The following is an introduction to each module: The data acquisition and preprocessing module is primarily used to obtain raw log data from various security devices, including firewalls, intrusion detection systems, intrusion prevention systems, host security systems, and database auditing systems. The collected log data is typically inconsistent in format and complex in content, thus requiring preprocessing. The preprocessing process includes data cleaning operations to remove null values ​​and duplicate data, as well as field parsing of the log content to extract key information such as timestamps, source IP addresses, destination IP addresses, port numbers, and protocol types. Simultaneously, logs from different sources are standardized in format, and in some implementations, time alignment is also performed to ensure consistency in subsequent analysis. After the above processing, the standardized log data is transmitted to the log feature extraction module.

[0116] The log feature extraction module is used to construct features from standardized logs. This module not only extracts structured features, such as IP addresses, port distribution, and protocol types, but also constructs behavioral features by combining them with the time dimension to reflect changes in access patterns. Furthermore, semantic encoding is performed on the textual descriptive information in the logs to obtain semantic features. Based on this, multiple features are fused to form a unified log vector representation, which is then transmitted as input data to both the discriminative AI alarm denoising engine and the generative AI alarm analysis engine.

[0117] The rule-based filtering engine module, as the core pre-processor for system log screening, receives the output from the log feature extraction module and forms a collaborative filtering mechanism with the discriminative AI alarm and noise reduction engine. It primarily performs preliminary screening of log vectors and raw feature data based on preset rules, filtering out invalid and low-value logs. This reduces the computational burden on subsequent AI engine analysis and improves overall analysis efficiency. The module has a built-in configurable rule library, covering two main types: static and dynamic rules. Static rules include preset security policy rules (such as prohibiting access from specific IP ranges, restricting abnormal port communication, and blocking illegal protocol transmissions) and log validity rules (such as filtering logs with incomplete formats or missing key fields). Dynamic rules can be updated and adjusted in real time based on system operating status, historical alarm data, and human-machine collaborative feedback. For example, temporary filtering rules can be set for log features with recent high-frequency false alarms, or interception rules can be added based on newly emerging attack features.

[0118] The discriminative AI alarm noise reduction engine is primarily used for initial screening of log data to reduce alarm noise. This engine classifies logs into normal logs, suspicious logs, and high-risk alarms by processing log vectors, and aggregates duplicate alarms with similar characteristics to reduce redundant information. Furthermore, the engine can identify abnormal behavior patterns and calculate a risk score or confidence level for each alarm. Through this processing, only high-value alarm data is retained and output to the generative AI alarm analysis engine.

[0119] The generative AI alerting and analysis engine performs deep semantic analysis on filtered logs. This module first constructs a prompt template for log analysis, organizing the log content and related contextual information as model input, and then calls the generative model for inference analysis. The model output is parsed and converted into structured information, including attack type, attack path, risk level, and corresponding handling suggestions. This approach enables semantic-level understanding of log content and improves the accuracy of analysis results. The processed results are then transmitted to the multi-source log correlation analysis module.

[0120] The multi-source log correlation analysis module is used to correlate logs from different devices. This module first aggregates logs based on time windows, then establishes correlations based on IP addresses, user identifiers, or asset information, and constructs correlation rules by combining behavioral characteristics. Based on this, a log correlation graph structure can be built. By analyzing the graph structure, attack paths can be reconstructed and multi-stage attack behaviors can be identified, thereby improving overall security incident analysis capabilities.

[0121] The human-machine collaborative feedback module receives human analysis results and corrects the system output. Security personnel can flag alarms identified by the system, such as false alarms or missed alarms, and make necessary corrections to the analysis results. This feedback data is used in the model optimization process, including updating the training data of the discriminative model, optimizing the prompt templates of the generative model, and adjusting relevant rule bases. Through continuous feedback, the system acquires the ability to continuously self-optimize.

[0122] Based on the aforementioned intelligent network security log analysis system, the following intelligent analysis methods can be implemented: First, multi-source security logs are collected, and the raw logs are cleaned, parsed, and standardized. Then, features are extracted from the log data, and a unified vector representation is generated. On this basis, a discriminative model is used to filter the logs to reduce alarm noise. For the filtered logs, a generative model is further used for semantic analysis to obtain structured analysis results. Next, correlation analysis is performed on the multi-source logs to construct attack chains and output security event analysis results. Finally, the model and rules are optimized and updated based on human feedback to continuously improve system performance. In some implementations, alarm noise reduction and semantic analysis processes can be executed in parallel.

[0123] One embodiment of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the network security log intelligent analysis method described in the foregoing embodiments.

[0124] An embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent analysis method for network security logs described in the foregoing embodiments.

[0125] An embodiment of this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the network security log intelligent analysis method described in the foregoing embodiments.

[0126] Figure 10A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] like Figure 10 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0128] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0129] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as a network security log intelligent analysis method.

[0130] For example, in some embodiments, a network security log intelligent analysis method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by computing unit 901, one or more steps of the network security log intelligent analysis method described above can be performed. Alternatively, in other embodiments, computing unit 901 can be configured to perform a network security log intelligent analysis method by any other suitable means (e.g., by means of firmware).

[0131] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0133] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0135] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0136] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0137] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0138] It should be noted that in the description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an." Furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context explicitly indicates otherwise. Additionally, the term "according to" should be understood as "at least partially based on…," and the term "based on" should be understood as "at least partially based on…," unless the context explicitly indicates otherwise.

[0140] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0141] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for intelligent analysis of network security logs, characterized in that, include: Based on a pre-configured transmission protocol, raw log data from each security device is received in real time, and preprocessing operations are performed on the raw log data to obtain standard log data. Based on the logic of combining operators and multiple fields, a matching operation is performed on the standard log data to determine candidate log data. Based on a pre-trained discriminative model, the malicious probability score of each candidate log is determined, and candidate log data that is greater than or equal to a preset score threshold is identified as target log data. Using the full field information and context information of the target log data as input, and based on a pre-trained generative model, the analysis results of the target log data are output.

2. The method according to claim 1, characterized in that, The preprocessing operation performed on the raw log data to obtain standard log data includes: Remove null values ​​and duplicate data from the original log data to obtain the first log data; Based on a pre-configured rule repository, the first log data is parsed to extract its structured fields, thus obtaining the second log data. Redundant fields are removed from the second log data, and the remaining fields are normalized across devices to obtain the standard log data.

3. The method according to claim 2, characterized in that, The steps for configuring the rule repository include: A parsing rule metadata table is established with equipment manufacturer, product model and log version number as three-level indexes, and the parsing rule metadata table is stored in the rule warehouse in a first preset format. Each parsing rule consists of a matching anchor, a parsing mode identifier, a field mapping template and a log version number. The rule repository is monitored by a rule hot-loading daemon. When a change in the log version number in the rule repository is detected, the newly added log data is subjected to syntax verification and sample replay testing. If the verification passes, the parsing rule metadata table is updated based on the newly added log data.

4. The method according to claim 2, characterized in that, The process of removing redundant fields from the second log data and normalizing the remaining fields includes: Configure a field whitelist based on the input characteristics of the generative model and the discriminative model; Based on the field whitelist, redundant fields in the second log data are removed, wherein the remaining fields include timestamp, source IP address, destination IP address, and event severity level; For the remaining fields, those that are semantically equivalent but have inconsistent field names and value formats are uniformly mapped to standard field names. Different values ​​corresponding to the severity level of the event are uniformly mapped to a risk quantification score range; The timestamps are uniformly converted into a second preset format, and the source IP address and the target IP address are subjected to demasking and address restoration operations.

5. The method according to claim 1, characterized in that, The step of performing a matching operation on the standard log data based on operator and multi-field combination logic to determine candidate log data from the standard log data includes: Based on preset filtering rules, a rule syntax tree is constructed, wherein the nodes of the rule syntax tree are triples consisting of fields, operators, and values. The standard log data is traversed from top to bottom through the rule syntax tree to determine the current hit status of the standard log data in the rule syntax tree; Based on the current hit status and matching priority, candidate log data is determined from the standard log data.

6. The method according to claim 5, characterized in that, Determining the current hit status of the standard log data in the rule syntax tree includes: The matching method is determined based on the type of the operator in the currently traversed node; Determine the matching termination condition based on the logical type of the currently traversed node; Based on the matching method and the matching termination condition, the current hit status of the standard log data in the rule syntax tree is determined.

7. The method according to claim 5, characterized in that, The preset filtering rules include whitelist allow rules, abnormal list interception rules, and rules to be analyzed by intelligent channels. The matching priority is that whitelist allow rules take precedence over abnormal list interception rules, and abnormal list interception rules are preceded by rules to be analyzed by intelligent channels. The step of determining candidate log data from the standard log data based on the current hit status and matching priority includes: Step A1: Determine whether the standard log data matches the whitelist allow rule. If it matches, discard the standard log data; if it does not match, proceed to step A2. Step A2: Determine whether the standard log data matches the anomaly list interception rule. If it matches, trigger an alarm operation; if it does not match, proceed to step A3. Step A3: Determine whether the standard log data matches the intelligent channel analysis rule. If it matches, determine that the standard log data is the log data to be intelligently analyzed; if it does not match, determine that the standard log data is unmatched log data. The candidate log data consists of the log data to be intelligently analyzed and the log data without matching.

8. The method according to claim 1, characterized in that, The pre-trained discriminative model determines the malicious probability score of each candidate log, including: The candidate log data is input into the pre-trained discriminative model to generate a high-dimensional semantic vector; Density clustering algorithm is used to perform density clustering on the vector set composed of the high-dimensional semantic vectors; Outliers and anomalous clusters are extracted from the clustering results. The candidate log data corresponding to the extracted outliers and anomalous clusters are used as samples to be labeled, and a fine-tuning training set is constructed by combining the results of manual labeling. The discriminative model is adjusted using the fine-tuning training set, and the adjusted discriminative model is deployed to the streaming computing engine to perform streaming inference on the candidate log data that comes in in real time, and output the malicious probability score corresponding to each candidate log data.

9. A network security log intelligent analysis device, characterized in that, include: The data preprocessing module is configured to receive raw log data from each security device in real time based on a pre-configured transmission protocol, and perform preprocessing operations on the raw log data to obtain standard log data. The candidate log data determination module is configured to perform a matching operation on the standard log data based on the combination logic of operators and multiple fields, and determine candidate log data from the standard log data; The target log data determination module is configured to determine the malicious probability score of each candidate log based on a pre-trained discriminative model, and determine the candidate log data that is greater than or equal to a preset score threshold as the target log data. The intelligent analysis module is configured to take the full field information and context information of the target log data as input, and output the analysis results of the target log data based on a pre-trained generative model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the intelligent analysis method for network security logs as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the intelligent analysis method for network security logs as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the intelligent analysis method for network security logs as described in any one of claims 1 to 8.

Images