A security log processing method and system, electronic equipment and storage medium

Abstract

This application discloses a method, system, electronic device, and storage medium for processing security logs, belonging to the technical field of network security technology. The method for processing security logs includes: acquiring multiple security logs from a target device; inputting the security logs into a log bypass processing unit and a log qualitative processing unit respectively; aggregating the security logs based on the log qualitative results output by the log qualitative processing unit to obtain security alarms; qualitatively classifying the security alarms; and generating corresponding security events based on the alarm qualitative results. This application can avoid aggregating normal business behavior and malicious attack behavior in the same security alarm, accurately generating security events.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to a method, system, electronic device and storage medium for processing security logs. Background Technology

[0002] Security logs refer to raw log data generated by network devices, security devices, hosts, etc., that records security-related activities. Due to the massive volume, high frequency, and fragmented nature of security logs, a single security log entry often cannot independently reflect a complete attack chain or abnormal behavior pattern, and direct processing would lead to low analysis efficiency and a surge in storage costs. Therefore, it is necessary to aggregate security logs before analysis.

[0003] In related technologies, aggregation is usually performed based on shallow features such as source IP (Internet Protocol) address, destination IP address, and rule ID. However, the above-mentioned methods of processing security logs cannot understand the behavioral intent behind the logs. They are very likely to aggregate attacker scanning behavior with normal business crawling, peak business access and other white logs into a single alarm, resulting in an extremely low alarm signal-to-noise ratio and failure to identify key threats.

[0004] Therefore, how to avoid aggregating normal business behavior and malicious attack behavior in the same security alert and accurately generate security events is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a method, system, electronic device, and storage medium for processing security logs, which can avoid aggregating normal business behavior and malicious attack behavior in the same security alarm and accurately generate security events.

[0006] To address the aforementioned technical problems, this application provides a method for processing security logs, comprising: Multiple security logs from the target device are acquired, and the security logs are respectively input into a log bypass processing unit and a log qualitative processing unit; wherein, the log bypass processing unit is used to perform feature learning on the security logs received within a first preset period using an artificial intelligence model, and stores the learning results in a knowledge base; the log qualitative processing unit is used to perform qualitative analysis on the currently received security logs based on the knowledge base, and obtain log qualitative results; The security logs are aggregated based on the log qualitative results output by the log qualitative processing unit to obtain security alerts; The security alarms are characterized, and corresponding security events are generated based on the alarm characterization results.

[0007] Optionally, the security logs are aggregated based on the log qualitative results output by the log qualitative processing unit, including: Based on the log qualitative results output by the log qualitative processing unit, the security log is divided into multiple log groups; Within each log group, security logs with the same characteristics are aggregated according to a preset time window.

[0008] Optionally, the security logs are aggregated based on the log qualitative results output by the log qualitative processing unit, including: The log qualitative results output by the log qualitative processing unit are optimized using a first optimization rule, and the security logs are aggregated based on the optimized log qualitative results; wherein, the first optimization rule includes log qualitative rules generated by the log analysis model and / or the expert system.

[0009] Optionally, the security alarm is characterized, including: The security alarm is input into the alarm characterization processing unit, which performs one or a combination of operations on the security alarm, including large language model analysis, alarm scenario characterization, and vulnerability characterization, to obtain the alarm characterization result.

[0010] Optionally, the security alarm is input into the alarm qualitative processing unit, including: The security alerts are compared with historical alerts for similarity. If the similarity is less than a preset value, the security alarm is input into the alarm characterization processing unit to obtain the alarm characterization result; Correspondingly, it also includes: If the similarity is greater than or equal to the preset value, the type of the security alarm is set to the target type, and it is determined whether the time difference between the push time of the last push of the historical alarm of the target type and the current time is greater than the preset duration. If so, the security alert will be pushed to the alerting platform; If not, discard the security alert.

[0011] Optionally, generating a corresponding security event based on the alarm characterization result includes: The alarm qualitative results are optimized using a second optimization rule; wherein the second optimization rule is generated based on the bypass processing results; the process of generating the bypass processing results is as follows: the alarm bypass processing unit performs correlation analysis, webshell analysis, and delay investigation analysis on the security alarms generated within the second preset period to obtain the bypass processing results; The corresponding security events are generated based on the optimized alarm classification results.

[0012] Optionally, after receiving a security alert, the following may also be included: The security alerts are correlated with historical alerts across time periods. Based on the correlation results, continuous attack activities are identified, and the associated alert sequences are aggregated into corresponding security events.

[0013] This application also provides a security log processing system, the system comprising: The log processing module is used to acquire multiple security logs from the target device and input the security logs into a log bypass processing unit and a log qualitative processing unit, respectively. The log bypass processing unit is used to perform feature learning on the security logs received within a first preset period using an artificial intelligence model and store the learning results in a knowledge base. The log qualitative processing unit is used to perform qualitative analysis on the currently received security logs based on the knowledge base to obtain the log qualitative analysis result. The alarm generation module is used to aggregate the security logs based on the log qualitative results output by the log qualitative processing unit to obtain security alarms. The event generation module is used to characterize the security alarm and generate corresponding security events based on the alarm characterization results.

[0014] This application also provides a storage medium storing a computer program thereon, which, when executed, implements the steps of the above-described security log processing method.

[0015] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor, when calling the computer program in the memory, implements the steps of the above-described security log processing method.

[0016] This application provides a method for processing security logs. The method inputs multiple security logs from a target device into a log bypass processing unit and a log qualitative processing unit. The log bypass processing unit uses an artificial intelligence model to learn from the security logs received within a preset period and stores the learning results in a knowledge base. The log qualitative processing unit can directly qualitatively analyze the currently received security logs based on the knowledge base to obtain the log qualitative results. Based on the above log qualitative results, this application aggregates the received security logs and generates security alarms for the aggregation results, so as to generate corresponding security events based on the qualitative results of the security alarms. In the above process, the security logs are first qualitatively analyzed based on an artificial intelligence model, and then the security logs are aggregated based on the log qualitative results to obtain security alarms, avoiding the situation of mixed black and white logs caused by aggregation based on simple information fields in conventional solutions. Therefore, this application can avoid aggregating normal business behavior and malicious attack behavior in the same security alarm, and accurately generate security events. This application also provides a security log processing system, a storage medium, and an electronic device, which have the above-mentioned beneficial effects, and will not be elaborated further here. Attached Figure Description

[0017] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a security log processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a security operation system provided in an embodiment of this application; Figure 3 A flowchart illustrating log qualitative processing provided in this application embodiment; Figure 4 A flowchart illustrating an alarm characterization process provided in an embodiment of this application; Figure 5 This is a flowchart illustrating an AI interpretation and event generation method provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Please see below. Figure 1 , Figure 1 This is a flowchart illustrating a security log processing method provided in an embodiment of this application.

[0021] Specific steps may include: S101: Obtain multiple security logs from the target device, and input the security logs into the log bypass processing unit and the log qualitative processing unit respectively.

[0022] This embodiment can be applied to a security operations platform capable of analyzing log data. The platform includes a log bypass processing unit, a log qualitative processing unit, and other functional units for processing logs or alarms.

[0023] The aforementioned log bypass processing unit is used to perform feature learning on the security logs received within a first preset period using an artificial intelligence model, and stores the learning results in a knowledge base; the log qualitative processing unit is used to qualitatively classify the currently received security logs based on the knowledge base to obtain log qualitative results. Qualitative classification of security logs means determining the threat type of the logs; the obtained log qualitative results can include non-threatening white logs, threatening black logs, and gray logs requiring manual judgment. The qualitative operation described in this embodiment refers to the operation of judging the nature and classifying security logs or security alarms.

[0024] Specifically, the log bypass processing unit can deploy various artificial intelligence (AI) models. These AI models can extract log semantics and behavioral patterns from security logs, automatically identify abnormal features, and generate learning results. The learning results construct a high-dimensional behavioral model that accurately characterizes normal business operations and abnormal attacks. Feature learning performed by the AI ​​models can include: business behavior learning, scanning behavior learning, and malicious IP learning. After storing the above learning results in a knowledge base, the knowledge stored in the knowledge base includes: normal business characteristics, abnormal behavior characteristics, a malicious attack fingerprint database, and dynamically updated threat intelligence association rules. The IP mentioned in this document refers to IP addresses.

[0025] This step allows for the acquisition of multiple security logs from the target device, enabling batch processing of these logs. Specifically, this step inputs the security logs into the log bypass processing unit and the log qualitative processing unit. The log qualitative processing unit utilizes knowledge from a knowledge base to compare and contrast the current security logs, distinguishing between normal business operations, false alarms, or real attacks, and then outputs the corresponding log qualitative results.

[0026] S102: Aggregate the security logs based on the log qualitative results output by the log qualitative processing unit to obtain security alarms.

[0027] After obtaining the log qualitative processing unit's output, this step can group all received security logs based on the qualitative results, resulting in multiple log groups. Each log group can include at least one security log. This step can then aggregate the security logs in each log group to obtain the corresponding security alert. Specifically, this step can logically merge the security logs according to preset dimensions (such as time window, attack source IP, target asset, or behavioral characteristics), and the resulting aggregation result is the security alert.

[0028] A security alert is a notification that alerts you to potential threats, generated by security detection rules or models after aggregating security logs. Aggregation refers to the process of reducing noise and summarizing key information from the original data.

[0029] S103: The security alarm is characterized, and a corresponding security event is generated based on the alarm characterization result.

[0030] This step, based on the security alert received, involves classifying the alert to determine its threat type. The classification result describes the type of the security alert, such as a real attack, false alarm, testing behavior, or suspicious activity. It also clarifies its specific nature (e.g., scanning and probing, successful exploitation, brute-force attack, lateral movement, etc.) and severity level, providing a basis for subsequent security incident generation and response strategies. A security incident refers to a confirmed security threat event that requires action. Specifically, if the security alert is classified as a threat alert, key attack characteristics can be extracted, and relevant context logs and asset information can be correlated to obtain the corresponding security incident.

[0031] In this embodiment, multiple security logs from the target device are input into a log bypass processing unit and a log qualitative processing unit. The log bypass processing unit uses an artificial intelligence model to learn from the security logs received within a preset period and stores the learning results in a knowledge base. The log qualitative processing unit can directly qualitatively analyze the currently received security logs based on the knowledge base to obtain the log qualitative results. Based on the above log qualitative results, this embodiment aggregates the received security logs and generates a security alarm for the aggregation result, so as to generate a corresponding security event based on the qualitative results of the security alarm. In the above process, the security logs are first qualitatively analyzed based on an artificial intelligence model, and then the security logs are aggregated based on the log qualitative results to obtain a security alarm, avoiding the situation of mixed black and white logs caused by aggregation based on simple information fields in conventional solutions. Therefore, this embodiment can avoid aggregating normal business behavior and malicious attack behavior in the same security alarm and accurately generate security events.

[0032] As for Figure 1 A further description of the corresponding embodiment: the process of aggregating the security logs based on the log qualitative results output by the log qualitative processing unit includes: dividing the security logs into multiple log groups based on the log qualitative results output by the log qualitative processing unit, so that security logs with the same log qualitative results are in the same log group; within each log group, security logs with the same characteristics are aggregated according to a preset time window. These characteristics may include source IP address, destination IP address, attack type, port number, protocol type, or session identifier.

[0033] The log qualitative processing unit outputs a log qualitative result that may include a primary qualitative conclusion and secondary qualitative conclusions. The primary qualitative conclusion describes whether a threat exists in the log, while the secondary qualitative conclusion describes secondary conclusions such as asset type and whether proxy nodes are involved. In this embodiment, the security log can be divided into multiple log groups, such as a black log group, a white log group, and a gray log group, based on the primary qualitative conclusion in the log qualitative result.

[0034] The above process, by aggregating security logs according to a preset time window, can effectively compress duplicate alarm data, restore the complete attack behavior chain, and significantly reduce the false alarm rate to improve analysis efficiency.

[0035] As for Figure 1 A further description of the corresponding embodiment: the process of aggregating the security logs based on the log qualitative results output by the log qualitative processing unit includes: optimizing the log qualitative results output by the log qualitative processing unit using a first optimization rule, and aggregating the security logs based on the optimized log qualitative results; wherein, the first optimization rule includes log qualitative rules generated by a log analysis model and / or an expert system. The optimized log qualitative results may be the same as or different from the log qualitative results output by the log qualitative processing unit.

[0036] This embodiment can utilize a log analysis model to perform machine learning or deep training on historical security logs and their corresponding qualitative results, automatically mining attack behavior patterns and optimizing judgment thresholds to obtain tuning rules for the log analysis model. This embodiment can also determine the tuning rules for the expert system based on the log qualitative rules set by experts. The aforementioned first tuning rule is a set of rules used to correct the weight coefficients, confidence scores, false positive filtering conditions, or dynamic aggregation strategies of the initial qualitative conclusions. The aforementioned tuning refers to the operation of correcting the log qualitative results to make them more accurate.

[0037] By optimizing the log characterization results using the first optimization rule, this solution can effectively eliminate false positives and false negatives in the initial characterization process, dynamically adapt to changes in attack characteristics under different network environments, and significantly improve the accuracy and confidence of the log characterization results.

[0038] As for Figure 1 In a further description of the corresponding embodiment, after receiving a security alarm, the security alarm can be characterized in the following way: the security alarm is input into the alarm characterization processing unit, so that the alarm characterization processing unit performs any one or a combination of any of the following operations: large language model analysis, alarm scenario characterization, and vulnerability characterization, to obtain the alarm characterization result.

[0039] The aforementioned large language model analysis refers to: utilizing the powerful semantic understanding and logical reasoning capabilities of pre-trained large language models to conduct in-depth analysis of the content, context logs, and related information of security alerts, automatically identifying attack intent, determining the authenticity of security alerts (distinguishing between false alarms and real attacks), and generating a natural language analysis report that includes a description of the attack method, confidence score, and handling suggestions.

[0040] The aforementioned alarm scenario qualitative analysis refers to mapping discrete alarm events to specific attack methods based on the time sequence, source and destination addresses, port protocols, and behavioral characteristics of security alarms, thereby identifying the attack stage to which the security alarm belongs and the specific attack scenario type (such as brute-force attacks, malicious file downloads, data theft, ransomware propagation, etc.).

[0041] The aforementioned vulnerability assessment refers to: combining the target asset information involved in the security alert with the currently known vulnerability knowledge base, analyzing the specific security weaknesses that the alert exploits or attempts to exploit, assessing whether the target asset has corresponding unpatched vulnerabilities, and determining the likelihood and potential severity of the vulnerability exploitation.

[0042] Furthermore, in the process of inputting the security alarm into the alarm qualitative processing unit, this embodiment can compare the security alarm with historical alarms for similarity. If the similarity is less than a preset value, the security alarm is input into the alarm qualitative processing unit to obtain the alarm qualitative result. When comparing the security alarm with historical alarms for similarity, features such as attack type, network quintuple, and threat level can be compared to obtain the similarity between the two.

[0043] Correspondingly, if the similarity is greater than or equal to the preset value, the type of the security alarm is set as the target type. It is then determined whether the time difference between the last push time of the historical alarm of the target type and the current time is greater than a preset duration. If yes, the security alarm is pushed to the alarm platform; otherwise, the security alarm is discarded. Through the above similarity comparison process, this embodiment can effectively achieve alarm deduplication and suppression, avoiding repeated pushes of known and frequently occurring similar alarms, thereby reducing the computational load of the alarm qualitative processing unit.

[0044] Furthermore, the process of generating corresponding security events based on alarm characterization results includes: optimizing the alarm characterization results using a second optimization rule; and generating corresponding security events based on the optimized alarm characterization results. The second optimization rule is generated based on bypass processing results; the process of generating the bypass processing results involves using an alarm bypass processing unit to perform correlation analysis, webshell analysis, and delay investigation analysis on security alarms generated within a second preset period to obtain the bypass processing results. The aforementioned webshell is a code execution environment existing in the form of a web page file, and is also a type of malicious script.

[0045] The aforementioned correlation analysis refers to: based on key features such as time window, source IP address, destination IP address, attack fingerprint or session identifier, logically aggregating and chaining multiple discrete security alarms generated within the second preset period to identify alarm sets belonging to the same attack activity, thereby distinguishing between single-point scanning noise and persistent attack behavior.

[0046] The aforementioned webshell analysis refers to: analyzing whether the file content contains webshell characteristics (such as encrypted and obfuscated code, system command execution functions, file operation interfaces, etc.) in response to security alerts involving script file uploads, abnormal code execution, or sensitive function calls. Combining the file creation time, permission changes, and subsequent process behavior, it is necessary to confirm whether a malicious webshell script has been successfully implanted on the target server, and to assess the active status, connection source, and potential control risks of the webshell script.

[0047] The aforementioned delayed investigation and assessment (i.e., secondary investigation and assessment) refers to the following: for suspicious alarms with insufficient initial evidence, low confidence, or that are under observation, a final determination is not made immediately, but the alarm is suspended and the relevant logs and behavioral trajectories are continuously monitored within a subsequent preset time period; by introducing new evidence in the time dimension, the initial alarm is retrospectively verified and re-scored to eliminate misjudgments or omissions caused by information lag.

[0048] The above process uses the second optimization rule to optimize the alarm qualitative results, which can fuse and calibrate the alarm qualitative results based on real-time single-point analysis with the bypass depth analysis results based on long period and multi-dimensionality, thereby reducing the false alarm rate.

[0049] Furthermore, after receiving a security alert, this application can also correlate the security alert with historical alerts across time, identify persistent attack activities based on the correlation results, and aggregate the associated alert sequences into corresponding security events. This process, by correlating current alerts with historical data across time, identifies dispersed but co-originating persistent attack behaviors, enabling the aggregation of alert sequences belonging to the same attack chain into a single security event, restoring the complete attack context, effectively avoiding alert fragmentation, and improving the efficiency of perceiving and handling long-term latent threats.

[0050] The process described in the above embodiments is illustrated below through examples in practical applications.

[0051] In this field, the conventional method for security log aggregation is to aggregate security logs based on source IP address, destination IP address, access direction, and rule ID to obtain security alerts. However, the above-mentioned conventional security log aggregation scheme has the following drawbacks: (1) Single aggregation dimension, low signal-to-noise ratio: Conventional solutions typically rely on shallow features such as source IP, destination IP, and rule ID (identifier) ​​for aggregation. This aggregation method fails to understand the behavioral intent behind the logs and easily combines attacker scanning behavior with normal business crawling and peak-hour access logs into a single alert, resulting in an extremely low signal-to-noise ratio and the suppression of critical threats.

[0052] (2) Lack of qualitative intelligence: Existing systems lack a deep understanding and qualitative ability of logs before aggregation, and cannot automatically distinguish complex scenarios such as "business false alarms", "vulnerability access", and "real attack success", resulting in semantic ambiguity in the aggregation results, which still requires security experts to invest a lot of time in manual analysis.

[0053] (3) The system is rigid and difficult to adapt to changes: Aggregation and analysis systems based on fixed rules are ill-equipped to cope with new attack methods and rapidly changing business environments; the systems have poor scalability, requiring manual rule writing for each new threat response, resulting in slow response times.

[0054] To address the technical issues present in the aforementioned conventional solutions, this embodiment provides a security operation system based on AI (Artificial Intelligence) qualitative log aggregation. This embodiment offers a two-stage processing paradigm: "AI-based qualitative analysis followed by homogeneous aggregation." The entire system is a pipeline tightly integrated with data flow and intelligent judgment, achieving intelligent qualitative analysis, accurate aggregation, and efficient operation of logs through multi-layered processing flows. AI qualitative analysis (AI Judgement) refers to using artificial intelligence technology to determine the threat type of logs or alerts. Aggregation refers to the process of merging multiple similar objects into a more representative and operational object.

[0055] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a security operation system provided in an embodiment of this application. The functional units in the system include: log preprocessing, log bypass processing, log qualitative processing, log qualitative tuning, log aggregation processing, alarm aggregation processing, alarm cooling processing, alarm preprocessing, alarm bypass processing, alarm qualitative processing, alarm qualitative tuning, alarm output, AI interpretation and event generation.

[0056] The raw security logs are input into the security operations system from the log preprocessing unit. The output of the log bypass processing is stored in the knowledge base, which can be referenced by the log qualitative processing unit. After the alarm cold processing unit outputs a security alarm, it determines whether the security alarm has been qualitatively determined. If it has been qualitatively determined, an alarm is output; otherwise, it enters the alarm preprocessing module for further processing.

[0057] This embodiment can solve the problems of low operational efficiency, delayed threat response, and high false positive / false negative rates caused by massive, noisy, and inefficient security logs and alarms in existing network security operations. Specifically, the objectives of this invention include: This embodiment fundamentally solves the "mixed black and white logs" problem caused by traditional aggregation based on simple information fields by introducing an innovative "qualitative first, aggregation later" paradigm. It accurately separates threats from false alarms, achieving precise aggregation. This embodiment utilizes AI technology (including tuning rules, algorithm models, and large language models) to intelligently qualitatively classify each log entry and then group and aggregate them based on the qualitative results. This compresses tens of millions or even hundreds of millions of raw logs into a small number of refined, high-value alerts and events, greatly reducing the workload of security operations personnel and improving operational efficiency. The closed-loop operation system constructed in this embodiment integrates the entire process from log preprocessing, AI qualitative analysis, intelligent aggregation to event generation and automatic handling. Through mechanisms such as side-channel learning and qualitative tuning, it continuously self-optimizes, forming a continuously evolving intelligent security operation closed loop.

[0058] This security operations system first performs AI-based qualitative analysis on security logs, then aggregates similar logs based on the qualitative results, achieving accurate aggregation and noise reduction. The system includes functional units such as log preprocessing, bypass processing, qualitative analysis, aggregation processing, alarm processing, and event generation. AI qualitative analysis includes business-related qualitative analysis, vulnerability qualitative analysis, IOC qualitative analysis, scan-based qualitative analysis, and LLM model qualitative analysis. The system uses a Redis knowledge base to store bypass processing results for quick querying by the main microservice. The system also includes a dynamic tuning module that continuously optimizes qualitative thresholds and aggregation strategies through reinforcement learning. The Large Language Model (LLM) is a pre-trained generative model based on the Transformer (a deep learning model) architecture, used for natural language understanding and generation. The IOC (Indicator of Compromise) is used to characterize clues that the system may have been compromised.

[0059] The following describes the workflow of the security operations system, including the following steps: Step 1: Input raw log data.

[0060] The aforementioned raw log data is the same as the security log data; specifically, the security operations system can receive security logs from various security devices, network devices, and application systems.

[0061] Step 2: Perform log preprocessing on the security log.

[0062] This step is implemented by the log preprocessing unit. The specific preprocessing operations include sequentially performing log filtering, pre-enrichment, log whitening, duplicate log deduplication, and post-enrichment.

[0063] The log filtering process is as follows: Based on expert operational experience, filter out false alarm log rule IDs that are clearly identified as low quality.

[0064] The pre-enrichment process involves assigning default values ​​to necessary fields in the log information (such as labels and matrix fields used to fill in analysis information).

[0065] The process of whitelisting logs is as follows: users manually mark logs as ignorable.

[0066] The process of deduplicating duplicate logs is as follows: logs that are completely duplicated within a short period of time (such as within 20 seconds) are deduplicated to reduce the pressure on subsequent processing.

[0067] The post-enrichment process involves enriching information such as scanner IP and asset tags to provide richer features for AI qualitative analysis.

[0068] The preprocessed security logs are input from the log bypass to the log bypass processing unit and from the log main link to the log qualitative processing unit.

[0069] Step 3: Perform log bypass processing.

[0070] This step can be implemented by the log bypass processing unit, and the specific operations performed include: business behavior learning, scanning behavior learning, and malicious IP learning.

[0071] Business behavior learning refers to: deeply understanding user business scenarios (such as business logs, business interfaces, etc.) by fine-tuning the model, and storing the learning results in the Redis knowledge base.

[0072] Scanning behavior learning refers to identifying the scanner IP based on a time window and outputting it to the Redis knowledge base.

[0073] Malicious IP learning refers to recording IPs that AI identifies as attacking and storing them in a Redis knowledge base to avoid subsequent misjudgments and improve the efficiency of black log identification.

[0074] The learning results can be stored in both the Redis and MongoDB knowledge bases. The log qualitative processing unit can reference data from the Redis knowledge base during its operation, and users can use the MongoDB knowledge base to understand the operational status. Redis is an open-source in-memory data structure store used as a database, cache, and message broker; MongoDB is a database based on distributed file storage.

[0075] Furthermore, the log bypass processing unit can be equipped with an artificial intelligence model for learning business behavior, an artificial intelligence model for learning scanning behavior, and an artificial intelligence model for learning malicious IPs. The preprocessed security logs are learned using these artificial intelligence models, and the corresponding learning results are stored in the knowledge base.

[0076] Step 4: Perform log qualitative processing.

[0077] This step can be implemented by the log qualitative processing unit; please refer to [link / reference]. Figure 3 , Figure 3 A flowchart of a log qualitative processing method provided in an embodiment of this application is shown below. Figure 3 As shown, the specific operations performed by the log qualitative processing unit include: vulnerability scanning qualitative analysis, insufficient data qualitative analysis, IOC qualitative analysis, business qualitative analysis, vulnerability qualitative analysis, rule qualitative analysis, and scan qualitative analysis. If it belongs to the IOC type, intelligence query qualitative analysis is performed. If the security log meets the vulnerability criteria, the corresponding vulnerability rules are output. Figure 3The dashed arrows indicate that if the previous qualitative operation fails, the next qualitative operation will proceed. The results of each log qualitative processing step can be input into the log qualitative tuning unit for AI qualitative tuning.

[0078] The process of identifying missed scans includes: identifying logs generated by the IP addresses of the missed scan assets.

[0079] The process of identifying insufficient data includes: processing incomplete information logs from third parties.

[0080] The qualitative process of IOC includes: querying threat intelligence interfaces and matching known attack indicators. Threat intelligence is evidentiary knowledge about existing or potential threats.

[0081] The process of business characterization includes: analyzing business scenarios to distinguish between normal business behavior and attacks.

[0082] The process of identifying vulnerabilities includes: identifying weak passwords, insecure configurations, etc.

[0083] The process of defining rules includes: rapid classification based on expert experience rules.

[0084] The process of qualitative scanning includes: identifying scanning behavior (such as short-term, high-frequency requests) through time windows.

[0085] Specifically, the qualitative analysis of missed scans, insufficient data, business operations, and scanning can be achieved based on a knowledge base; the qualitative analysis of IOC can be achieved through intelligence lookup interfaces; the qualitative analysis of vulnerabilities can be achieved through artificial intelligence models; and the qualitative analysis of rules can be achieved manually.

[0086] Step 5: Perform qualitative log tuning.

[0087] This step can be implemented by the log qualitative tuning unit, which can form tuning rules through AI analysis and expert judgment to improve the accuracy of qualitative analysis.

[0088] Step 6: Perform log aggregation processing.

[0089] This step can be implemented by the log aggregation processing unit, which can sequentially perform qualitative grouping and log aggregation operations to obtain corresponding security alerts.

[0090] The qualitative grouping process includes grouping logs based on their primary qualitative conclusion. For example, all logs classified as "false alarms" are placed in the white log group, all logs classified as "successful real attacks" are placed in the black log group, and the rest are placed in the gray log group. Black logs are those classified as real attacks, viruses, or other valid threats and require immediate action. White logs are those classified as false alarms or normal behavior and do not require attention. Gray logs are those that cannot be directly classified but are suspected of posing a threat and require further manual analysis.

[0091] The log aggregation process includes: within each qualitative group, logs with similar characteristics (such as the same attacker IP, the same target asset, and the same attack method) are aggregated into a security alert in a configurable time window (such as 20 seconds).

[0092] The above operations ensure that the aggregated alerts do not contain logs of different qualitative natures.

[0093] Step 7: Perform alarm aggregation processing.

[0094] This step can be implemented by the alarm aggregation processing unit, which can sequentially perform alarm filtering, alarm aggregation grouping, and alarm aggregation operations.

[0095] The alarm filtering process includes filtering alarm types with high false alarm rates based on expert operational experience.

[0096] The process of alarm aggregation and grouping includes: combining historical alarms and grouping them by similarity.

[0097] The alert aggregation process includes: generating alerts are further correlated across time (such as matching with historical alerts from the past 7 days for similarity), identifying persistent attack activities, and aggregating them into higher-level security events.

[0098] Step 8: Perform alarm cooling operation.

[0099] This step can be implemented by the alarm cooling processing unit. The alarm cooling processing unit sets a time window suppression mechanism for safety alarms with the same source or similar characteristics. Within a preset cooling period, it automatically merges duplicate alarms or postpones the push of newly generated similar alarms to avoid frequent push of safety alarms and reduce downstream pressure.

[0100] Step 9: Perform alarm preprocessing operations.

[0101] This step can be implemented by the alarm preprocessing unit, which can assign default values ​​to the necessary fields of the security alarm.

[0102] The pre-processed security alarms are input from the alarm bypass to the alarm bypass processing unit and from the alarm main link to the alarm qualitative processing unit.

[0103] Step 10: Perform alarm bypass processing.

[0104] This step can be implemented by the alarm bypass processing unit, which can perform correlation analysis, webshell analysis, and secondary investigation analysis.

[0105] The process of correlation analysis includes: correlating weak signals (such as multiple failed login attempts or abnormal port access) and identifying potential threats.

[0106] The process of webshell analysis includes: specific analysis and characterization of webshell-related alerts.

[0107] The secondary investigation and assessment process includes: querying the host behavior log 5 minutes after the alarm occurs and dynamically updating the qualitative conclusion.

[0108] Furthermore, the alarm bypass processing unit can be equipped with an artificial intelligence model for correlation analysis, an artificial intelligence model for webshell analysis, and an artificial intelligence model for secondary investigation analysis. The above artificial intelligence models are used to process security alarms so that the processing results can be transmitted to the alarm qualitative optimization unit.

[0109] Step 11: Perform alarm characterization processing.

[0110] This step can be implemented by the alarm characterization processing unit, which can perform LLM analysis, alarm scenario characterization, and vulnerability characterization.

[0111] The LLM analysis process includes: for alarms with complex data packets or ambiguous semantics, invoking a large model for inference. The alarm scenario characterization process includes: characterizing the alarm based on high-frequency attack scenarios (such as brute-force attacks and malicious file downloads). Vulnerability characterization is used to identify vulnerability alarms.

[0112] Alert classification can be categorized into black alerts, white alerts, and gray alerts. Black alerts are those identified as genuine attacks, viruses, or other valid threats and require immediate attention. White alerts are those identified as false alarms or normal business behavior and do not require attention. Gray alerts are those that cannot be directly classified but are suspected of posing a threat and require further manual analysis.

[0113] Please see Figure 4 , Figure 4This is a flowchart illustrating an alarm characterization process provided in an embodiment of this application. Upon receiving a security alarm, an LLM (Limited Least Meaning) assessment is first performed. If the security alarm meets the LLM assessment requirements, an LLM model assessment process is initiated. If the security alarm does not meet the LLM assessment requirements, a vulnerability rule assessment process is initiated. If the security alarm meets the vulnerability rules, vulnerability characterization is performed. If the security alarm does not meet the vulnerability rules, alarm scenario characterization is performed. Figure 4 The dashed arrows in the diagram indicate that if the previous qualitative operation fails, the next qualitative operation will proceed.

[0114] Step 12: Perform alarm qualitative adjustment operations.

[0115] This step can be implemented by the alarm qualitative optimization unit, which can improve the accuracy of alarm qualitative analysis through AI and expert judgment.

[0116] Step 13: Alarm output.

[0117] This step can be implemented by the alarm output unit, and the output results include: black alarm ai_black_alert, white alarm ai_white_alert, and gray alarm ai_gray_alert.

[0118] Black alerts require immediate action. Events can be generated and pushed to the platform for immediate handling or automatic blocking.

[0119] White alerts do not need to be monitored. They can be entered into the false alarm database for system learning, so as not to disturb maintenance personnel.

[0120] Gray alerts require manual confirmation. They are then pushed to the manual analysis platform for security experts to analyze whether a security event has been generated.

[0121] Step 14: AI Interpretation and Event Generation.

[0122] This step can be implemented by the AI ​​interpretation and event generation unit, which performs operations including alarm interpretation, event generation, and event interpretation.

[0123] The alarm interpretation process includes: AI automatically interpreting the generated black alarms and generating threat analysis and handling suggestions in natural language.

[0124] The process of event generation includes: generating security events from similar black alerts, pushing them to users, or handling them automatically.

[0125] The process of event interpretation includes: AI automatically interpreting the generated security events and generating threat analysis and handling suggestions in natural language.

[0126] Addressing the technical challenge of combining black and white logs in the same alarm unit, this embodiment fundamentally avoids aggregating normal business behavior (white logs) and malicious attack behavior (black logs) within the same alarm unit, thereby improving the purity and operability of alarms. Regarding the processing efficiency issue with massive log volumes, this embodiment achieves real-time, efficient intelligent characterization and aggregation on log volumes of tens of millions or even hundreds of millions of records, placing extremely high demands on system architecture and algorithm performance. Addressing the challenges of characterization accuracy and adaptability, this embodiment comprehensively utilizes multiple AI capabilities to achieve accurate characterization of complex and covert threats, and demonstrates how a feedback mechanism enables the system to continuously learn and self-optimize.

[0127] Please see Figure 5 , Figure 5 The flowchart provided in this application embodiment illustrates an AI interpretation and event generation process, which includes: interpreting a security alarm using the alarm qualitative result (cot_alert) to obtain an AI preliminary alarm (ai_outer_alert); if the alarm is subscribed to, the platform alarm event is stored in the event-access database. After the AI ​​preliminary alarm, the event generation engine can further process the event to obtain the original external event (ori_outer_incident); interpreting the event using the event (cot_incident) yields an AI confirmation event (ai_outer_incident); if the event is subscribed to, the platform alarm event is stored in the database.

[0128] This embodiment is based on a collaborative qualitative mechanism of multimodal AI: it does not rely solely on LLM, but instead constructs a collaborative qualitative system of "rule engine (fast) + dedicated model (accurate) + LLM (complex scenarios)". Rules and dedicated models handle most clear scenarios, while LLM handles complex and unknown threats that rules cannot cover and that require deep reasoning, pursuing the highest accuracy while ensuring efficiency. The rule engine is a CPU-level judgment program that makes judgments based on expert experience and model output knowledge. The dedicated model is the IOC qualitative model.

[0129] This embodiment provides the following "qualitative-aggregation" decoupling design: decoupling complex AI judgment from aggregation logic. The AI ​​is only responsible for tagging individual log entries, while the aggregation module is only responsible for merging by tag and feature. This design allows both to iterate and expand independently.

[0130] This embodiment provides a feedback-driven closed-loop optimization scheme: Security experts' feedback on the handling of black, white, and gray alerts output by the system (confirmation, false alarm, missed alarm) is collected by the system and used to optimize the qualitative model, generate / optimize rules, and adjust aggregation strategies. Specifically, in the model optimization phase, feedback data is used as training samples to fine-tune the qualitative model; in the rule evolution phase, expert experience is automatically accumulated to generate new qualitative or filtering rules; and in the strategy adjustment phase, the aggregation time window and similarity threshold are specifically optimized, thereby achieving continuous self-evolution of security operation capabilities.

[0131] Below are two examples of security log processing: (1) The following are examples of lateral movement attack detection: Input: Abnormal SMB connection logs of host A, suspicious process creation logs of host B.

[0132] AI characterization: Host A logs are characterized as gray logs (scanning behavior, but attack not confirmed), and Host B logs are characterized as gray logs (process creation, but malicious intent not confirmed).

[0133] Intelligent aggregation: If two logs occur within 5 minutes and involve the same user account, the system identifies them as conforming to the MITRE ATT&CK T1021 (remote service) tactic and generates a "suspected lateral movement" aggregated alarm with a confidence level of 85%.

[0134] Output: After confirmation by security personnel, the system records this pattern, and similar behaviors in the future can be directly identified as black logs.

[0135] (2) Case studies of handling false alarms during peak business hours are as follows: Input: Logs of a large number of users logging in during peak business hours.

[0136] AI qualitative analysis: The business qualitative analysis module identifies it as a white log (based on historical business patterns), while the scanning qualitative analysis module confirms it as normal business behavior.

[0137] Processing: The logs are classified as White and archived directly without requiring manual review.

[0138] Output result: 100,000 log entries were prevented from entering the alarm process, significantly reducing alarm noise.

[0139] After a large financial platform deployed this system, actual test data showed the following improvements: (1) Enhanced aggregation capability: The daily raw log volume has been increased from 100 million to 320,000 alarms and 280 security events.

[0140] (2) Operational efficiency improved, and the response time for security incidents was reduced from 1 hour to 8 minutes.

[0141] (3) Improved judgment rate: By accumulating 150,000 business behavior features through the Redis knowledge base, the automatic judgment rate has been improved by 50%.

[0142] (4) Improved accuracy: The overall qualitative accuracy rate is over 96%, which is significantly higher than that of traditional rule engines.

[0143] The comparison of the measured data is shown in the table below: Table 1 Comparison of Measured Data

[0144] As can be seen, using this solution improves alert quality, significantly increasing alert value and allowing security personnel to focus on genuine threats. This solution enables a shift in operational models from "human" to "AI-driven." It also reduces operating costs, drastically decreasing the manpower and time required for security operations and maintenance.

[0145] This application provides a security log processing system, comprising: The log processing module is used to acquire multiple security logs from the target device and input the security logs into a log bypass processing unit and a log qualitative processing unit, respectively. The log bypass processing unit is used to perform feature learning on the security logs received within a first preset period using an artificial intelligence model and store the learning results in a knowledge base. The log qualitative processing unit is used to perform qualitative analysis on the currently received security logs based on the knowledge base to obtain the log qualitative analysis result. The alarm generation module is used to aggregate the security logs based on the log qualitative results output by the log qualitative processing unit to obtain security alarms. The event generation module is used to characterize the security alarm and generate corresponding security events based on the alarm characterization results.

[0146] In this embodiment, multiple security logs from the target device are input into a log bypass processing unit and a log qualitative processing unit. The log bypass processing unit uses an artificial intelligence model to learn from the security logs received within a preset period and stores the learning results in a knowledge base. The log qualitative processing unit can directly qualitatively analyze the currently received security logs based on the knowledge base to obtain the log qualitative results. Based on the above log qualitative results, this embodiment aggregates the received security logs and generates a security alarm for the aggregation result, so as to generate a corresponding security event based on the qualitative results of the security alarm. In the above process, the security logs are first qualitatively analyzed based on an artificial intelligence model, and then the security logs are aggregated based on the log qualitative results to obtain a security alarm, avoiding the situation of mixed black and white logs caused by aggregation based on simple information fields in conventional solutions. Therefore, this embodiment can avoid aggregating normal business behavior and malicious attack behavior in the same security alarm and accurately generate security events.

[0147] Furthermore, the process by which the alarm generation module aggregates the security logs based on the log qualitative results output by the log qualitative processing unit includes: dividing the security logs into multiple log groups based on the log qualitative results output by the log qualitative processing unit; and aggregating security logs with the same characteristics within each log group according to a preset time window.

[0148] Furthermore, the process by which the alarm generation module aggregates the security logs based on the log qualitative results output by the log qualitative processing unit includes: optimizing the log qualitative results output by the log qualitative processing unit using a first optimization rule, and aggregating the security logs based on the optimized log qualitative results; wherein, the first optimization rule includes log qualitative rules generated by the log analysis model and / or the expert system.

[0149] Furthermore, the process of the event generation module to characterize the security alarm includes: inputting the security alarm into the alarm characterization processing unit, so that the alarm characterization processing unit performs any one or a combination of any of the following operations: large language model analysis, alarm scenario characterization, and vulnerability characterization, to obtain the alarm characterization result.

[0150] Furthermore, the process by which the event generation module inputs the security alarm into the alarm characterization processing unit includes: comparing the security alarm with historical alarms for similarity; if the similarity is less than a preset value, then inputting the security alarm into the alarm characterization processing unit to obtain the alarm characterization result; Correspondingly, it also includes: The cooling module is used to set the type of the security alarm to the target type if the similarity is greater than or equal to the preset value, and to determine whether the time difference between the push time of the last historical alarm of the target type and the current time is greater than a preset duration; if yes, the security alarm is pushed to the alarm platform; if no, the security alarm is discarded.

[0151] Furthermore, the process by which the event generation module generates a corresponding security event based on the alarm characterization result includes: optimizing the alarm characterization result using a second optimization rule; wherein, the second optimization rule is generated based on the bypass processing result; the process of generating the bypass processing result is as follows: using the alarm bypass processing unit to perform correlation analysis, webshell analysis, and delay investigation analysis on the security alarms generated within the second preset period to obtain the bypass processing result; and generating a corresponding security event based on the optimized alarm characterization result.

[0152] Furthermore, it also includes: The alarm association module is used to associate the security alarm with historical alarms across time after receiving the security alarm, identify continuous attack activities based on the association results, and aggregate the associated alarm sequences into corresponding security events.

[0153] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, and they will not be repeated here.

[0154] This application also provides a storage medium on which a computer program is stored, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0155] This application also provides an electronic device that may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the electronic device may also include various network interfaces, power supplies, and other components.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

[0157] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method of processing a security log, characterized by, include: Multiple security logs from the target device are acquired, and the security logs are respectively input into a log bypass processing unit and a log qualitative processing unit; wherein, the log bypass processing unit is used to perform feature learning on the security logs received within a first preset period using an artificial intelligence model, and stores the learning results in a knowledge base; the log qualitative processing unit is used to perform qualitative analysis on the currently received security logs based on the knowledge base, and obtain log qualitative results; The security logs are aggregated based on the log qualitative results output by the log qualitative processing unit to obtain security alerts; The security alarms are characterized, and corresponding security events are generated based on the alarm characterization results.

2. The method for processing security logs according to claim 1, characterized in that, The security logs are aggregated based on the log qualitative results output by the log qualitative processing unit, including: Based on the log qualitative results output by the log qualitative processing unit, the security log is divided into multiple log groups; Within each log group, security logs with the same characteristics are aggregated according to a preset time window.

3. The method for processing security logs according to claim 1, characterized in that, The security logs are aggregated based on the log qualitative results output by the log qualitative processing unit, including: The log qualitative results output by the log qualitative processing unit are optimized using a first optimization rule, and the security logs are aggregated based on the optimized log qualitative results; wherein, the first optimization rule includes log qualitative rules generated by the log analysis model and / or the expert system.

4. The method for processing security logs according to claim 1, characterized in that, The security alarm is characterized, including: The security alarm is input into the alarm characterization processing unit, which performs one or a combination of operations on the security alarm, including large language model analysis, alarm scenario characterization, and vulnerability characterization, to obtain the alarm characterization result.

5. The method for processing security logs according to claim 4, characterized in that, The security alarm is input into the alarm qualitative processing unit, including: The security alerts are compared with historical alerts for similarity. If the similarity is less than a preset value, the security alarm is input into the alarm characterization processing unit to obtain the alarm characterization result; Correspondingly, it also includes: If the similarity is greater than or equal to the preset value, the type of the security alarm is set to the target type, and it is determined whether the time difference between the push time of the last push of the historical alarm of the target type and the current time is greater than the preset duration. If so, the security alert will be pushed to the alerting platform; If not, discard the security alert.

6. The method for processing security logs according to claim 4, characterized in that, The generation of corresponding security events based on the alarm characterization results includes: The alarm qualitative results are optimized using a second optimization rule; wherein the second optimization rule is generated based on the bypass processing results; the process of generating the bypass processing results is as follows: the alarm bypass processing unit performs correlation analysis, webshell analysis, and delay investigation analysis on the security alarms generated within the second preset period to obtain the bypass processing results; The corresponding security events are generated based on the optimized alarm classification results.

7. The method for processing security logs according to claim 1, characterized in that, After receiving a security alert, the following is also included: The security alerts are correlated with historical alerts across time periods. Based on the correlation results, continuous attack activities are identified, and the associated alert sequences are aggregated into corresponding security events.

8. A security log processing system, characterized in that, include: The log processing module is used to acquire multiple security logs from the target device and input the security logs into a log bypass processing unit and a log qualitative processing unit, respectively. The log bypass processing unit is used to perform feature learning on the security logs received within a first preset period using an artificial intelligence model and store the learning results in a knowledge base. The log qualitative processing unit is used to perform qualitative analysis on the currently received security logs based on the knowledge base to obtain the log qualitative analysis result. The alarm generation module is used to aggregate the security logs based on the log qualitative results output by the log qualitative processing unit to obtain security alarms. The event generation module is used to characterize the security alarm and generate corresponding security events based on the alarm characterization results.

9. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and the processor, when invoking the computer program in the memory, implements the steps of the security log processing method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the steps of the security log processing method as described in any one of claims 1 to 7.

Images