Log data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a log data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps of performing feature matching processing on an information security log, and determining an equipment type and a log type; performing log analysis processing on the information security log based on the equipment type and the log type to obtain a key field; scene recognition processing is carried out based on the key field, and a threat scene to which the information security log belongs is determined; merging the information security logs according to the threat scene to obtain alarm log information; and generating a log alarm message based on the alarm log information. According to the method, the information security logs are processed through feature matching and log analysis, then scene recognition is performed through the extracted key field information, and the information security logs are merged, so that the logs can be effectively aggregated into alarm log information, valuable security events are efficiently screened out, and the safety of the security events is improved. And the alarm processing accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log data processing method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] With the development of computer technology, logging technology has emerged. During computer operation, network devices, systems, and service programs generate a record of events called a log. Each log line contains a description of the relevant operation, including the date, time, user, and action. For example, in the field of information security, log files that record and store security events and threats are called information security logs. Within this field, various security devices and systems (such as firewalls, intrusion detection systems, and security information and event management systems) generate a large number of alarm logs to record possible security issues, abnormal behavior, or potential attacks.

[0003] Currently, the processing and analysis of information security logs generally uses machine learning methods to discover corresponding anomalies in the information security logs, and then issue alarms based on these anomalies. However, this method faces massive, diverse, and complex information security logs. It will identify a large number of abnormal logs but fail to screen out valuable security events, thus affecting the accuracy of alarm processing. Summary of the Invention

[0004] Based on this, it is necessary to provide a log data processing method, device, computer equipment, computer-readable storage medium and computer program product that can effectively improve the accuracy of alarm processing based on log information in order to address the above technical problems.

[0005] In a first aspect, the present application provides a log data processing method, comprising:

[0006] Performing feature matching on the information security log to determine the device type and log type of the information security log;

[0007] Performing log parsing processing on the information security log based on the device type and log type of the information security log to obtain key fields of the information security log;

[0008] Performing scenario recognition processing based on key fields of the information security log to determine the threat scenario to which the information security log belongs;

[0009] Merging the information security logs according to the threat scenarios to which the information security logs belong to obtain alarm log information;

[0010] A log alarm message is generated based on the alarm log information.

[0011] In a second aspect, the present application further provides a log data processing device, comprising:

[0012] A feature matching module is used to perform feature matching processing on the information security log to determine the device type and log type of the information security log;

[0013] A log parsing module, configured to perform log parsing processing on the information security log based on the device type and log type of the information security log to obtain key fields of the information security log;

[0014] A scenario recognition module, configured to perform scenario recognition processing based on key fields of the information security log to determine the threat scenario to which the information security log belongs;

[0015] A log classification module is used to merge and process the information security logs according to the threat scenarios to which the information security logs belong, and obtain alarm log information;

[0016] The alarm processing module is used to generate a log alarm message based on the alarm log information.

[0017] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Performing feature matching on the information security log to determine the device type and log type of the information security log;

[0019] Performing log parsing processing on the information security log based on the device type and log type of the information security log to obtain key fields of the information security log;

[0020] Performing scenario recognition processing based on key fields of the information security log to determine the threat scenario to which the information security log belongs;

[0021] Merging the information security logs according to the threat scenarios to which the information security logs belong to obtain alarm log information;

[0022] A log alarm message is generated based on the alarm log information.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0024] Performing feature matching on the information security log to determine the device type and log type of the information security log;

[0025] Performing log parsing processing on the information security log based on the device type and log type of the information security log to obtain key fields of the information security log;

[0026] Performing scenario recognition processing based on key fields of the information security log to determine the threat scenario to which the information security log belongs;

[0027] Merging the information security logs according to the threat scenarios to which the information security logs belong to obtain alarm log information;

[0028] A log alarm message is generated based on the alarm log information.

[0029] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0030] Performing feature matching on the information security log to determine the device type and log type of the information security log;

[0031] Performing log parsing processing on the information security log based on the device type and log type of the information security log to obtain key fields of the information security log;

[0032] Performing scenario recognition processing based on key fields of the information security log to determine the threat scenario to which the information security log belongs;

[0033] Merging the information security logs according to the threat scenarios to which the information security logs belong to obtain alarm log information;

[0034] A log alarm message is generated based on the alarm log information.

[0035] The log data processing method, apparatus, computer device, storage medium, and computer program product described above perform feature matching on information security logs to determine the device type and log type of the information security logs. Log matching then identifies specific information in the logs. Based on the device type and log type of the information security logs, the information security logs are parsed to obtain key fields. Log parsing then extracts key information from the logs. Scenario recognition is then performed based on the key fields of the information security logs to determine the threat scenarios to which the information security logs belong. The logs are then categorized into different scenarios based on the key information in the logs. The information security logs are then merged based on the threat scenarios to obtain alarm log information. Log alarm messages are generated based on the alarm log information. Finally, the logs are merged and alarms are issued based on the threat scenarios to which the logs belong. In this embodiment, the information security log is processed through feature matching and log parsing to extract key field information. Then, a large number of information security logs are used to identify scenarios through the key field information, and the information security logs are merged and processed based on the threat scenarios of the information security logs. This can effectively aggregate a large number of information security logs into alarm log information, thereby efficiently screening out valuable security events and improving the accuracy of alarm processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is an application environment diagram of a log data processing method in one embodiment;

[0038] Figure 2 Schematic diagram of a flow chart of a log data processing method in one embodiment;

[0039] Figure 3 A schematic diagram of a log data access method configuration in one embodiment;

[0040] Figure 4 A schematic diagram of a log parsing configuration page in one embodiment;

[0041] Figure 5 A schematic diagram of JSON format log parsing processing in one embodiment;

[0042] Figure 6 A schematic diagram of classifying various threat scenarios in one embodiment;

[0043] Figure 7 A schematic diagram illustrating a mapping relationship between keywords and threat scenarios in one embodiment;

[0044] Figure 8 A schematic diagram of a log parsing process in one embodiment;

[0045] Figure 9 A schematic diagram of a merging cycle setting page in one embodiment;

[0046] Figure 10 A schematic diagram of the correspondence between log features and alarm identifiers in a threat scenario in one embodiment;

[0047] Figure 11 A schematic diagram of the operation logic for determining a threat scenario based on key fields in one embodiment;

[0048] Figure 12 A diagram showing changes in information security situation in one embodiment;

[0049] Figure 13 1 is a flow chart of the overall process of a log data processing method in one embodiment;

[0050] Figure 14 Schematic diagram of a flow chart of a log data processing method according to another embodiment;

[0051] Figure 15 is a structural block diagram of a log data processing device in one embodiment;

[0052] Figure 16 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] The technical terms involved in this application include:

[0055] Threat scenario: refers to a specific information security incident or attack scenario, including external threats such as external network vulnerability exploitation and internal threats such as host infection with Trojans.

[0056] Information security logs: These are log files that record and store security events and threats. In the information security field, various security devices and systems (such as firewalls, intrusion detection systems, and security information and event management systems) generate large amounts of alert logs, which are used to record possible security issues, abnormal behavior, or potential attacks.

[0057] Alarm noise reduction: This refers to using technical means to reduce a large number of redundant security alarm logs, retaining only a small number of alarms that require attention and processing.

[0058] SIEM / SOC systems: SIEM (Security Information and Event Management) stands for Security Information and Event Management System, and SOC (Security Operations Center) stands for Security Operations Center. These terms are used to describe security management and monitoring systems.

[0059] Real security threats: These are security risks or threats that actually exist and pose a potential risk to systems, networks, or data. These threats may come from various forms, such as malware, hacker attacks, data leaks, and physical intrusions.

[0060] Lossless merging: This refers to merging log data from different systems or devices into a single log or alarm information while preserving the integrity and details of the original logs for more effective alarm analysis and incident response.

[0061] JSON format: JSON (JavaScript Object Notation) is a lightweight data exchange format commonly used for data serialization and transmission. It uses an easy-to-read and easy-to-write text format to represent data in the form of key-value pairs.

[0062] LEEF format: LEEF (Log Event Extended Format) is a standard format for log events.

[0063] CEF format: CEF (Common Event Format) is a standard format for security events.

[0064] XML format: XML (eXtensible Markup Language) is a markup language used to describe and transmit data. It uses extensible tags and elements to organize and represent data.

[0065] The log data processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, multiple security devices 102 communicate with a log processing server 104 via a network, and the log processing server 104 communicates with a terminal 106 via a network. The data storage system can store data that the log processing server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The security device 102 will regularly generate a large number of information security logs. The information security logs can record internal security events and external security events generated by the security device 102 during operation. However, due to the variety and complex format of the security device's information security logs, they are difficult to use for direct analysis. At this time, these security devices 102 can submit the generated information security logs to the log processing server 104, and the log processing server 104 will analyze and process these log data to obtain the alarm event analysis results of the security devices 102. The specific analysis and processing process includes: performing feature matching processing on the information security log to determine the device type and log type of the information security log; performing log parsing processing on the information security log based on the device type and log type of the information security log to obtain the key fields of the information security log; performing scene recognition processing based on the key fields of the information security log to determine the threat scenario to which the information security log belongs; merging the information security logs according to the threat scenario to which the information security log belongs to obtain alarm log information; and generating log alarm messages based on the alarm log information. The terminal 106 is used to manage the log processing server 104. For example, after generating a log alarm message, the administrator of the terminal 106 can send the log alarm message to the security device server or directly manage the security device 102 based on the log alarm message generated by the log processing server 104. Terminal 106 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Security device 102 may be a software device or software system running on a host server, such as a firewall, intrusion detection system, or security information and event management system. Log processing server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0066] In an exemplary embodiment, Figure 2 As shown, a log data processing method is provided, which is applied to Figure 1 Taking the log processing server 104 in FIG. 1 as an example, the following steps 201 to 207 are included. Among them:

[0067] Step 201 : performing feature matching processing on the information security log to determine the device type and log type of the information security log.

[0068] Information security logs refer to log files that record and store security events and threats. In the field of information security, various security devices and systems (such as firewalls, intrusion detection systems, and security information and event management systems) generate a large number of alarm logs, which are used to record possible security issues, abnormal behavior, or potential attack activities. In this embodiment, information security logs refer to alarm log information generated by the operation of security device 102, which is used to record possible security issues, abnormal behavior, or potential attack activities of security device 102. Feature matching processing refers to identifying the device type and log type of information security logs based on the features of the alarm logs. In a specific embodiment, a signature feature database can be constructed based on the signature features of alarm logs of different device types or log types. Then, when receiving information security logs that need to be processed, the pre-built signature feature database can be used to match the device type and log type of the information security logs. Regarding the device type and log type of information security logs, different vendors' security systems typically include vendor and device type tags when outputting logs. For example, Company A's firewall system logs (syslogs) are tagged with product model, Company B's traffic device logs are tagged with product name, and Company C's operations and maintenance audit system is tagged with device tags. These tags can be used to identify the device type and log type of information security logs.

[0069] For example, during operation, the security device 102 generates a large number of alarm logs, thereby recording various security issues, abnormal behaviors, or potential attack activities that may exist in the security device 102. For an information security system, it includes multiple different security devices 102, which can generate tens of millions of information security logs every day. These information security logs are of many types and have complex formats. It is impossible to accurately and efficiently analyze these information security logs. In this case, the logs generated by the different security devices 102 on the system can be submitted as information security logs to the log processing server 104. The log processing server 104 will classify and reduce the noise of these information security logs, losslessly merge logs under the same type of threat scenario, and reduce the noise of the system's alarm volume. The log processing server 104 can set a merging cycle and collect the information security logs generated by the security devices 102 at different points within the merging cycle for processing. After obtaining the information security log, the log processing server 104 first needs to perform feature matching processing on the information security log. Specifically, it can extract the marker features present in the information security log, and then determine the basic log information such as the device type and log type of the information security log based on the extracted marker features, thereby obtaining the basic data for log parsing and processing. In one embodiment, the present application is applicable to processing the information security log generated by the information security system. After the information security system generates the log, it can send the generated information security log to the log processing server 104 through syslog, transmission control protocol (TCP), user datagram protocol (UDP), data acquisition components (such as beats of elastic stack), etc., or actively write to databases such as mysql and kafka, and then the log processing server 104 reads the information security log from the database. The present application can collect security device logs within the organization through a variety of access methods, through passive reception or active acquisition. The specific access method setting diagram can refer to Figure 3 shown.

[0070] Step 203 : performing log parsing processing on the information security log based on the device type and log type of the information security log to obtain key fields of the information security log.

[0071] Log parsing refers to the process of extracting the required information from information security logs based on the device and log type. Key fields in information security logs represent key information within the logs. Key fields may include occurrence time, name, category, outcome, attack stage, and host address (host_ip).

[0072] For example, after determining the device type and log type of the information security log, the parsing method of the information security log can be determined, and the key fields of the information security log can be obtained by performing log parsing processing on the information security log using the determined parsing method. The key fields of the information security log can then be used to perform lossless merging of subsequent logs. In a specific embodiment, different log parsing strategies can be preset for information security logs of different device types and log types. After determining the device type and log type, the parsing process of the log can be completed by searching for the corresponding log parsing strategy in the policy library. For information security logs that cannot be parsed by the log parsing strategy, they can also be parsed by regular expressions and other methods to extract the corresponding data. Figure 4 The log parsing configuration page shown in the figure can configure different parsing strategies for different types of logs. Then, based on the parsing strategy, different methods are used to complete the log parsing of the information security log and extract the corresponding key fields, such as Figure 5 As shown, key fields can be extracted from it by parsing JSON.

[0073] Step 205 : Perform scenario recognition processing based on the key fields of the information security log to determine the threat scenario to which the information security log belongs.

[0074] Among them, scenario identification refers to determining the threat scenario of the information security log. The threat scenario is a specific information security event or attack scenario, including external threat categories such as external network vulnerability exploitation, internal threat categories such as host infection with Trojans, etc. Different threat scenarios correspond to security threats. Through scenario identification, information security logs can be classified into different threat scenarios, thereby achieving log noise reduction processing. In one of the embodiments, the applicant summarized the three major categories / 18 subcategories of security threats commonly faced by current information security systems. If these threats can be identified and controlled, the security issues of the information security system can be basically guaranteed. These three major categories specifically include external threats, intranet threats, and internal violations, and the 18 subcategories under the three major categories and the English names of the subcategories and the specific threat scenario expressions can be referred to. Figure 6 shown.

[0075] For example, after obtaining the key fields of the information security log, scene recognition processing can be performed based on the obtained keywords, thereby dividing the information security log into different threat scenarios, that is, determining the threat scenario to which the information security log belongs. Regarding the specific process of scene division, keywords that can represent the scene type can be first extracted from the key fields, and then scene matching processing can be performed based on these keywords that represent the scene type to identify the threat scenario to which each information security log belongs. In a specific embodiment, if Figure 7As shown in the figure, you can create corresponding scenario matching strategies for different types of information security logs, and set the mapping between keywords and threat scenarios in the scenario matching strategies. Figure 8 After the log parsing process shown above extracts the corresponding key fields, scenario identification can be implemented through scenario matching strategies to determine the threat scenario to which each information security log belongs.

[0076] Step 207 : The information security logs are merged according to the threat scenarios to which the information security logs belong, to obtain alarm log information.

[0077] Step 209: Generate a log alarm message based on the alarm log information.

[0078] Merging refers to the process of combining information recorded in information security logs belonging to the same threat scenario. By merging information security logs, multiple information security logs can be combined into one, thereby achieving noise reduction and reducing the number of redundant information security logs. The resulting alarm log information is the log information retained after merging the information security logs, which only retains a small number of alarms that require attention and processing. Log alarm messages can then be generated based on the alarm log information to implement log alarm processing.

[0079] For example, after determining the threat scenario to which each information security log belongs, log merging can be performed based on the threat scenario to achieve noise reduction. For the log merging process, information security logs with the same threat scenario can be directly merged to obtain an alarm log message, or information security logs with the same threat scenario and the same host address can be merged to obtain an alarm log message. The specific merging method can be set according to the needs of information security analysis. For example, for threat scenarios such as malware connection, weak passwords on the intranet, successful execution of malware, abnormal user accounts, illegal software, illegal account operations, and data leakage, the information security logs can be directly merged. For threat scenarios such as successful external network attacks, high-risk attacks launched on the external network, weak passwords on the external network, honeypots capturing attackers, asset loss, asset abnormalities, and lateral penetration, a merging method with the same threat scenario and the same host address can be adopted to merge information security logs with the same host address and the same threat scenario. After the log merging process is completed, a log alarm message can be generated based on the merged logs to perform alarm processing.

[0080] The log data processing method performs feature matching on information security logs to determine the device type and log type of the information security logs. Log matching identifies specific information in the logs. Based on the device type and log type of the information security logs, the logs are parsed to obtain key fields. Log parsing extracts key information from the logs. Scenario recognition is then performed based on the key fields of the information security logs to determine the threat scenarios to which the information security logs belong. The logs are then categorized into different scenarios based on the key information in the logs. The information security logs are then merged based on the threat scenarios to obtain alarm log information. Log alarm messages are generated based on the alarm log information. Finally, the logs are merged and alarms are issued based on the threat scenarios to which the logs belong. In this embodiment, the information security log is processed through feature matching and log parsing to extract key field information. Then, a large number of information security logs are used to identify scenarios through the key field information, and the information security logs are merged and processed based on the threat scenarios of the information security logs. This can effectively aggregate a large number of information security logs into alarm log information, thereby efficiently screening out valuable security events and improving the accuracy of alarm processing.

[0081] In an exemplary embodiment, step 201 includes: performing feature matching processing on the information security log through a preset tag feature database to determine the manufacturer tag features and product tag features contained in the information security log; based on the manufacturer tag features and product tag features contained in the information security log, identifying the device type and log type of the information security log.

[0082] The pre-set signature database is a database used to store signatures corresponding to different log types. The vendor signature and product signature represent the signatures set by the vendor for the security system and the product signature set by the vendor for the security system, respectively. These two signatures can be used to determine the vendor or product associated with an information security log, thereby identifying the corresponding device type and log type.

[0083] For example, the applicant discovered that security systems from different manufacturers typically carry manufacturer and device type tags when outputting logs. Therefore, by collecting these tags, a corresponding preset tag feature database is established, and a corresponding relationship between the tag features, device types, and log types is established. Subsequently, the preset tag feature database can be used to compare and determine the device type and log type of the input information security log. Once the information security log is obtained, the features in the preset tag feature database can be compared with the log content in the information security log to determine the tag feature types contained in the information security log, specifically including manufacturer tag features and product tag features. For example, if the product model tag feature a is read from the information security log, it can be determined that the information security log belongs to the firewall system log of Company A. If the product name tag feature b is read from the information security log, it can be determined that the information security log belongs to the traffic device log of Company B. If the device tag feature c is read from the information security log, it can be determined that the information security log belongs to the operation and maintenance upgrade log output by the operation and maintenance audit system of Company C. In this embodiment, feature matching processing is performed on the information security log using the preset tag feature database, thereby accurately identifying the device type and log type in heterogeneous information security logs, which can effectively improve the efficiency and accuracy of log processing.

[0084] In an exemplary embodiment, step 203 includes: determining the log format of the information security log based on the device type and log type of the information security log; and performing log parsing processing on the information security log according to the log format of the information security log to obtain key fields of the information security log.

[0085] Among them, the log format refers to the data format of the information security log. The logs generated by the security system usually use the standard syslog format, JSON (JavaScript Object Notation) format, KV (key-value) segmentation format, XML (Extensible Markup Language) format, CEF (Common Event Format) format, LEEF (Log Event Extended Format) format, etc.

[0086] For example, after determining the device type and log type of the information security log through feature matching, the data format of the log data under the device type and log type can be determined at the same time. These formats have obvious characteristics, so log parsing is performed according to different format characteristics to obtain the key fields of the information security log. For example, for information security logs in formats such as JSON, delimiter, XML, CEF, and LEEF, each has a corresponding universal log data parsing method. Therefore, these information security logs belong to standard log formats. Log parsing is performed on the information security log according to the format characteristics of the log format to which the information security log belongs, and the key fields of the information security log are obtained, such as Figure 5 As shown, for information security logs in JSON format, log parsing can be performed directly to obtain key fields. If the information security log format is non-standard, regular expressions can be used to parse the log to obtain key fields. A regular expression, also known as a regular expression, is a text pattern commonly represented by `regex`, `regexp`, or `RE` in code. It uses a single string to describe and match a series of strings that conform to specific syntactic rules. Regular expressions can perform string extraction. Therefore, the solution of this application utilizes regular expressions to isolate interesting sections from information security logs in non-standard formats. These sections serve as key fields for log parsing of the information security logs. Key fields such as occurrence time, name, category, result, attack stage, and host address (host_ip) are extracted for subsequent log classification and noise reduction. In this embodiment, log data is parsed based on the log format of the information security log, accurately separating key fields from the text content of the information security log, thereby ensuring the efficiency and accuracy of subsequent log data processing.

[0087] In one embodiment, step 205 includes: identifying a target field within a key field; and performing scenario recognition processing based on the target field to determine the threat scenario to which the information security log belongs.

[0088] The target field refers to one or more sets of field data in the key fields pre-specified according to the scenario recognition rules. The purpose of scenario recognition processing is to determine the threat scenario to which the information security log belongs based on the correspondence between the target field and the threat scenario.

[0089] For example, in the process of scene recognition processing, the applicant found that the content of some fields in the log content is closely related to the threat scene to which the information security log belongs. Therefore, a mapping relationship between key fields and threat scenes can be pre-built, and when scene recognition is required, matching can be performed based on these mapping relationships to determine the threat scene to which the information security log belongs. For the target field used, the type field in the key field of the log can be specifically selected. This field represents the type of log to which the information security log belongs. Therefore, when the key field contains the type field, the type field can be determined as the target field, and then the threat scene to which the information security log belongs can be matched based on the type field. In the case where the key field of the information security log does not contain the type field, the name of the log can also represent the type information to which the log belongs. Therefore, the name field in the key field can be identified and the name field can be determined as the target field. In another embodiment, the key field does not include the type field or the name field. In this case, the "message" field can be selected as the target field. The "message" field represents the specific log content of the information security log. The final "message" field can be extracted by processing the original log information. For example, the information security log "Sep 15 12:41:19 Linux_test192.168.227.130 [login][failure]XXXfrom IP 204.58.X6.247: Invalid username or password!" generated under the Linux system can be directly removed from the header of this part of the information content, and then "[login][failure]XXX from IP 204.58.X6.247: Invalidusername or password!" can be taken, and regular matching can be performed on this large section of log information, and finally the feature "Invalid .* password" can be found as the final "message" field. In this embodiment, by identifying the target field and then performing scene recognition processing based on the target field, the threat scene is classified, which can effectively improve the efficiency and accuracy of threat scene classification.

[0090] As for the process of scenario identification processing for the target field, the mapping relationship table of the target field can be searched based on the device type and log type of the information security log; the target field can be matched through the mapping relationship table to obtain the field matching result; and the threat scenario to which the information security log belongs can be determined based on the field matching result.

[0091] For example, the mapping relationship table is generated by the applicant based on information security logs of different device types and log types, and the type field, name field and other key fields corresponding to various threat scenarios are pre-summarized. When it is necessary to determine the threat scenario, the field matching results can be obtained based on the mapping relationship table, and the threat scenario can be determined based on the field matching results. In one embodiment, Figure 7 As shown, scenario matching policies can be created for different types of information security logs. Within these policies, mappings between keywords and threat scenarios can be set. For example, " / ConfigRisk / (S / D).*" is a key field extracted using a regular expression, corresponding to the threat scenario "improper configuration." Meanwhile, " / ConfigRisk / Database" is a key field obtained through an exact match of the type or name fields, corresponding to the threat scenario "unauthorized access." In this embodiment, by pre-establishing a mapping table between threat scenarios and target fields and then matching target fields based on this mapping table for scenario identification, the efficiency and accuracy of threat scenario classification can be effectively improved.

[0092] In one embodiment, step 207 includes screening the information security log according to the threat scenario to which the information security log belongs to obtain the information security log of the target threat scenario; merging the information security log according to the scenario type of the target threat scenario corresponding to the information security log to obtain alarm log information.

[0093] Among them, screening and processing the information security logs according to the threat scenarios to which they belong refers to the screening and processing of the logs before merging them. Some information security logs that do not need to be processed can be eliminated, and the information security logs can also be preliminarily classified and processed. The target threat scenario refers to various pre-specified important threat scenarios that can pose a major threat to information security. Regarding the definition of important threat scenarios, based on the applicant's many years of experience and summary in information security, the security threat scenarios that are of most concern today can be divided into three categories: external threat, internal threat, and violation. Among them, external threat scenarios include: successful external network attacks, high-risk attacks launched from the external network, weak external network passwords, malware connections, honeypots capturing attacker information, etc.; internal threat scenarios include: loss of internal network assets, abnormal internal network assets, successfully executed malware, lateral penetration, abnormal user accounts, etc.; violation scenarios include: assets installing illegal software, accounts performing illegal operations, and accounts experiencing data leaks, etc. Target threat scenarios can be divided into host address-related scenarios and non-host address-related scenarios. In the host address-related scenario, the threat scenario is associated with the host IP address that generated the information security log, while in the non-host address-related scenario, the threat scenario is not associated with the host IP address that generated the information security log. Specifically, host address-related scenarios include the aforementioned threat scenarios such as successful external network attacks, high-risk attacks launched from the external network, weak external network passwords, attackers captured in honeypots, asset compromise, asset anomalies, and lateral movement. Non-host address-related scenarios include threat scenarios such as malware connections, weak internal network passwords, successful malware execution, user account anomalies, illegal software, illegal account operations, and data leaks.

[0094] For example, some threats are unimportant and require no attention, while others are closely related to the environment, such as being associated with host addresses. Therefore, information security logs can be first filtered based on the threat scenario to obtain information security logs for the target threat scenario. Then, based on the scenario type of the target threat scenario to which the information security logs correspond, the information security logs are merged to obtain alarm log information. Scenario types can be categorized as host address-related scenarios and non-host address-related scenarios. For information security logs associated with host addresses, a "threat scenario and host address-based merging method" is used to categorize the information security logs and generate alarm log information corresponding to the scenario type of the target threat scenario. This method merges all logs generated by a host address within the threat scenario into a single alarm. For information security logs associated with non-host address-related scenarios, a "threat scenario-based direct merging method" is used to categorize the information security logs and generate alarm log information corresponding to the scenario type of the target threat scenario. This method directly merges all logs within that threat scenario into a single alarm. In this embodiment, by first screening the information security logs and then merging the information security logs according to the scenario type of the target threat scenario to which the screening results belong, different types of information security logs can be effectively merged, thereby improving the efficiency and accuracy of the alarm log merging processing.

[0095] In an exemplary embodiment, step 209 includes: obtaining the log alarm message generated within the merging period; in a case where the generated log alarm message includes the log alarm message of the target threat scenario, updating the message content of the generated log alarm message according to the alarm log information to obtain an updated log alarm message; in a case where the generated log alarm message does not include the log alarm message of the target threat scenario, performing message generation processing according to the alarm log information to obtain the log alarm message of the target threat scenario.

[0096] Among them, the merging cycle refers to the generation cycle of alarm messages. All information security logs generated in the current merging cycle can be merged and processed in the current merging cycle, and the information security logs generated in the next merging cycle will be merged in the next merging cycle.

[0097] Exemplarily, for the generated log alarm messages, after the information security logs are merged and processed to obtain the alarm log information, each target threat scenario corresponds to a log alarm message. If a log alarm message for the target threat scenario already exists, it means that the alarm processing has been performed on the target threat scenario. At this time, the generated log alarm message can be directly updated with the message content according to the alarm log information to obtain the updated log alarm message. The update can specifically be to add the log identifier in the alarm log information to the generated log alarm message to establish the association relationship between the original information security log and the log alarm message. In the case that the generated log alarm message does not contain the log alarm message for the target threat scenario, it means that a new log alarm message needs to be generated. At this time, the message generation processing can be performed according to the alarm log information to obtain the log alarm message for the target threat scenario, and the association relationship between the original information security log and the log alarm message can be established in the log alarm message. In one embodiment, if Figure 9 As shown, in the "asset compromise" scenario, if the merging period is set to 24 hours, the method of this application will merge all asset compromise logs under a single host address (host_ip) into a single alert within 24 hours. In this embodiment, log alert message generation and processing are performed by comparing already generated log alert messages, effectively avoiding duplicate alerts, ensuring the accuracy of alert processing based on message security logs, and effectively reducing the number of alerts.

[0098] In one embodiment, step 209 includes: performing feature extraction processing on the information security log according to the scenario type of the target threat scenario corresponding to the information security log to obtain log tuple information; searching for log tuple information based on the log identifier in the alarm log information to obtain feature information and alarm identifier of each log in the alarm log information; and generating a log alarm message based on the log identifier, feature information and alarm identifier.

[0099] Information security logs often record multiple forms of information for the same threat scenario. Therefore, to effectively record log case information, after determining the target threat scenario to which the information security log corresponds, feature extraction can be performed on the information security log based on the scenario type of the target threat scenario to obtain log tuple information. The log tuple information includes the feature information obtained through feature extraction and the alarm identifier corresponding to the feature.

[0100] For example, the log alarm message specifically includes the log identifier, feature information and alarm identifier of the information security log. For each information security log, in order to more effectively record the key log information recorded in the information security log, while classifying these information security logs, it is also possible to perform feature extraction processing on different types of information security logs based on the scenario type of the target threat scenario corresponding to the information security log, extract the key feature information recorded in the information security log, and determine the alarm identifier corresponding to the information security log in the threat scenario. In one embodiment, for the threat scenario of "asset loss", the common manifestations of its log features are as follows: Figure 10 As shown in the figure, it includes two sub-scenarios and different performance characteristics under the scenarios. Different characteristics also correspond to their own alarm rule numbers as alarm identifiers. In addition, for Figure 10 The summarized log feature representation can also be used in the target threat scenario identification process, and is suitable for determining threat scenarios by matching target fields in the mapping relationship table. Specifically, operations can be performed on the fields in the information security log to find logs that meet the threat scenario. Figure 11 The operation logic for the "lateral penetration" scenario is to match the "attack type and attack direction" in the log to identify the intranet-to-intranet attack log and locate it as a "lateral penetration" scenario. As for the generation process of the log alarm message, after the information security log is merged and processed to obtain the alarm log information, the corresponding feature information and alarm identifier can be searched for each log identifier in the alarm log information, and the association relationship between the log identifier, feature information and alarm identifier can be established to generate the log alarm message. In this embodiment, the feature information and alarm identifier of the information security log are obtained through feature extraction processing, thereby generating the log alarm message, which can effectively ensure the information richness of the log alarm message, thereby effectively ensuring the accuracy and efficiency of the processing based on the log alarm message.

[0101] In an exemplary embodiment, the method also includes: obtaining historical log alarm messages; generating an information security situation change graph corresponding to the information security log based on the threat scenario and merging period to which the log alarm messages in the historical log alarm messages belong; and feeding back the information security situation change graph.

[0102] For example, after generating a log alarm message, in order to more effectively summarize and analyze the security situation faced by the system, historical log alarm messages can be obtained; and based on the threat scenarios and merging cycles to which the log alarm messages in the historical log alarm messages belong, an information security situation change graph corresponding to the information security log is generated. The information security situation change graph is used to reflect the changes in the amount of alarms in different merging cycles, as well as the changes in the number of threat scenarios, and at the same time, the main threat scenarios can be listed in the information security situation change graph to help security operation and maintenance personnel more efficiently and intuitively determine the security threats faced by the system. In one embodiment, the information security situation change graph can refer to Figure 12 As shown, the upper left corner shows the total number of received logs, approximately 244.9 million. After automatic noise reduction through log data processing, 222 alert messages were generated, corresponding to 149 security incidents. The wave chart shows the changes in log volume and alert volume over different periods. In this embodiment, by generating an information security situation change chart, we can effectively and intuitively display the various threat scenarios faced by the security system based on the log data merging and processing, thereby ensuring the efficiency and accuracy of information security log processing.

[0103] This application also provides an application scenario, which applies the above-mentioned log data processing method. Specifically, the application of the log data processing method in this application scenario is as follows:

[0104] During the operation of a certain city's subway, the number of information security logs generated every day is 40 million. After analysis using existing log analysis methods, the number of alarms generated is 10,000. There is a large number of security alarms, and security alarms are difficult to effectively screen and process. At this time, the log data processing method of this application can be used to perform alarm "noise reduction" processing on the information security logs of the subway system, thereby losslessly merging logs based on threat types, reducing the number of output alarms, and ensuring the efficiency and accuracy of the log analysis and processing process.

[0105] The overall flow chart of log data processing in this application can be referred to Figure 13As shown, before performing log analysis, the system will analyze the various threat scenarios faced by the subway system, thereby establishing corresponding key threat scenarios based on the security issues currently facing the subway system. As for the actual log data processing process, after the subway's security system generates an information security log, it will send the information security log to a security management and monitoring system (SIEM / SOC system) equipped with the log data processing method of this application, thereby performing feature matching processing on the information security log to determine the device type and log type of the information security log. Specifically, feature matching processing can be performed on the information security log using a preset tag feature database to determine the vendor tag features and product tag features contained in the information security log; based on the vendor tag features and product tag features contained in the information security log, the device type and log type of the information security log can be identified. After determining the device type and log type, the log format of the information security log can be determined based on the device type and log type of the information security log; then, it is determined whether the log format belongs to the standard log format. If the log format of the information security log is the standard log format, the information security log is parsed according to the format characteristics of the log format to which the information security log belongs (JSON / delimiter / CEF) to obtain the key fields of the information security log; if the log format of the information security log is a non-standard log format, the information security log is parsed using regular expressions, and the key fields of the information security log are split as tuples, and the tuples are stored in JSON format.

[0106] After obtaining the key fields, it is determined whether the key fields contain the type field. If so, the type field is identified as the target field. If not, the name field or the "message" field within the key fields is identified and the name field is identified as the target field. The target field is then analyzed by combining it with pre-set key threat scenarios. Specifically, the mapping table for the target field is searched based on the device type and log type of the information security log. The target field is matched against the mapping table to obtain a field matching result. Based on the field matching result, the threat scenario to which the information security log belongs is determined. Based on the scenario type of the target threat scenario corresponding to the information security log, feature extraction is performed on the information security log to obtain log tuple information, and the threat scenario is written into the determined log tuple information. The alert generation process first determines whether the threat scenario corresponding to the log is a critical threat scenario. If not, the log is ignored without generating an alert. If it is a critical threat scenario, it is determined whether the log is associated with the host address (IP). If not, the logs can be directly merged based on the threat scenario type. When associating, it is necessary to combine the "threat scenario + IP" for merging. After the merging is completed, the log alarm messages generated during the merging cycle can be obtained; when the generated log alarm message contains the log alarm message of the target threat scenario, the message content of the generated log alarm message is updated according to the alarm log information to obtain the updated log alarm message; when the generated log alarm message does not contain the log alarm message of the target threat scenario, the message generation is processed according to the alarm log information to obtain the log alarm message of the target threat scenario. When generating an alarm, the management of the alarm message and the generated log tuple information can be established to facilitate the staff to obtain the corresponding feature information and alarm identifier based on the alarm to perform alarm analysis.

[0107] In one embodiment, the flowchart of the log data processing method of the present application can be referred to Figure 14 As shown, including:

[0108] Step 1401 : performing feature matching processing on the information security log through a preset tag feature database to determine the manufacturer tag feature and product tag feature contained in the information security log.

[0109] By pre-building a tag feature database, and then using the tag feature database to perform an initial analysis of the information security log, the manufacturer tag features and product tag features contained in the information security log are extracted, thereby effectively improving the efficiency of log data analysis.

[0110] Step 1403: Identify the device type and log type of the information security log based on the manufacturer identification feature and product identification feature contained in the information security log.

[0111] After obtaining the manufacturer tag feature and the product tag feature, the device type and log type of the information security log can be identified through the correspondence between the tag feature and the device type and log type.

[0112] Step 1405: Determine the log format of the information security log based on the device type and log type of the information security log.

[0113] By determining the device type and log type of the information security log, we can determine whether the log format of the information security log belongs to JSON / delimited / XML / CEF and other formats, thereby determining the subsequent key field extraction method.

[0114] Step 1407 : performing log parsing processing on the information security log according to the log format of the information security log to obtain key fields of the information security log.

[0115] For different log formats, corresponding log parsing methods need to be used to perform log parsing. For logs in standard formats such as JSON / delimited / XML / CEF, the log parsing method corresponding to the format can be used to extract key fields. For logs in non-standard formats, methods such as regular expressions can be used to extract the required key fields.

[0116] Step 1409: Identify the target field within the key field.

[0117] After obtaining the key fields, target fields required for threat scenario identification, such as type fields or name fields, may be obtained from the key fields as target fields.

[0118] Step 1411: Search the mapping relationship table of the target field based on the device type and log type of the information security log.

[0119] Information security logs of different device types and log types correspond to different mapping relationship tables. The mapping relationship table can be determined based on the device type and log type of the information security log, thereby performing threat scenario identification and processing.

[0120] Step 1413: Match the target field through the mapping relationship table to obtain a field matching result.

[0121] After obtaining the mapping relationship table, the field matching result can be obtained through the matching relationship between the fields in the mapping relationship table, so that subsequent scene recognition processing can be performed based on the field matching result.

[0122] Step 1415: Determine the threat scenario to which the information security log belongs based on the field matching result.

[0123] After the field matching process is completed, the threat scenario identification process can be directly performed based on the result of the field matching process.

[0124] Step 1417 , screening the information security log according to the threat scenario to which the information security log belongs, and obtaining the information security log of the target threat scenario.

[0125] After identifying the threat scenario, the information security log can be further screened to determine whether it belongs to the corresponding target threat scenario. If the threat scenario corresponding to the information security log is negligible, it can be ignored without processing. For information security logs that belong to the target threat scenario, the corresponding scenario field can be added.

[0126] Step 1419 , merging the information security logs according to the scenario type of the target threat scenario corresponding to the information security logs to obtain alarm log information.

[0127] After determining the scenario type of the target threat scenario corresponding to the information security log, the information security logs of the same type may be merged, for example, the log identifiers of the information security logs may be merged together to generate alarm log information corresponding to the scenario type.

[0128] Step 1421 : Perform feature extraction processing on the information security log according to the scenario type of the target threat scenario corresponding to the information security log to obtain log tuple information.

[0129] At the same time, after the scenario type of the target threat scenario corresponding to the information security log is determined, feature extraction processing can be performed on the information security log to obtain log tuple information containing feature information and alarm identification, thereby effectively recording the specific information of the log.

[0130] Step 1423 : searching for log tuple information based on the log identifier in the alarm log information to obtain characteristic information and alarm identifier of each log in the alarm log information.

[0131] After the alarm log information is generated, the corresponding feature information and alarm identifier can be searched for each log in the alarm log information, thereby effectively providing the security operation and maintenance personnel with the features of each alarm log based on the alarm log information.

[0132] Step 1425: Generate a log alarm message based on the log identifier, characteristic information, and alarm identifier.

[0133] Finally, the log identifier, characteristic information and alarm identifier are integrated to generate log alarm information. The generated log alarm information can be used to effectively process the alarms of information security logs, while effectively reducing the number of generated alarms and ensuring the accuracy of log alarms.

[0134] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0135] Based on the same inventive concept, the embodiments of the present application also provide a log data processing device for implementing the log data processing method involved above. The implementation solution provided by this device is similar to the implementation solution described in the above method, so the specific limitations of one or more log data processing device embodiments provided below can be found in the above limitations of the log data processing method and will not be repeated here.

[0136] In an exemplary embodiment, Figure 15 As shown, a log data processing device is provided, comprising:

[0137] The feature matching module 1502 is used to perform feature matching processing on the information security log to determine the device type and log type of the information security log.

[0138] The log parsing module 1504 is configured to perform log parsing processing on the information security log based on the device type and log type of the information security log to obtain key fields of the information security log.

[0139] The scenario identification module 1506 is configured to perform scenario identification processing based on key fields of the information security log to determine the threat scenario to which the information security log belongs.

[0140] The log classification module 1508 is used to merge the information security logs according to the threat scenarios to which the information security logs belong, and obtain alarm log information.

[0141] The alarm processing module 1510 is configured to generate a log alarm message based on the alarm log information.

[0142] In one embodiment, the feature matching module 1502 is specifically used to: perform feature matching processing on the information security log through a preset tag feature database to determine the manufacturer tag features and product tag features contained in the information security log; based on the manufacturer tag features and product tag features contained in the information security log, identify the device type and log type of the information security log.

[0143] In one embodiment, the log parsing module 1504 is specifically used to: determine the log format of the information security log based on the device type and log type of the information security log; and perform log parsing processing on the information security log according to the log format of the information security log to obtain key fields of the information security log.

[0144] In one embodiment, the log format includes a standard log format and a non-standard log format, and the log parsing module 1504 is further used to: when the log format of the information security log is a standard log format, perform log parsing processing on the information security log according to the format characteristics of the log format to which the information security log belongs to obtain the key fields of the information security log; when the log format of the information security log is a non-standard log format, perform log parsing processing on the information security log through regular expressions to obtain the key fields of the information security log.

[0145] In one embodiment, the scenario identification module 1506 is specifically configured to: identify a target field within a key field; and perform scenario identification processing based on the target field to determine the threat scenario to which the information security log belongs.

[0146] In one embodiment, the scene recognition module 1506 is further used to: when the key field includes the type field, determine the type field as the target field; when the key field does not include the type field, identify the name field in the key field and determine the name field as the target field.

[0147] In one embodiment, the scenario identification module 1506 is also used to: search the mapping relationship table of the target field based on the device type and log type of the information security log; match the target field through the mapping relationship table to obtain a field matching result; and determine the threat scenario to which the information security log belongs based on the field matching result.

[0148] In one embodiment, the log classification module 1508 is specifically used to: filter and process the information security log according to the threat scenario to which the information security log belongs, and obtain the information security log of the target threat scenario; merge and process the information security log according to the scenario type of the target threat scenario corresponding to the information security log, and obtain alarm log information.

[0149] In one embodiment, the log classification module 1508 is also used to: when it is determined that the information security log is a host address associated scenario based on the scenario type of the target threat scenario corresponding to the information security log, classify the information security log according to the source host address of the information security log and the scenario type of the target threat scenario, and generate alarm log information corresponding to the scenario type of the target threat scenario; when it is determined that the information security log is a non-host address associated scenario based on the scenario type of the target threat scenario corresponding to the information security log, classify the information security log according to the scenario type of the target threat scenario, and generate alarm log information corresponding to the scenario type of the target threat scenario.

[0150] In one embodiment, the alarm processing module 1510 is specifically used to: obtain the log alarm message generated within the merging cycle; when the generated log alarm message contains the log alarm message of the target threat scenario, update the message content of the generated log alarm message according to the alarm log information to obtain an updated log alarm message; when the generated log alarm message does not contain the log alarm message of the target threat scenario, perform message generation processing according to the alarm log information to obtain the log alarm message of the target threat scenario.

[0151] In one embodiment, the alarm processing module 1510 is specifically used to: perform feature extraction processing on the information security log according to the scenario type of the target threat scenario corresponding to the information security log to obtain log tuple information; search for log tuple information based on the log identifier in the alarm log information to obtain feature information and alarm identifier of each log in the alarm log information; generate a log alarm message based on the log identifier, feature information and alarm identifier.

[0152] In one embodiment, a change graph generation module is also included, which is used to: obtain historical log alarm messages; generate an information security situation change graph corresponding to the information security log based on the threat scenario and merging period to which the log alarm messages in the historical log alarm messages belong; and feedback the information security situation change graph.

[0153] Each module in the above-mentioned log data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0154] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 16As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data related to log data processing. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a log data processing method is implemented.

[0155] Those skilled in the art will understand that Figure 16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0156] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0157] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0158] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0160] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0161] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A log data processing method, characterized in that: The method comprises: Performing feature matching on the information security log to determine the device type and log type of the information security log; Performing log parsing processing on the information security log based on the device type and log type of the information security log to obtain key fields of the information security log; Performing scenario recognition processing based on key fields of the information security log to determine the threat scenario to which the information security log belongs; Merging the information security logs according to the threat scenarios to which the information security logs belong to obtain alarm log information; A log alarm message is generated based on the alarm log information.

2. The method according to claim 1, characterized in that The performing feature matching processing on the information security log to determine the device type and log type of the information security log includes: Perform feature matching processing on the information security log using a preset tag feature database to determine the manufacturer tag feature and product tag feature contained in the information security log; Based on the manufacturer identification feature and product identification feature contained in the information security log, the device type and log type of the information security log are identified.

3. The method according to claim 1, characterized in that The information security log is parsed based on the device type and log type of the information security log to obtain key fields of the information security log including: Determining a log format of the information security log based on a device type and a log type of the information security log; The information security log is parsed according to the log format of the information security log to obtain key fields of the information security log.

4. The method according to claim 3, characterized in that The log format includes a standard log format and a non-standard log format; The information security log is parsed according to the log format of the information security log to obtain key fields of the information security log including: When the log format of the information security log is a standard log format, performing log parsing processing on the information security log according to the format characteristics of the log format to which the information security log belongs to obtain key fields of the information security log; In the case that the log format of the information security log is a non-standard log format, log parsing processing is performed on the information security log through regular expressions to obtain key fields of the information security log.

5. The method according to claim 1, wherein The performing scenario identification processing based on the key fields of the information security log to determine the threat scenario to which the information security log belongs includes: identifying a target field within the key field; Scenario recognition processing is performed based on the target field to determine the threat scenario to which the information security log belongs.

6. The method according to claim 5, characterized in that The identifying the target field within the key field comprises: In a case where the key field includes a type field, determining the type field as a target field; In a case where the key field does not include a type field, a name field in the key field is identified and the name field is determined as a target field.

7. The method according to claim 5, characterized in that The performing scenario identification processing based on the target field to determine the threat scenario to which the information security log belongs includes: Searching a mapping relationship table for the target field based on the device type and log type of the information security log; Match the target field through the mapping relationship table to obtain a field matching result; The threat scenario to which the information security log belongs is determined according to the field matching result.

8. The method according to claim 1, characterized in that The merging and processing of the information security logs according to the threat scenarios to which the information security logs belong to obtain the alarm log information includes: Filtering the information security log according to the threat scenario to which the information security log belongs to obtain an information security log of a target threat scenario; The information security logs are merged and processed according to the scenario type of the target threat scenario corresponding to the information security log to obtain alarm log information.

9. The method according to claim 8, characterized in that The merging of the information security logs according to the scenario type of the target threat scenario corresponding to the information security log to obtain the alarm log information includes: In a case where it is determined that the information security log is a host address associated scenario based on the scenario type of the target threat scenario corresponding to the information security log, the information security log is classified according to the source host address of the information security log and the scenario type of the target threat scenario, and alarm log information corresponding to the scenario type of the target threat scenario is generated; When it is determined that the information security log is a non-host address associated scenario based on the scenario type of the target threat scenario corresponding to the information security log, the information security log is classified according to the scenario type of the target threat scenario to generate alarm log information corresponding to the scenario type of the target threat scenario.

10. The method according to claim 9, characterized in that Generating a log alarm message based on the alarm log information includes: Get the log warning messages generated during the merge cycle; In a case where the generated log warning message includes the log warning message of the target threat scenario, performing message content updating processing on the generated log warning message according to the warning log information to obtain an updated log warning message; In a case where the generated log alarm message does not include the log alarm message of the target threat scenario, message generation processing is performed according to the alarm log information to obtain the log alarm message of the target threat scenario.

11. The method according to claim 9, characterized in that Generating a log alarm message based on the alarm log information includes: Performing feature extraction processing on the information security log according to the scenario type of the target threat scenario corresponding to the information security log to obtain log tuple information; Searching for log tuple information based on the log identifier in the alarm log information to obtain characteristic information and an alarm identifier for each log in the alarm log information; A log alarm message is generated based on the log identifier, the characteristic information and the alarm identifier.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Get historical log alarm messages; Generate an information security situation change graph corresponding to the information security log according to the threat scenario and merging period to which the log warning message in the historical log warning message belongs; Feedback the information security situation change diagram.

13. A log data processing device, characterized in that: The device comprises: A feature matching module is used to perform feature matching processing on the information security log to determine the device type and log type of the information security log; A log parsing module, configured to perform log parsing processing on the information security log based on the device type and log type of the information security log to obtain key fields of the information security log; A scenario recognition module, configured to perform scenario recognition processing based on key fields of the information security log to determine the threat scenario to which the information security log belongs; A log classification module is used to merge and process the information security logs according to the threat scenarios to which the information security logs belong, and obtain alarm log information; The alarm processing module is used to generate a log alarm message based on the alarm log information.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images