Log anomaly automatic analysis system

Abstract

The application discloses a log exception automatic analysis system and relates to the technical field of exception analysis. The system comprises a collection module, an analysis module, a big data storage module, a console module and an exception analysis module. The collection module is used for collecting relevant type data of a host end. The relevant type data comprises security product data and host security event data. The analysis module is used for standardizing analysis of the collected data. The application helps users solve the problems of security data collection, data mining, event backtracking and security capability integration based on an ATT&CK framework. The application directly mines data in depth to discover potential threats. The application realizes cross-node query operation through deep integration of original data, and can uniformly query, count and analyze heterogeneous data through embedded database data retrieval and threat hunting syntax. The application fully utilizes the data advantages of enterprises and truly hunts threats that cannot be discovered by conventional means.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of anomaly analysis, in particular to a log anomaly automatic analysis system. BACKGROUND

[0002] With the increasing requirements of security policies, the means of hacking attacks are constantly enriched, and the investment of enterprises in security is constantly expanding. More and more devices and products are adopted by enterprises to protect enterprise security in all aspects at different levels. The adaptive model believes that prediction->defense->monitoring->backtracking is an indispensable process for the whole enterprise. However, the old products still focus on defense and monitoring means, and the backtracking and prediction means are very scarce. Enterprises obtain a large amount of data and event behaviors from various products, but due to the lack of defensive professional knowledge, how to utilize these massive data is a great challenge for the security department of each enterprise.

[0003] In the modern information technology environment, system logs play a key role as an important tool for recording the running state of software and hardware. However, with the increasing complexity and data volume of systems, manual analysis of logs becomes increasingly difficult and is prone to miss potential anomalies and security threats. Therefore, it is particularly important to develop a log anomaly automatic analysis system.

[0004] At present, the Chinese invention application with application number CN202310644111.5 discloses an abnormal log analysis method, device, equipment and medium. A pre-trained language model is used in combination with an attention mechanism to determine the description information vector of the abnormal log. The semantic information more closely related to the current task is filtered out from the abnormal log, making the description information vector of the abnormal log more accurate. The LSTM network model is used to accurately extract the context semantic features of the description information vectors corresponding to multiple abnormal logs, improving the recognition accuracy of abnormal types during abnormal log analysis, reducing the cost of locating errors in logs, and meeting the user's modeling needs and the timeliness of rapid modeling in a short time. However, the existing technology cannot help users solve the problems of security data collection, data mining, event backtracking, and security capability integration based on the ATT&CK framework. It cannot directly perform deep mining on data to discover potential threats, cannot perform cross-node query operations through deep integration of functional raw data, does not have embedded database data retrieval and threat hunting syntax, cannot perform unified query, statistics, and analysis on heterogeneous data, and is not allowed to call various external detection capabilities for analysis. SUMMARY

[0005] The technical problem solved by the present application is that the prior art cannot help users solve the problems of security data collection, data mining, event backtracking and security capability integration based on the ATT&CK framework, cannot directly perform deep mining on data to discover potential threats, cannot realize cross-node query operation through deep integration of original data, does not have embedded database data retrieval and threat hunting syntax, cannot uniformly query, count and analyze heterogeneous data, and is not allowed to call various external detection capabilities for analysis.

[0006] To solve the above technical problems, the present application provides the following technical solutions: a log anomaly automatic analysis system, comprising a collection module, an analysis module, a big data storage module, a console module and an anomaly analysis module:

[0007] The collection module is used to collect relevant type data of the host end, and the relevant type data comprises security product data and host security event data;

[0008] The analysis module is used to perform standardized analysis on the collected data;

[0009] The big data storage module is used to perform distributed high-performance big data bottom layer storage, data storage after analysis of the original data source, and performance tuning on the distributed database;

[0010] The console module comprises a display query list and an instruction list, and is used to quickly obtain data and perform data query analysis;

[0011] The anomaly analysis module is used to uniformly analyze heterogeneous data and deeply integrate ATT&CK hunting threats.

[0012] Preferably, the collection module comprises:

[0013] The security product data comprises asset inventory, risk assessment, intrusion detection and compliance baseline of the host security product;

[0014] The host security event data comprises host audit events, process creation events, network creation events, DNS request events and user login events.

[0015] Preferably, the analysis module comprises:

[0016] The data of the visual configuration interface of the console module is supplemented, modified and associated in fields.

[0017] Preferably, the big data storage module comprises a data analysis unit and a data source access unit:

[0018] The data analysis unit comprises:

[0019] Integrating the data set passing through the analysis module into a database, optimizing the database includes:

[0020] Using a multi-process pipeline parallel mechanism to execute data analysis, and using a process pool to manage the pipeline process to obtain a subset of the database includes:

[0021] The log anomaly automatic analysis system automatically generates a data volume threshold for the subset of data after pre-training of the neural network, the larger the data volume threshold, the stronger the analysis capability, collects related type data of historical host configuration, and uses the log anomaly automatic analysis system to train the related type data to obtain default settings.

[0022] Preferably, the data source access unit includes:

[0023] The query statement is parsed and optimized, and includes:

[0024] According to the priority scheduling algorithm, part of the query statement and the aggregation condition are called, and the part of the query statement and the aggregation condition are sunk to the data source, and the characteristics of the data source are analyzed, and includes:

[0025] The default access parameters are preset, and the default access parameters include the number of concurrent connections, the cache size and the default timeout parameters, and the default access parameters are adjusted, and includes:

[0026] If the hardware performance of the data source is higher than the performance threshold, the number of concurrent connections and the cache size are increased;

[0027] If the data volume of the data source is greater than the data volume threshold, the data of the data source is processed and streamed using a distributed processing and streaming;

[0028] The characteristics of the data source include structured data, semi-structured data and unstructured data.

[0029] Preferably, the console module includes a basic control unit and a query unit:

[0030] The basic control unit includes a dashboard, a display panel and a unified identity authentication platform, the display panel is used to display a query list and an instruction list, the query list and the instruction list support export modes including API export, file export and chart export, and the unified identity authentication platform is used to add, delete, modify and query user account information, user account login, user account logout and display version information.

[0031] Preferably, the query unit includes a class query engine, a joint query engine, an advanced query engine and an auxiliary query engine:

[0032] The class query engine includes:

[0033] Retrieving data using query syntax of database query, the data using unified query syntax, the query syntax including but not limited to where, having and limit;

[0034] The joint query engine includes:

[0035] Querying multiple data sources using joint query, the joint query including inner join, left join and right join;

[0036] The advanced query engine includes:

[0037] Continuous analysis of optimizing data includes:

[0038] Respectively performing chain query in each query process to analyze query task, the advanced query engine automatically transferring previous segment result data into next segment task;

[0039] The auxiliary query engine includes:

[0040] Collecting user error syntax in historical query process, constructing user query syntax database, the user query syntax database being used to prompt user to input correct query content, saving query result, saving saved query instruction and query content in table, setting query instruction filtering condition input interface on control panel.

[0041] Preferably, the anomaly analysis module includes original data integration unit, ATT&CK hunting threat unit and deployment unit:

[0042] The original data integration unit includes:

[0043] Deeply integrating functional original data, the functional original data including data of functions of asset, risk, intrusion, log and task; the deep integration including integrating the functional original data as a whole into a database, integrating data entities, abstracting a plurality of security analysis objects, the data entities including host, process, file, application, site and user data, respectively assigning unique data ID to each data entity, the data ID including host information and business information, the security analysis objects being obtained after randomly extracting from the data entities, extracting abstract feature values after embedding layer, and regenerating data table according to characteristics of THP function;

[0044] The process of generating the data table includes:

[0045] The characteristics of the THP function are taken as target parameters of the pre-trained bat algorithm, the characteristics of the THP function include threat detection, response and tracking, and the characteristic feature vectors are extracted after the characteristics of the THP function pass through an embedding layer, the abstract feature values of the security analysis objects related to the characteristics of the THP function are extracted as input abstract feature values, and it is judged that the indicators related to the characteristics of the THP function include:

[0046] The probability correlation coefficient of the abstract feature values corresponding to the security analysis objects and the characteristic feature vectors is calculated:

[0047] When the probability correlation coefficient is greater than 0, it indicates that the security analysis object is positively correlated with the characteristics of the THP function;

[0048] When the probability correlation coefficient is less than 0 or equal to 0, it indicates that the security analysis object has no correlation with the characteristics of the THP function;

[0049] The output of the bat algorithm is integrated into a table according to the type, and the table is the data table, and the data table is updated according to a preset interval time period, and new security analysis objects and event records are added;

[0050] The user connects all function original data sources according to the data table and performs cross-node query operation.

[0051] Preferably, the ATT&CK hunting threat unit includes:

[0052] The hunting target is defined based on the MITRE ATT&CK knowledge base framework, the MITRE ATT&CK knowledge base framework includes techniques and tactics of known attacker behaviors, the hunting target includes hunting tactics of the known attackers, an attack chain model is created according to the techniques and tactics of the known attacker behaviors, an attacker's potential attack path is analyzed and obtained by using the attack chain model, the attack chain model is used to describe the complete process of the attacker from initial intrusion to final target implementation, and hunting operation is performed according to the attack chain model and the attacker's potential attack path, the hunting operation includes finding suspicious activities, the suspicious activities are behavior activities that are threats to the host, including behavior activities that damage host software and hardware, and the behavior activities that are threats to the host are obtained by host background detection.

[0053] The detected suspicious activities and the subsequent effects of the suspicious activities are identified, and the suspicious activities and the subsequent effects of the suspicious activities are spliced into a combined form and fed back to the MITRE ATT&CK knowledge base framework.

[0054] Preferably, the deployment unit includes:

[0055] When the log exception automatic analysis system is updated, the newly upgraded log exception automatic analysis system installation package is uploaded, based on the same installation environment, and based on the bottom logic code block of the log exception automatic analysis system, a new log exception automatic analysis system is directly installed and updated to be enabled.

[0056] The log exception automatic analysis system provided by the embodiment of the present application has the advantages that the ATT&CK framework is used to help users solve the problems of security data collection, data mining, event backtracking and security capability integration, the data is directly deeply mined to discover potential threats, cross-node query operations are realized through deep integration of original data, database data retrieval and threat hunting syntax are embedded, heterogeneous data can be uniformly queried, counted and analyzed, various external detection capabilities can be called for analysis, the data advantages of an enterprise are fully utilized, and threats that cannot be discovered by conventional means can be truly hunted. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 A basic flow diagram of the log exception automatic analysis system provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] In order to make the above objectives, characteristics and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments.

[0059] REFERENCE Figure 1 For an embodiment of the present application, a log exception automatic analysis system is provided, which includes a collection module, an analysis module, a big data storage module, a console module and an exception analysis module.

[0060] The collection module is configured to collect relevant type data of a host end, and the relevant type data includes security product data and host security event data.

[0061] The analysis module is configured to standardize the collected data.

[0062] The big data storage module is configured to perform distributed high-performance big data bottom storage, and to perform data storage after the original data source is analyzed, and to perform performance tuning on a distributed database.

[0063] The console module includes a display query list and an instruction list, and is configured to quickly acquire data and perform data query analysis.

[0064] The exception analysis module is configured to uniformly analyze heterogeneous data and deeply integrate ATT&CK hunting threats.

[0065] Directly mining data to discover potential threats, optimizing the performance of distributed databases, and embedding database data retrieval and threat hunting syntax can unify heterogeneous data for unified query, statistics and analysis.

[0066] The collection module comprises:

[0067] The security product data comprises asset inventory, risk assessment, intrusion detection and compliance baseline of the host security product.

[0068] The host security event data comprises host audit events, process creation events, network creation events, DNS request events and user login events.

[0069] The parsing module comprises:

[0070] The data of the visual configuration interface of the console module is supplemented, modified and associated in fields, flexible configuration and parsing rule logic are supported, and the system is adapted to SIEM and SOC products and can be seamlessly connected with the SIEM and SOC products.

[0071] The big data storage module comprises a data analysis unit and a data source access unit:

[0072] The data analysis unit comprises:

[0073] The data set by the parsing module is integrated into a database, and the database is optimized, comprising:

[0074] The data analysis is performed by using a multi-process pipeline parallel mechanism, and the pipeline processes are managed and analyzed by using a process pool to obtain a subset of the database, and the subset classification comprises:

[0075] The number of processes of the process pool is directly related to the data analysis and service capability, in order to improve the analysis efficiency, each analysis pipeline is performed on a random subset of the database in the corresponding analysis iteration process, in order to ensure the stability and analysis efficiency in the analysis process, the log abnormality automatic parsing system automatically generates a data amount threshold for the data amount of the subset after being pre-trained by a neural network, the larger the data amount threshold is, the stronger the analysis capability is, but the higher the requirements for memory and disk are, related type data of historical host configurations is collected, the related type data is trained by using the log abnormality automatic parsing system, and default settings are obtained.

[0076] The user can also manually set appropriate analysis process quantity and data amount threshold according to the resource quantity of CPU, memory and disk of the host load when deploying.

[0077] The data source access unit comprises:

[0078] Various data sources commonly used in security fields can be accessed, and the efficiency of loading data from the data sources directly affects the analysis efficiency. The query statements are parsed and optimized, including:

[0079] Part of the query statements and aggregation conditions are invoked according to the priority scheduling algorithm, and the part of the query statements and aggregation conditions are sunk to the data source to reduce the data transmission amount. The characteristics of the data source are analyzed, including:

[0080] Default access parameters are preset to meet the needs of most data sources, including the number of concurrent connections, cache size, and default timeout parameters, to ensure that the system can respond quickly in abnormal situations. The default access parameters are adjusted, including:

[0081] If the hardware performance of the data source is higher than the performance threshold, increase the number of concurrent connections and the cache size;

[0082] If the data amount of the data source is greater than the data amount threshold, use distributed processing and stream the data of the data source;

[0083] Through the analysis of the characteristics of the data source, reasonable default access parameters can be formulated, and flexible configuration options can be provided for users. In actual deployment, optimization can be performed according to hardware configuration and data amount, which helps to improve system performance and response speed, thereby realizing more efficient data management and analysis.

[0084] The characteristics of the data source include structured data, semi-structured data, and unstructured data.

[0085] The console module includes a basic control unit and a query unit:

[0086] The basic control unit includes a dashboard, a display panel, and a unified identity authentication platform. The display panel is used to display a query list and an instruction list. Users can quickly obtain data and perform data query analysis. The query list and the instruction list support export methods including API export, file export, and chart export. The unified identity authentication platform is used for adding, deleting, modifying, and inquiring user account information, logging in a user account, logging out a user account, and displaying version information. The user account information includes a username, a password, an email, a mobile phone number, an avatar, a personal photo, user role permissions, login time, login IP, and security settings.

[0087] The query unit includes a class query engine, a joint query engine, an advanced query engine, and an auxiliary query engine:

[0088] The class query engine includes:

[0089] Retrieving data using query syntax of database query, the data uses unified query syntax, the query syntax includes but is not limited to where, having and limit;

[0090] The joint query engine includes:

[0091] Query multiple data sources using joint query, the joint query includes inner join, left join and right join, and more comprehensive information is obtained through the association of data;

[0092] The advanced query engine includes:

[0093] Continuous analysis for optimizing data includes:

[0094] In order to meet the comprehensive security analysis scene, simplify the difficulty of continuous analysis, chain query is executed in each query process to analyze query task, the advanced query engine automatically transfers the previous result data into the next task, avoids intermediate result interference and storage waste, and directly obtains final conclusion.

[0095] The auxiliary query engine includes:

[0096] Collecting user error syntax in historical query process, constructing user query syntax database, the user query syntax database is used to prompt user to input correct query content, saving query result, saving saved query instruction and query content in table, facilitating user to call again, setting filter condition input interface of query instruction on control panel, and user can set filter condition of query instruction through the filter condition input interface.

[0097] The abnormal analysis module includes original data integration unit, ATT&CK hunting threat unit and deployment unit:

[0098] The original data integration unit includes:

[0099] Deeply integrate function raw data, the function raw data including data of functions of assets, risks, intrusions, logs and tasks; the deep integration includes integrally integrating the function raw data as one database to facilitate comprehensive analysis, provide a more comprehensive perspective, integrate data entities, abstract several security analysis objects, users can query a certain entity in different data tables, quickly expand the scope of investigation, find hidden dangers, the data entities including host, process, file, application, site and user data, respectively assigning a unique data ID to each data entity, the data ID including host information and business information, helping users quickly locate the problem location, the security analysis objects being obtained after randomly extracting from the data entities and extracting abstract feature values after an embedding layer, regenerating data tables according to the characteristics of the THP function, improving data readability and reducing the difficulty of customer data query;

[0100] The process of generating the data table includes:

[0101] Taking the characteristics of the THP function as the target parameters of the pre-trained bat algorithm, the characteristics of the THP function including threat detection, response and tracking, extracting characteristic feature vectors after the characteristics of the THP function passing through an embedding layer, extracting security analysis objects related to the characteristics of the THP function as input abstract feature values, judging the indicators related to the characteristics of the THP function including:

[0102] Calculating the probability correlation coefficient of the abstract feature values corresponding to the security analysis objects and the characteristic feature vectors:

[0103] When the probability correlation coefficient is greater than 0, it indicates that the security analysis object is positively correlated with the characteristics of the THP function;

[0104] When the probability correlation coefficient is less than 0 or equal to 0, it indicates that the security analysis object has no correlation with the characteristics of the THP function;

[0105] Integrating the output of the bat algorithm into the form of a table according to the type, the table being the data table, updating the data table according to a preset interval time period, adding new security analysis objects and event records to maintain the timeliness of the information;

[0106] Artificially preset the interval time period;

[0107] Users connect all function raw data sources according to the data table and perform cross-node query operations.

[0108] Users can connect other product raw data sources by themselves, jointly analyze in the product, associate query and solve the problem of insufficient capacity of a single product.

[0109] The ATT&CK hunting threat unit includes:

[0110] Based on the MITRE ATT&CK knowledge base framework, which includes techniques and tactics of known attacker behaviors, define hunting targets, which include hunting tactics for known attackers, create attack chain models according to the techniques and tactics of known attacker behaviors, analyze attacker potential attack paths using the attack chain models, the attack chain model is used to describe the complete process of the attacker from the initial intrusion to the final target implementation, the attack chain model is the prior art, according to the attack chain model and the attacker potential attack path, the hunting operation is carried out, the hunting operation includes finding suspicious activities, the suspicious activities are behavior activities that threaten the host, including behavior activities that damage the software and hardware of the host, and the behavior activities that threaten the host are obtained by the host background detection;

[0111] The MITRE ATT&CK knowledge base framework provides detection methods for more than 100 types of attack means, and each attack means is associated with the MITRE ATT&CK knowledge base framework;

[0112] The MITRE ATT&CK knowledge base framework provides a threat model to help security teams understand the attacker's operation mode.

[0113] Identify detected suspicious activities and subsequent impacts of the suspicious activities, and splice the suspicious activities and the subsequent impacts of the suspicious activities into a combined form and feed back to the MITRE ATT&CK knowledge base framework to optimize subsequent hunting activities, and can help customers update detection methods quickly according to the latest security intelligence, and immediately detect the latest security threats;

[0114] By combining the hunting threat operation with the MITRE ATT&CK knowledge base framework, the hunting operation is more targeted and systematic, the host detection capability is improved, the potential threat is quickly responded through the clear attack path identification, the response speed is enhanced, the MITRE ATT&CK knowledge base framework is continuously updated and enriched, the understanding and response ability of the MITRE ATT&CK knowledge base framework to emerging threats is improved, and the ATT&CK hunting threat unit helps the security team to more effectively defend and respond to the evolving threats in the complex network environment. The techniques and tactics of known attacker behaviors in the MITRE ATT&CK knowledge base framework include:

[0115] Initial access includes the way the attacker tries to enter the target system;

[0116] Execution includes the means by which the attacker executes malicious code on the target system;

[0117] Persistence includes strategies that allow an attacker to maintain long-term access to a target system;

[0118] Data theft includes ways to obtain sensitive data from the target environment;

[0119] Defense recommendations include the framework also provides recommended defense measures to help organizations improve security;

[0120] Detection and response includes providing detection methods for attack activities to help security teams identify and respond to threats;

[0121] The MITRE ATT&CK knowledge base framework is not only used for the daily work of the security team, but also provides valuable reference for security training, red-blue confrontation exercises, threat modeling, etc.

[0122] The deployment unit comprises:

[0123] When updating the log exception automatic analysis system, upload the newly upgraded log exception automatic analysis system installation package, based on the same installation environment, based on the bottom logic code block of the log exception automatic analysis system Directly install and update the new log exception automatic analysis system.

[0124] Only 1 ordinary server is required to meet the functional operation requirements, and if the user has requirements for data query speed, the related configuration can be expanded arbitrarily;

[0125] The customer only needs to fill in the relevant connection parameters to connect the corresponding data source and start the function.

[0126] The application helps users solve the problems of security data collection, data mining, event backtracking and security capability integration based on the ATT&CK framework, directly mines the data to find potential threats, realizes cross-node query operation through deep integration of function original data, and realizes unified query, statistics and analysis of heterogeneous data through embedded database data retrieval and threat hunting syntax, allows calling various external detection capabilities for analysis, fully utilizes the data advantages of enterprises, and truly hunts threats that cannot be found by conventional means.

[0127] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (or computer- readable storage media) having computer-usable program code embodied in the medium. The medium can be any available storage media that can be accessed by a computer. By way of example, and not limitation, such computer-usable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other storage medium(s) that can be used to carry or store desired computer program code in the form of instructions or data structures and that can be accessed by a computer. Also, the present application can be embodied in a computer program product that can be traded as goods or merchandise, through the storage medium described above or any other suitable medium. Accordingly, the computer medium can be any entity or device containing, or Figure 1 the functions specified in the flow or flows and / or blocks Figure 1 the functions specified in the flow or flows and / or blocks

[0128] It should be noted that the above-mentioned embodiments are only used to illustrate but not to limit the technical solutions of the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A system for automatic log anomaly resolution, characterized in that, The system comprises a collection module, an analysis module, a big data storage module, a console module and an abnormality analysis module. The collection module is used to collect relevant type data of the host end, and the relevant type data comprises security product data and host security event data. The analysis module is used to standardize the collected data. The big data storage module is used to perform distributed high-performance big data bottom layer storage, store data after analyzing the original data source, and perform performance tuning on the distributed database. The console module comprises a display query list and an instruction list, and is used to quickly obtain data and perform data query analysis. The abnormality analysis module is used to uniformly analyze heterogeneous data and deeply integrate ATT&CK hunting threats. The abnormality analysis module comprises an original data integration unit, an ATT&CK hunting threat unit and a deployment unit. The original data integration unit comprises: The deep integration function original data comprises asset, risk, intrusion, log and task function data; the deep integration comprises integrating the function original data into a database, integrating data entities, abstracting a plurality of security analysis objects, the data entities comprise host, process, file, application, site and user data, each data entity is assigned a unique data ID, the data ID comprises host information and business information, the security analysis objects are obtained after extracting abstract feature values through an embedding layer after being randomly extracted from the data entities, and the security analysis objects are regenerated into a data table according to the characteristics of the THP function; The process of generating the data table comprises: The characteristics of the THP function are used as target parameters of a pre-trained bat algorithm, the characteristics of the THP function comprise threat detection, response and tracking, the characteristics of the THP function are extracted through an embedding layer to obtain characteristic feature vectors, the security analysis objects related to the characteristics of the THP function are extracted as abstract feature values, and the indexes related to the characteristics of the THP function are judged, comprising: The probability correlation coefficient of the abstract feature values corresponding to the security analysis objects and the characteristic feature vectors is calculated: When the probability correlation coefficient is greater than 0, it indicates that the security analysis object is positively correlated with the characteristics of the THP function; When the probability correlation coefficient is less than 0 or equal to 0, it indicates that the security analysis object has no correlation with the characteristics of the THP function; The output of the bat algorithm is integrated into a table according to the type, and the table is the data table, the data table is updated according to a preset interval, new security analysis objects and event records are added; The user connects all function original data sources according to the data table, and performs cross-node query operation.

2. The log anomaly automatic resolution system of claim 1, wherein, The collection module comprises: The security product data comprises asset inventory, risk assessment, intrusion detection and compliance baseline of the host security product. The host security event data comprises host audit events, process creation events, network creation events, DNS request events and user login events.

3. The log anomaly automatic resolution system of claim 1, wherein, The analysis module comprises: Field supplement, modification and association are made to the data of the visual configuration interface of the console module.

4. The log anomaly automatic resolution system of claim 1, wherein, The big data storage module comprises a data analysis unit and a data source access unit: The data analysis unit comprises: The data set through the analysis module is integrated into a database, and optimization of the database comprises: The data analysis is performed by using a multi-process pipeline parallel mechanism, and the pipeline processes are managed by using a process pool to obtain a subset of the database, and the subset is classified: The log anomaly automatic analysis system automatically generates a data volume threshold for the subset after pre-training by using a neural network, the larger the data volume threshold is, the stronger the analysis capability is, relevant type data of historical host configurations is collected, the relevant type data is trained by using the log anomaly automatic analysis system, and a default setting is obtained.

5. The log anomaly automatic resolution system of claim 4, wherein, The data source access unit comprises: The query statement is analyzed and optimized, and the query statement is analyzed and optimized: According to the priority scheduling algorithm, part of the query statement and the aggregation condition are called, and the part of the query statement and the aggregation condition are sunk to the data source, and the characteristics of the data source are analyzed: A default access parameter is preset, the default access parameter comprises a concurrent connection number, a cache size and a default timeout parameter, and the default access parameter is adjusted: If the hardware performance of the data source is higher than the performance threshold, the concurrent connection number and the cache size are increased; If the data volume of the data source is greater than the data volume threshold, the data of the data source is processed by using a distributed processing and a streaming transmission; The characteristics of the data source comprise structured data, semi-structured data and unstructured data.

6. The log anomaly automatic resolution system of claim 1, wherein, The console module comprises a basic control unit and a query unit: The basic control unit comprises a dashboard, a display panel and a unified identity authentication platform, the display panel is used for displaying a query list and an instruction list, the query list and the instruction list support export modes including API export, file export and chart export, and the unified identity authentication platform is used for adding, deleting, modifying and inquiring the basic information of a user account, logging in the user account, logging out the user account and displaying version information.

7. The log anomaly automatic resolution system of claim 6, wherein, The query unit comprises a class query engine, a joint query engine, an advanced query engine and an auxiliary query engine: The class query engine comprises: The data is retrieved by using a query syntax of a database query, the data uses a unified query syntax, and the query syntax comprises where, having and limit; The joint query engine comprises: A plurality of data sources are queried by using a joint query, and the joint query comprises an inner join, a left join and a right join; The advanced query engine comprises: Continuous analysis of the data comprises: Chain queries are respectively performed in each query process to analyze the query task, and the advanced query engine automatically transfers the result data of a previous section to a next section of the task; The auxiliary query engine comprises: User error syntax in a historical query process is collected, a user query syntax database is constructed, the user query syntax database is used to prompt a user to input correct query content, a query result is saved, a saved query instruction and query content are saved in a table, and a query instruction filtering condition input interface is set on a control panel.

8. The log anomaly automatic resolution system of claim 1, wherein, The ATT&CK threat hunting unit includes: Hunting targets are defined based on the MITRE ATT&CK knowledge base framework, which includes techniques and tactics for known attacker behaviors. Hunting targets include hunting tactics for the known attackers. An attack chain model is created based on the techniques and tactics for the known attacker behaviors. The attack chain model is used to analyze and obtain the attacker's potential attack paths. The attack chain model is used to describe the attacker's complete process from initial intrusion to the final goal. Hunting operations are performed based on the attack chain model and the attacker's potential attack paths. The hunting operations include searching for suspicious activities. Suspicious activities are behavioral activities that threaten the host, including activities that damage the host's software and hardware. Behavioral activities that threaten the host are detected by the host background. Identify detected suspicious activities and subsequent impacts of the suspicious activities, and splice the suspicious activities and the subsequent impacts of the suspicious activities into a combined form and feed them back into the MITRE ATT&CK knowledge base framework.

9. The log anomaly automatic resolution system of claim 1, wherein, The deployment unit includes: When the log exception automatic parsing system is updated, the newly upgraded log exception automatic parsing system installation package is uploaded, and based on the same installation environment, the new log exception automatic parsing system is directly installed, updated and enabled based on the underlying logic code blocks of the log exception automatic parsing system.

Images