Audit log analysis method, device and storage device

Abstract

The present invention discloses an audit log analysis method, device and storage device, and the method is as follows: log file and database information collection: storing various sql databases of users in log files of local disks; reading local log files, and pushing local log files to topics corresponding to kafka platforms; directly pulling table information in various sql databases of users, and persisting field information in the tables locally; log file and database information processing: persisting the first operation data after parsing the local log file locally; performing field collision detection on the table information, and then parsing to obtain the second operation data, and persisting it locally; operation data analysis: statistically analyzing the first operation data and the second operation data, and displaying the analysis results externally through a unified interface service. The beneficial effects are: improving the compatibility of log parsing, improving the processing performance under large-scale data processing, and improving the overall security of the system.

Description

Technical Field

[0001] The present invention relates to the field of audit log analysis, and in particular to an audit log analysis method, device and storage device. Background Art

[0002] At present, all walks of life are paying more and more attention to data security, and the network environment is becoming more complex. The operations on the database come from various internal and external entities, generating thousands of query and modification operations. Some operations may be sensitive, or there may be some potential security risks (for example, the password field of a user table is accessed multiple times, or the transaction amount of an order is abnormally modified, an unknown IP accesses the database, and a table that should not be accessed is accessed multiple times). At this time, various security-conscious organizations are in urgent need of providing a mechanism to monitor the current database operation statements and present them to users after statistical analysis, so that users can more intuitively discover potential security risks.

[0003] The existing technology is generally not universal for various databases, and often relies on the database's parsing engine to parse its own SQL. This method can only parse and analyze SQL statements of a specific database, but there is often more than one database in a complex environment, and parsing only one of them often cannot meet the needs. In addition, according to the conventional field collision logic, the solution of comparing whether the corresponding characters are contained one by one is inefficient. Summary of the invention

[0004] In order to solve the technical problems of poor compatibility and low efficiency of existing audit log parsing, the present invention provides an audit log analysis method, device and storage device, wherein the method specifically includes the following steps:

[0005] S1. Log file and database information collection:

[0006] For log files, store various sql databases of users in log files on the local disk; read local log files and push local log files to the corresponding topics of the Kafka platform;

[0007] For database information, directly pull table information from various SQL databases of users and persist the field information in the table locally;

[0008] S2. Log file and database information processing:

[0009] Persist the first operation data after parsing the local log file locally;

[0010] After performing field collision detection on the table information, the second operation data is obtained through parsing and persisted locally;

[0011] S3. Operation data analysis: Perform statistical analysis on the first operation data and the second operation data, and display the analysis results externally through a unified interface service.

[0012] A storage device stores instructions and data for implementing an audit log analysis method.

[0013] An audit log alarm device comprises: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement an audit log analysis method.

[0014] The beneficial effects provided by the present invention are: improving the compatibility of log analysis, improving the processing performance when processing large amounts of data, and being able to issue an alarm in a timely manner after colliding with sensitive data, thereby improving the overall security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the method of the present invention;

[0016] Figure 2 It is a schematic diagram of taking data from the map level by level;

[0017] Figure 3 It is a working diagram of the hardware device of the present invention. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0019] The relevant technical concepts in this application are as follows:

[0020] (1) Kafka: It is a distributed, partitioned, multi-replica distributed messaging system based on Zookeeper coordination. Its biggest feature is that it can process large amounts of data in real time to meet various demand scenarios.

[0021] (2) Persistence is a mechanism for converting program data between a persistent state and a transient state. In layman's terms, it is the process of converting transient data (such as data in memory, which cannot be permanently saved) into persistent data (such as data persisted in a database, which can be stored for a long time);

[0022] Please refer to Figure 1 , Figure 1 is a flow chart of the method of the present invention.

[0023] The present invention provides an audit log analysis method, device and storage device, wherein the method specifically comprises the following steps:

[0024] S1. Log file and database information collection:

[0025] For log files, store various sql databases of users in log files on the local disk; read local log files and push local log files to the corresponding topics of the Kafka platform;

[0026] For database information, directly pull table information from various SQL databases of users and persist the field information in the table locally;

[0027] It should be noted that the process of persisting the fields in the table to the local in step S1 is as follows:

[0028] Read the local database field table database_field and load it into the local fieldList;

[0029] Read the local database information table table_info and load it into the local tableList;

[0030] Read the local database table database_info and load it into the local databaseList.

[0031] S2. Log file and database information processing:

[0032] Persist the first operation data after parsing the local log file locally;

[0033] After performing field collision detection on the table information, the second operation data is obtained through parsing and persisted locally;

[0034] It should be noted that the conventional method receives the database operation log sent to Kafka by the log collector after normalization, and parses SQL according to the table field information, table information and other information stored in the data to enhance the original log data and store it in the local database for later analysis.

[0035] Different from the conventional scheme, the present invention converts the relevant information queried in the database into a Map structure and caches it in the local cache system. The SQL of the user operation log is divided into words, and data is retrieved from the Map level by level according to the word, and finally the field class containing all the information of the word is obtained to perform attribute enhancement and other operations. This measure can greatly reduce the time complexity, which is expected to be reduced from O(mn) to O(n)~o(nlogm), where n is the number of SQL word segments and m is the number of database fields.

[0036] Please refer to Figure 2 , Figure 2 It is a schematic diagram of taking data from the map level by level;

[0037] Specifically, the process of log and database information processing in step S2 is as follows:

[0038] Construct database information baseMap according to field table database_field, information table table_info, and database table database_info;

[0039] Cache the database information baseMMap;

[0040] Separate the SQL statements in the local log into a sqlWords list separated by spaces, ",", and ".";

[0041] For example: select id, date_time from network_attack will be segmented into ["select","id","date_time","from","network_attack"];

[0042] Traverse the sqlWords list and obtain the database dMap corresponding to the database table database_info through the baseMap.get(sqlWord) function;

[0043] Where sqlWord is the specified SQL statement;

[0044] Get the corresponding information table tMap through the dMap.get() function;

[0045] Get the corresponding field table fMap through the tMap.get() function;

[0046] Get the corresponding field class filed through the fMap.get() function;

[0047] Perform data enhancement on the field class flied;

[0048] Add enhanced data to the database operation table and related table;

[0049] In the present invention, the database field table (database_field), the operation table and the association table are shown in Tables 1, 2 and 3 respectively;

[0050] Table 1 Database field table

[0051]

[0052] Table 2 Database operation table (data_manipulate)

[0053]

[0054]

[0055] Table 3 Operation-Field Association Table (manipulate_relation)

[0056]

[0057] If the enhanced data is sensitive data, alarm data is directly generated and pushed to the Kafka platform, and corresponding notification or blocking operations are performed.

[0058] For example, during the SQL collision process, if a sensitive field marked in advance by the user (such as a password) is touched, or a sensitive operation is combined, and an update operation is performed on a field that should not be modified, such as a name, an alarm data will be directly generated and pushed to Kafka, and the alarm program will determine and execute the corresponding operation (send an email or text message, or directly block the corresponding IP through the firewall based on the operation IP in the log).

[0059] As an example, the normalized logs collected by the present invention are as follows:

[0060] {

[0061] "host":"192.168.184.131", / / Operation IP

[0062] "data_type":"mysql", / / database type

[0063] "manipulation":"SELECT id,sql FROM`mysql`.`general_log`LIMIT 0,1000", / / operation statement

[0064] }

[0065] The enhanced log after the above steps is:

[0066] Step 1: Decompose manipulation into a word list wordList(['SELECT','id','sql"FROM"mysql','general_log','LIMIT'])

[0067] Step 2: Get the mysql library list in baseMap according to the database type field data_type, collide wordList with the library information mysql one by one, and store it in the databases field

[0068] Step 3: Query the table information list under the MySQL database, use wordList to collide one by one to obtain the table information of general_log, and store it in the tables field

[0069] Step 4: Query the field list under the general_log table, obtain the field information, and store it in the fields field

[0070] Step 5: Determine the statement operation type based on whether the first word in wordList is select, alter, update, or delete, and store it in the type field.

[0071] Step 6: Determine whether sensitive operations have been performed on the field and store it in the alert field. Then decide whether to generate an alert and corresponding operations based on the field. This completes the operation.

[0072] After parsing:

[0073] {

[0074] "host":"192.168.184.131", / / Operation IP

[0075] "data_type":"mysql", / / database type

[0076] "manipulation":"SELECT id,sql FROM`mysql`.`general_log`LIMIT 0,1000", / / operation statement

[0077] "databases":["mysql"], / / database

[0078] "tables":["general_log"] / / table

[0079] "fields":["id","sql"] / / field

[0080] "type":"select", / / Operation type

[0081] "alert":"false" / / Whether to operate sensitive data

[0082] }

[0083] S3. Operation data analysis: Perform statistical analysis on the first operation data and the second operation data, and display the analysis results externally through a unified interface service.

[0084] For example, query the statement database after storage analysis and statistics show that a certain IP address (192.168.182.131) has performed 20 update operations on the MySQL database user information table user_info and checked the password field password of the table 120 times. Because the password field is a sensitive field, an alarm is generated and notified to the system administrator user.

[0085] See also Figure 3 , Figure 3 It is a schematic diagram of the working of the hardware device of an embodiment of the present invention, and the hardware device specifically includes: an audit log alarm device 401, a processor 402 and a storage device 403.

[0086] An audit log alarm device 401: The audit log alarm device 401 implements the audit log analysis method.

[0087] Processor 402: The processor 402 loads and executes instructions and data in the storage device 403 to implement the audit log analysis method.

[0088] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the audit log analysis method.

[0089] The beneficial effects of the present invention are: improving the compatibility of log analysis, improving the processing performance when processing large amounts of data, and being able to issue an alarm in a timely manner after colliding with sensitive data, thereby improving the overall security of the system.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An audit log analysis method, characterized in that: The following steps are involved: S1. Log file and database information collection: For log files, store various sql databases of users in log files on the local disk; Read local log files and push them to the corresponding topics of the Kafka platform; For database information, directly pull table information from various SQL databases of users and persist the field information in the table locally; The process of persisting the fields in the table to the local computer in step S1 is as follows: Read the local database field table database_field and load it into the local fieldList; Read the local database information table table_info and load it into the local tableList; Read the local database table database_info and load it into the local databaseList S2. Log file and database information processing: Persist the first operation data after parsing the local log file locally; After performing field collision detection on the table information, the second operation data is obtained through parsing and persisted locally; The process of log and database information processing in step S2 is as follows: Construct database information baseMap according to field table database_field, information table table_info, and database table database_info; Cache the database information baseMap; Separate the SQL statements in the local log into a sqlWords list separated by spaces, ",", and "."; Traverse the sqlWords list and obtain the database dMap corresponding to the database table database_info through the baseMap.get(sqlWord) function; where sqlWord is the specified sql statement; Get the corresponding information table tMap through the dMap.get() function; Get the corresponding field table fMap through the tMap.get() function; Get the corresponding field class filed through the fMap.get() function; Perform data enhancement on the field class flied; Add enhanced data to the database operation table and related table; S3. Operation data analysis: Perform statistical analysis on the first operation data and the second operation data, and display the analysis results externally through a unified interface service.

2. An audit log analysis method as claimed in claim 1, characterized in that: If the enhanced data is sensitive data, alarm data is directly generated and pushed to the Kafka platform, and corresponding notification or blocking operations are performed.

3. A storage device, characterized in that: The storage device stores instructions and data for implementing an audit log analysis method as described in any one of claims 1 to 2.

4. An audit log alarm device, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement an audit log analysis method as described in any one of claims 1 to 2.

Images