A log aggregation query method for massive data

Abstract

The present invention discloses a log aggregation query method for massive data, including: log collection: collecting different types of log data from a server; processing the data fields in the log using a dynamic and static combination field to obtain processed data; analyzing the processed data using a cold-hot separation model to obtain cold and hot data, and shunting the cold and hot data into a database; first performing log query through a hot database, and then performing log query through a cold database. The present invention shuns log data by using dynamic and static combination field processing and a cold-hot separation model. When querying, the user directly queries from the hot data, which greatly reduces the query scope, reduces the magnitude, and improves the overall query speed.

Description

Technical Field

[0001] The present invention relates to the field of log processing, and in particular to a log aggregation query method for massive data. Background Art

[0002] With the development of the information age, a large amount of log data will be monitored in real time, and the analyzed data will be statistically analyzed to discover and solve problems in production through different dimensions. How to efficiently and quickly analyze according to the time range, be able to flexibly switch dimensions for analysis, and obtain the corresponding data very quickly, these issues have gradually become the focus of attention in the market. The more common logs in the market are:

[0003] 1. Security log: records application operations, errors, exceptions, performance indicators and other information for tracking and troubleshooting application problems.

[0004] 2. Application log: records application operations, errors, exceptions, performance indicators and other information for tracking and troubleshooting application problems.

[0005] 3. System log: records operating system events, errors, warnings and other information, including login information, service start and stop, resource utilization, etc.

[0006] 4. Database log: records database operations, queries, transactions and other information, which helps track and debug database-related issues.

[0007] 5. Server log: records the performance indicators, errors, events and other information of the server hardware and operating system, including CPU, memory, disk utilization, network traffic, etc.

[0008] 6. User log: records user operation activities, login records, access rights and other information in the system.

[0009] 7. Debug log: The log recorded by developers during the development and debugging process, used to track code execution paths, view variable values, etc.

[0010] 8. Business log: A log that records specific business operations, such as transaction logs, order logs, log queries, etc., used to track business processes and subsequent analysis.

[0011] As a management strategy to improve efficiency and reduce noise, log aggregation has become an important practice in modern IT operations. The following is some background information about log aggregation:

[0012] 1. The number of log aggregations has increased dramatically: As the scale and complexity of IT systems continue to increase, the number of logs generated by monitoring systems is also increasing. A large number of duplicate, redundant, and insignificant logs have brought tremendous pressure to operation and maintenance personnel, so log aggregation has become a necessary means.

[0013] 2. Accurate contextual information is needed: It is often difficult to obtain the full picture and contextual information of the problem through only a single log. Log aggregation can integrate related logs, provide more accurate contextual information, and help the operation and maintenance team better understand and analyze the problem.

[0014] 3. Improve fault handling efficiency: Log aggregation can help the operation and maintenance team locate and handle faults more quickly and accurately. By aggregating and filtering logs, operation and maintenance personnel can solve problems more specifically and avoid getting lost in massive logs.

[0015] In summary, log aggregation, as a strategy to deal with log flooding and improve operation and maintenance efficiency, is gaining widespread application and attention.

[0016] At present, the technology used for log aggregation is mainly offline processing. The data is calculated offline through MapReduce at fixed minutes, hours, and days, and the aggregated data is stored in a table. The fields in the table contain all the fields.

[0017] The disadvantages of offline processing are as follows:

[0018] 1. Offline processing has a long delay response time. Data must be stored in the database first and then queried from the database, which increases the pressure on the database and reduces the processing performance.

[0019] 2. The original data fields are redundant to the aggregate fields. For conditional queries, there are too many fields and the query acquisition speed is slow.

[0020] 3. Changes in the log aggregation range result in slow data response. For example, if a user wants to change the aggregation time range from 5 minutes to 1 hour, then a new task must be run to calculate the data from a certain point in time to the present. This often takes a long time to process, and the recovery is slow, which cannot respond to user needs in a timely manner. Alternatively, a separate hourly table or a daily table must be maintained, which takes up server resources. Summary of the invention

[0021] In order to solve the above problems, the present invention proposes a log aggregation query method for massive data, which greatly reduces the magnitude of log processing, speeds up the query speed and shortens the response time.

[0022] The method comprises the following steps:

[0023] S1. Log collection: collect different types of log data from the server;

[0024] S2, log processing: the data fields in the log are processed using the dynamic and static combined fields to obtain the processed data;

[0025] S3, log storage: The processed data is analyzed using the cold-hot separation model to obtain cold and hot data, and the cold and hot data are separated and stored in the warehouse;

[0026] S4. Log query: First perform log query through the hot database, and then perform log query through the cold database.

[0027] The beneficial effect provided by the present invention is that: by adopting the dynamic and static combined field processing and the cold and hot separation model, the log data is diverted, and when querying, the user directly queries from the hot data, which greatly narrows the query scope, reduces the magnitude, and improves the overall query speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic flow chart of the method of the present invention;

[0029] Figure 2 It is a flow chart of real-time data processing of the present invention;

[0030] Figure 3 It is a user query flow diagram;

[0031] Figure 4 It is a schematic diagram of the data dimensionality reduction process of the present invention. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0033] Please refer to Figure 1 , Figure 1 It is a schematic flow chart of the method of the present invention;

[0034] The present invention provides a log aggregation query method for massive data, comprising:

[0035] S1. Log collection: collect different types of log data from the server;

[0036] It should be noted that different types of logs include: security logs, application logs, system logs, database logs, server logs, user logs, debug logs, business logs, etc.

[0037] S2, log processing: the data fields in the log are processed using the dynamic and static combined fields to obtain the processed data;

[0038] It should be noted that the static and dynamic combined fields specifically refer to dividing the data fields into fixed fields and dynamic column fields.

[0039] The fixed fields are specifically fields that can be aggregated. Specifically, fixed fields are usually the most basic attributes of the log, which are variable aggregate data. For example, fixed fields are: the latest occurrence time, the latest occurrence time, the number of aggregated items, the highest level, whether blocked, etc. Fixed fields are used to calculate and count the data on the result page.

[0040] As an example, there is a batch of log data as follows:

[0041] 1. IP: 172.10.12.3 Traffic: 3512MB Level: Medium Time of generation: 2023-09-09 08:08:00 Number of attacks on external network: 1 Blocking: Yes;

[0042] 2. IP: 172.10.12.3 Traffic: 1024MB Level: Low Time of generation: 2023-11-10 20:08:00 Number of attacks on external network: 2 Blocking: Yes;

[0043] 3. IP: 172.10.12.3 Traffic: 7789MB Level: Medium Time of generation: 2023-12-10 16:08:00 Number of attacks on external network: 3 Blocking: No;

[0044] Then the fixed fields can be defined as:

[0045] Minimum flow, maximum flow, highest level, earliest occurrence time, latest occurrence time, total number of attacks on the external network, total number of blocked attacks, and number of unblocked attacks;

[0046] It should be noted that the fixed fields can be customized according to the attributes of the fields. According to the above definition of the fixed fields, the final display results of the fixed fields are: 1024MB, 7789MB, medium, 2023-09-09 08:08:00, 2023-12-10 16:08:00, 6, 2, 1.

[0047] The dynamic column field is a user-defined feature field.

[0048] It should be noted that the dynamic column fields are the fields that users need to pay attention to.

[0049] For example, user-defined dynamic column fields are: source address, source port, destination address, destination port, and geographic location.

[0050] As an embodiment, the actual characteristic values ​​of several logs of the above-defined dynamic column fields are:

[0051] 172.16.1.11_8080_172.16.12.3_8081_Wuhan;

[0052] 172.16.1.11_8080_172.16.12.52_8082_Yellowstone;

[0053] 222.16.1.12_8097_172.16.12.52_8082_Hangzhou.

[0054] S3, log storage: The processed data is analyzed using the cold-hot separation model to obtain cold and hot data, and the cold and hot data are separated and stored in the warehouse;

[0055] It should be noted that in step S3, the cold-hot separation model specifically refers to:

[0056] The processed data is represented according to the dynamic column fields;

[0057] Convert dynamic column fields into corresponding unincode values;

[0058] A new batch of log data to be processed is compared in a certain data classification through a cluster analysis algorithm. If the difference between the unincode value of the new batch of log data to be processed and the unincode value of a certain data classification is less than or equal to a preset value, the new batch of log data to be processed is marked with a hot label, and the log data is hot data; otherwise, the data is cold data.

[0059] As an example, for the dynamic column field in the above example, converting it into a unincode value is as follows:

[0060] Field Source Address_Field Source Port_Field Destination Address_Field Destination Port_Field Geographic Location:

[0061] 20320_22909_65111_19990_123331;

[0062] 20320_22909_65292_19220_256431;

[0063] 52789_22898_65292_19220_751621;

[0064] The above unincode values ​​are analyzed and compared through the clustering algorithm. First, the matching rules are set. The matching rules use the method of exact matching + fuzzy field.

[0065] Exact match: requires the source address and source port to be exactly the same;

[0066] Fuzzy fields: It is required that the destination address, destination port, and geographical location are fuzzily the same.

[0067] Regarding being fuzzily the same, for example, if the destination ports are fuzzily the same, it means that the destination port contains 80; another example, if the geographical locations are fuzzily the same, it means that the geographical locations are both in Hubei.

[0068] Regarding the clustering analysis algorithm, the Kmeans clustering algorithm is adopted in the present invention:

[0069] 1. For the first batch of randomly arriving logs, randomly select k pieces of data as the initial centroids k (features)

[0070] 2. Calculate the distances from the data of other logs to these k centroids (in this application, it means that the above-mentioned differences are within the preset value);

[0071] If the distance of a certain log data is closer to the nth centroid, then this point is classified as n, and it is labeled as n. When the next batch of logs enters again, the calculated data is used to calculate the average value as classification n.

[0072] 3. The data with the same classification n will be quickly compared. The data that conforms to the same features (the aforementioned matching rules) will be grouped into the same aggregated data, and the data with non-conforming features will generate a new aggregated data.

[0073] The purpose of the above approach is to divide the massive data into multiple classification intervals, and the data is compared within the intervals of the same category. For example, for geographical location comparison, different types of cities can be divided with reference to the clustering analysis algorithm.

[0074] Taking the data of the above dynamic column fields as an example:

[0075] 172.16.11.1_8080_Wuhan and 172.16.11_8080_Xiaogan are similar data;

[0076] The aggregated feature (matching rule) is 172.16.11.1_8080_Hubei, which is hot data;

[0077] When the next batch of data comes in and collides with the same aggregated feature 172.16.11.1_8080_Hubei, the data is updated to hot data.

[0078] If the aggregated feature data has never been collided with, it is cold data; for cold data, there will be no change in the data that conforms to this aggregated feature, and the maximum value, minimum value, and time will not change either.

[0079] The above is about the distinction between cold and hot data. Cold and hot data are separately flowed into and stored in different databases, namely the cold database and the hot database.

[0080] As another embodiment, in the present application, the cold data and the hot data are also changeable, and the hot data can be changed into the cold data, and the cold data can also be changed into the hot data.

[0081] The specific strategy for converting hot data into cold data is as follows:

[0082] Set a scheduled task to scan hot data. If the latest log generation time is greater than the current t days, the hot data of the corresponding feature will be updated to cold data.

[0083] As an embodiment, t can be set to 30 days, half a year, or one year, etc.; the frequency of the scheduled task can be once a day, or once every few days, etc.;

[0084] The specific strategy for converting cold data into hot data is:

[0085] When a certain log data to be processed is compared in a certain data classification and fails to match in the hot data but is found in the cold data, the log data to be processed is updated to the hot data.

[0086] S4. Log query: First perform log query through the hot database, and then perform log query through the cold database.

[0087] As an embodiment, the present invention adopts a split data entry system, where cold data is specifically entered into a cold database, and hot data is specifically entered into a hot database. At the same time, according to the set strategy, the cold and hot databases are also frequently updated.

[0088] When performing log queries, first performing a collision with the hot database can greatly reduce query time and improve query efficiency.

[0089] Please refer to Figure 2 , Figure 2 It is a flow chart of real-time data processing of the present invention;

[0090] After the log data is collected and processed from the server, the processed data first enters the original database, and at the same time is introduced into another branch for clustering analysis to obtain cold and hot databases.

[0091] Please refer to Figure 3 , Figure 3 It is a user query flow chart.

[0092] When users query logs, they first specify the time range, and then perform data collision on the hot database to obtain fields that meet the conditions.

[0093] Finally, based on the fields that meet the conditions, more detailed log content is retrieved from the original database.

[0094] Please refer to Figure 4 , Figure 4It is a schematic diagram of the data dimensionality reduction process of the present invention.

[0095] The original data is 1 billion, and after processing the fixed fields and dynamic column fields, detailed data is obtained. The detailed data is analyzed using the clustering algorithm of the associated feature value to obtain hot data and cold data.

[0096] The technical implementation of the present invention relies on: Spark+Kafka+ElasticSearch+HDFS+Redis cluster+big data cluster;

[0097] Database used: ES\openTsdb;

[0098] Implementation method: Build servers and collection servers, build a big data cluster environment, simulate sending security logs and program logs, observe data aggregation, and use Kafka-Manager and java jvm monitoring tools to monitor log storage. In terms of performance, on average, 20,000 aggregated data can be processed per second, and it takes nearly 5 seconds to query tens of millions of data.

[0099] Finally, the beneficial effect of the present invention is that: due to the use of dynamic and static combined field processing and hot and cold separation model, the log data is diverted, and when querying, the user directly queries from the hot data, which greatly narrows the query scope, reduces the magnitude, and improves the overall query speed.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log aggregation query method for massive data, characterized by: include: S1. Log collection: collect different types of log data from the server; S2, log processing: the data fields in the log data are processed using the dynamic and static combined fields to obtain the processed log data; S3, log storage: The processed log data is analyzed using the cold-hot separation model to obtain cold and hot data, and the cold and hot data are separated and stored in the warehouse; S4, log query: first query the log through the hot database, and then query the log through the cold database; The static and dynamic combined fields specifically refer to dividing the data fields into fixed fields and dynamic column fields; The dynamic column field is a user-defined feature field; The process of analyzing the log data processed by the cold-hot separation model in step S3 is as follows: Represent the processed log data according to dynamic column fields; Dynamic column fields include: Source address, source port, destination address, destination port, geographic location; Convert dynamic column fields into corresponding unincode values; A new batch of log data to be processed is compared with a certain data classification through a cluster analysis algorithm. If the difference between the unincode value of the new batch of log data to be processed and the unincode value of a certain data classification is less than or equal to a preset value, the new batch of log data to be processed is marked with a hot label, and the log data is hot data; otherwise, the log data is cold data.

2. A log aggregation query method for massive data as claimed in claim 1, characterized in that: The fixed fields are specifically fields that can be aggregated.

3. A log aggregation query method for massive data as claimed in claim 1, characterized in that: When a certain log data to be processed fails to match in a certain data classification during comparison in the hot database but successfully matches in the cold database, the log data to be processed is updated to hot data.

Images