A method and device for quickly parsing log files

Abstract

The present invention relates to a method and device for rapid parsing of log files, which are applied to a distributed system, wherein the method comprises the following steps: obtaining and storing a regular parsing object set, obtaining a log file to be parsed; cutting the log file into multiple sub-files, reading and loading each of the sub-files; distributing the read and loaded sub-files to multiple processing units in parallel; each processing unit matches the regular parsing object of the corresponding sub-file in the regular parsing object set, and performs parsing of the corresponding sub-file; and storing the parsing results of each processing unit. The present invention has the technical effects of high log parsing efficiency, low memory and CPU consumption, and applicability to ultra-large log files.

Description

Technical Field

[0001] The present invention relates to the technical field of log analysis, and in particular to a method and device for quickly analyzing a log file, and a computer storage medium. Background Art

[0002] A large number of log files are generated in various types of equipment and systems in various industries. The analysis of log files can provide valuable and reliable indicators and parameters for various types of equipment and systems to support the operation, improvement and development of various types of equipment and systems. For example, banks, enterprises, and government units have many firewall devices. The log data generated by each firewall device during operation is massive. These logs are stored in units of days or categories to form log files; big data analysis based on these log files can provide multi-dimensional and different granularity security protection data, providing heavyweight data support for network security. In addition, the business systems or production management systems used by traditional manufacturing, e-commerce platforms, finance, logistics, aviation, and social platforms will generate a large amount of log data during operation, forming log files; these log files can be used to mine valuable data in the scenario of big data analysis, and generate reliable and usable reference indexes and reports to support the rapid development of the later business; based on log analysis, useful data indexes such as bottlenecks, limitations, and growth space for business development can also be found.

[0003] Currently, log parsing can be done by using regular expression parsing, using XML configuration to implement log parsing, and other solutions. Most of these solutions are not perfect. When parsing large log files, especially ultra-large log files (log file size exceeds 10G, log file line number is greater than or equal to 2 million), the parsing speed is too slow, and the consumption of memory and processor is large. Moreover, the memory will be occupied for a long time during parsing without being released, and the CPU usage rate will remain high, resulting in low efficiency of other software, delayed response of the operating system, and affecting the customer operation experience; if the log file is slightly larger, it may be stuck or data lost, and the stuck situation is even more serious when parsing ultra-large log files. Summary of the invention

[0004] In view of this, it is necessary to provide a log file fast parsing method and device to solve the problems of low parsing efficiency of large log files, high consumption of memory and processor, and easy jamming.

[0005] The present invention provides a log file fast parsing method, which is applied to a distributed system and includes the following steps:

[0006] Obtain and store a collection of regular expression parsing objects, and obtain the log file to be parsed;

[0007] Cut the log file into multiple sub-files, and read and load each of the sub-files;

[0008] Distribute the read and loaded sub-files to multiple processing units in parallel;

[0009] Each processing unit matches the regular parsing object of the corresponding sub-file in the regular parsing object set, and performs parsing of the corresponding sub-file;

[0010] The analysis results of each processing unit are stored in the database.

[0011] Furthermore, it also includes:

[0012] After completing the parsing process of all sub-files of a log file, the system sleeps for a set time and then performs the parsing process of the next log file.

[0013] Furthermore, a collection of regular expression parsing objects is obtained and stored, specifically:

[0014] Set up a dedicated storage space, receive the regular parsing object set sent by the java component through the activeMq message queue, and store it in the dedicated storage space.

[0015] Furthermore, the log file is cut into a plurality of sub-files, and before reading and loading each of the sub-files, the process further includes:

[0016] Determine whether the log file is a compressed file. If so, decompress the log file and then move the decompressed log file to a temporary parsing directory. Otherwise, directly move the log file to a temporary parsing directory.

[0017] Further, the log file is cut into multiple sub-files, and each sub-file is read and loaded, specifically:

[0018] Determine whether the total number of lines of the log file is greater than a set number of lines, if so, cut the log file into multiple sub-files with the set number of lines, otherwise do not cut;

[0019] Obtain the path address of each sub-file;

[0020] The sub-files are read in parallel or serially based on the path address and loaded into a memory buffer.

[0021] Furthermore, the set number of rows and the size of the memory buffer are set according to configuration parameters of the distributed system.

[0022] Furthermore, distributing the read and loaded sub-files to multiple processing units in parallel also includes:

[0023] Each of the processing units distributes the log data in the corresponding sub-file to a processing unit at the same level or a processing unit at the next level.

[0024] Further, each processing unit matches the regular parsing object of the corresponding sub-file in the regular parsing object set, and performs parsing of the corresponding sub-file, specifically:

[0025] Each processing unit preliminarily parses the corresponding sub-file, obtains the log header IP, and matches the corresponding regular parsing object in the regular parsing object set through the log header IP;

[0026] Each processing unit performs parsing field matching on the corresponding sub-file according to the matched regular parsing object, and if the match is successful, extracts the parsing field data of the corresponding sub-file;

[0027] The parsing field data obtained by matching each processing unit is combined to form a parsing result.

[0028] The present invention also provides a log file fast parsing device, comprising a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the log file fast parsing method is implemented.

[0029] The present invention also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the log file rapid parsing method is implemented.

[0030] Beneficial effect: before parsing the log file, the present invention first cuts the log file, and then uses multiple processing units to parse and process the cut sub-files respectively. The log parsing speed is improved by dividing the log file, and at the same time, the log parsing process can be run even in a low CPU and low memory environment, avoiding the problem of system jamming and data loss caused by the parsing of the log file. At the same time, when each processing unit performs log parsing, the regular parsing object it uses is obtained by real-time matching in the regular parsing object set, and the regular parsing object set is not configured in the distributed system, but is obtained and configured through other components / systems outside the distributed system. This decoupling design of regular parsing and distributed system makes the parsing process have high scalability, which is convenient for later business expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A method flow chart of a first embodiment of a method for rapid parsing of log files provided by the present invention;

[0032] Figure 2 A schematic diagram of a regular parsing object set loading process of the first embodiment of the log file fast parsing method provided by the present invention;

[0033] Figure 3 A specific flow chart of the first embodiment of the log file rapid parsing method provided by the present invention;

[0034] Figure 4 A data flow diagram of a processing unit of the first embodiment of the log file fast parsing method provided by the present invention;

[0035] Figure 5 A schematic diagram of the regularized parsing process of the first embodiment of the log file fast parsing method provided by the present invention;

[0036] Figure 6 A schematic diagram of the modular structure of the first embodiment of the log file rapid parsing device provided by the present invention. DETAILED DESCRIPTION

[0037] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0038] Example 1

[0039] like Figure 1 As shown, embodiment 1 of the present invention provides a log file fast parsing method, hereinafter referred to as this method, which is applied to a distributed system and includes the following steps:

[0040] S1. Obtain and store a set of regular expression parsing objects, and obtain the log file to be parsed;

[0041] S2. Cut the log file into multiple sub-files, and read and load each of the sub-files;

[0042] S3, distributing the read and loaded sub-files to multiple processing units in parallel;

[0043] S4, each processing unit matches the regular parsing object of the corresponding sub-file in the regular parsing object set, and performs parsing of the corresponding sub-file;

[0044] S5. The analysis results of each processing unit are stored in the database.

[0045] The log file fast parsing method provided by the method is deployed in a distributed system. In this embodiment, the distributed system is Apache Storm. Apache Storm is a free and open source distributed real-time computing system that is good at processing massive data and is suitable for real-time data processing rather than batch processing. It can perform various operations on real-time data in parallel, and can be conveniently and quickly expanded horizontally to obtain higher parsing performance. Specifically, S101, first obtain a set of regular parsing objects so that the implementation configuration of regular parsing objects can be performed during subsequent log parsing; S102, secondly, find all log files that need to be analyzed; then cut a single super-large log file into small files, that is, sub-files, obtain the path address of the cut sub-files, read and load each sub-file in parallel or serially, and the size of the memory buffer for loading sub-files can be set according to the configuration of the distributed system to achieve the optimal performance configuration; the read and loaded data is distributed in parallel to multiple distributed processing units of the distributed system, and each processing unit obtains the corresponding regular parsing object to parse the corresponding sub-file, and the parsing result set forms a .sgn suffix file to be stored, and the .sgn suffix file is stored and written.

[0046] In this embodiment, before parsing the log file, the log file is first cut, and then multiple processing units are used to parse and process the cut sub-files respectively. The log parsing speed is increased by splitting the log file, which can reach 150,000 / s; at the same time, the log parsing process can be run even in a low CPU and low memory environment, especially for super large log files. This processing method can greatly improve the efficiency of log parsing, and quickly and accurately read and parse super large log files under low memory and low CPU consumption conditions, avoiding the problem of system jamming and data loss caused by the parsing of log files. At the same time, when each processing unit performs log parsing, the regular parsing object it uses is matched in real time in the regular parsing object set, and the regular parsing object set is not configured in the distributed system, but is obtained and configured through other components / systems outside the distributed system. Therefore, the regular parsing is decoupled from the distributed system. This low coupling design makes the parsing process have high scalability and is convenient for later business expansion.

[0047] The log file fast parsing method provided in this embodiment is applicable to the parsing of log files of any size. However, due to its high parsing efficiency and low memory and CPU consumption, it is particularly applicable to very large log files.

[0048] Preferably, it also includes:

[0049] After completing the parsing process of all sub-files of a log file, the system sleeps for a set time and then performs the parsing process of the next log file.

[0050] On the basis of the low coupling configuration of log file cutting and regular parsing objects, this embodiment also sets a sleep time after completing a complete log file parsing process, for example, sleep for 1 millisecond, so that the memory and CPU can be released, which can further instantly reduce the use and occupation of the CPU and memory, and avoid the long-term occupation of memory and high CPU usage due to the parsing of log files, which leads to low efficiency of other software operations, operating system delays, and customer operation experience. Especially for very large log files, this method can read and parse very large log file data losslessly, with low CPU usage and low memory consumption, and can quickly adapt to more log formats with lightweight and low coupling.

[0051] Preferably, a set of regular expression parsing objects is obtained and stored, specifically:

[0052] Set up a dedicated storage space, receive the regular parsing object set sent by the java component through the activeMq message queue, and store it in the dedicated storage space.

[0053] Specifically, Figure 2 As shown, in this embodiment, the regular parsing object collection is configured in the Java component. The Java component is a component at the same level as Storm. It is a functional project component specially set up to implement log parsing. In order to understand the coupling design and facilitate customer operation, we put the association and binding relationship between the regular configuration file and the manufacturer's log format in the Java component. The work of the regular configuration parsing file can be set to any Java project component. In this embodiment, the Java component that implements the association and binding of the regular parsing configuration file and the manufacturer's log format is the risk component, such as Figure 2 As shown, the process of the risk component performing regular parsing object collection is specifically as follows: the mapping relationship setting unit in the risk component is used to set the mapping relationship between the device IP (device IP is used to describe different log formats) and the regular configuration parsing file, so as to realize the binding of the device IP and the regular configuration parsing file; the regular parsing object collection loading unit in the risk component is used to load all regular configuration parsing files and convert them into regular parsing object collections; in this embodiment, the regular parsing object collection is a regular parsing object hash collection, which is convenient for the subsequent parsing to use the hash matching method to match the regular parsing object. When the distributed system performs log file parsing, it receives the regular expression parsing object collection sent by the java component through the activeMq message queue. There is a dedicated storage space in the distributed system for storing the regular parsing object collection, which is directly used by the subsequent log processing unit to parse the log.

[0054] The low-coupling structural setting of this embodiment makes log parsing highly efficient and scalable, which is very suitable for connecting to new business scenarios and products, can quickly adapt to different log formats, and is easy to use.

[0055] Get the log file to be parsed, specifically:

[0056] Get the path addresses of all log files, filter the log files by file name, and select the log files to be analyzed.

[0057] Preferably, the log file is cut into a plurality of sub-files, and before reading and loading each of the sub-files, the method further comprises:

[0058] Determine whether the log file is a compressed file. If so, decompress the log file and then move the decompressed log file to a temporary parsing directory. Otherwise, directly move the log file to a temporary parsing directory.

[0059] After obtaining the log file, some preprocessing is performed on the log file, namely, decompression, file movement, etc.

[0060] Preferably, the log file is cut into multiple sub-files, and each sub-file is read and loaded, specifically:

[0061] Determine whether the total number of lines of the log file is greater than a set number of lines, if so, cut the log file into multiple sub-files with the set number of lines, otherwise do not cut;

[0062] Obtain the path address of each sub-file;

[0063] The sub-files are read in parallel or serially based on the path address and loaded into a memory buffer.

[0064] In this embodiment, it is first determined whether the log file is larger than 500,000 lines. If it is larger, the log file is cut into multiple sub-files with 500,000 lines each; then the path address of the cut sub-files is obtained, and the sub-files are read and loaded in parallel or serially.

[0065] Specifically, Figure 3As shown, the specific process of the method is: S101, obtain and store a set of regular parsing objects from the risk component; S102, obtain the log file to be analyzed; S201, before cutting the log file, determine whether the log is a compressed file, if so, go to step S202, otherwise go to step S203; S202, decompress the compressed file; S203, move the log file to a temporary directory; S204, determine whether the log file is greater than the set number of lines, if so, go to step S205, otherwise go to step S206; S205, cut the log file; S206, load and read the log file; S3, put the loaded sub-files into a queue and distribute them to each processing unit; S4, each processing unit parses the sub-files; S501, forms a .sgn file based on the parsing result; S502, stores the .sgn file in the warehouse.

[0066] Preferably, the set number of rows and the size of the memory buffer are set according to configuration parameters of the distributed system.

[0067] The size of the cut sub-file (i.e., the set number of lines) and the size of the memory buffer for loading the sub-file can be set according to the distributed system configuration to achieve the optimal performance configuration. Specifically, the higher the configuration of the distributed system, the stronger its computing processing capability, so the set number of lines can be set larger, and the memory buffer can also be set larger accordingly.

[0068] Preferably, distributing the read and loaded sub-files to multiple processing units in parallel also includes:

[0069] Each of the processing units distributes the log data in the corresponding sub-file to a processing unit at the same level or a processing unit at the next level.

[0070] Specifically, Figure 4 As shown in the figure, in a distributed system, data can be exchanged between processing units to form a data stream. Therefore, when each processing unit is parsing logs, if the processing time is too long or the log files are unevenly distributed, the sub-files that cannot be processed can be sent to the processing units at the same level or the next level for processing, thereby further improving the efficiency of log parsing. Figure 4 The thin solid arrows indicate the data flow between processing units at the same level, and the thick solid arrows indicate the data flow between upper and lower level processing units.

[0071] Preferably, each processing unit matches the regular parsing object of the corresponding sub-file in the regular parsing object set, and performs parsing of the corresponding sub-file, specifically:

[0072] Each processing unit preliminarily parses the corresponding sub-file, obtains the log header IP, and matches the corresponding regular parsing object in the regular parsing object set through the log header IP;

[0073] Each processing unit performs parsing field matching on the corresponding sub-file according to the matched regular parsing object, and if the match is successful, extracts the parsing field data of the corresponding sub-file;

[0074] The parsing field data obtained by matching each processing unit is combined to form a parsing result.

[0075] Specifically, Figure 5 As shown, when the regular parsing object is matched in this embodiment: S401, first obtain the sub-file and its log header IP; S402, obtain the log source of the sub-file through the header IP, and then match the regular object set corresponding to the log header IP of the sub-file in the regular parsing object set. If the match is successful, go to step S404, otherwise go to step S403; S403, discard the current sub-file and record the abnormal log; S404, perform a cyclic match between the sub-file and the regular object in the corresponding regular parsing object set. If the match is successful, go to step S405, otherwise go to step S403; S405, extract the regular object that has been matched successfully, and return the matched parsing field data; S406, after the current sub-file is parsed, go to the next sub-file for parsing until all sub-files of the log file are parsed.

[0076] A regular expression describes a string matching pattern, which can be used to check whether a string contains a certain substring, replace a matching substring, or extract a substring that meets a certain condition from a string. The regular expression itself has the function of successfully matching and capturing the corresponding matching data (i.e., parsed field data); if the regular expression in the regular parsing object successfully matches the log file, the predefined parsing fields in the log file will be extracted.

[0077] This embodiment is implemented by matching the IP address of the log header, and the IP address matching method has high matching efficiency.

[0078] In this embodiment, the parsing process of the processing unit includes: using regular data in the sub-file, performing regular verification on the regular data based on a regular expression, and after the regular verification passes, performing subscript matching on the regular data that passes the verification, and performing type processing and / or internal and external network distinction on the regular data that passes the verification and subscript matching; generating a parsing result set based on the regular data after data type processing and / or internal and external network distinction.

[0079] Specifically, the regular data in the sub-file is used as follows: the header IP is used to collect the regular data of the sub-file from the regular data of the globally existing serialized log. The type processing of the regular data is as follows: the IP of the sub-file is converted into a long type and / or the protocol number of the sub-file is mapped to the corresponding protocol type.

[0080] It should be understood that the parsing method provided in this embodiment is applicable to log files generated by various types of different devices / systems.

[0081] Example 2

[0082] Embodiment 2 of the present invention provides a log file fast parsing device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the log file fast parsing method provided in embodiment 1 is implemented.

[0083] Specifically, Figure 6 As shown, the processor in this embodiment is divided into several functional modules: a file acquisition unit 101, a file cutting unit 102, a file reading unit 103, a data distribution unit 104, a plurality of processing units 105, and a storage writing unit 106. The file acquisition unit 101 is used to obtain the path addresses of all log files, filter by file name, and find all log files that need to be analyzed; the file cutting unit 102 cuts a single super-large log file into multiple sub-files; the file reading unit 103 obtains the path addresses of the cut sub-files, and reads and loads them in parallel or serially; the data distribution unit 104 distributes the data read and loaded by the file reading unit 103 in parallel to the subsequent processing unit 105. The processing and parsing unit 105 outputs the result set to form a .sgn suffix file to be stored in the library; the storage writing unit 106 performs a storage writing operation on the result set output by the processing and parsing unit 105, and writes the parsing result into the persistent data storage 2 for use.

[0084] The log file rapid parsing device provided in the embodiment of the present invention is used to implement the log file rapid parsing method. Therefore, the log file rapid parsing device also has the technical effects possessed by the log file rapid parsing method, which will not be described in detail here.

[0085] Example 3

[0086] Embodiment 3 of the present invention provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the log file fast parsing method provided in Embodiment 1 is implemented.

[0087] The computer storage medium provided in the embodiment of the present invention is used to implement the log file fast parsing method. Therefore, the technical effects possessed by the log file fast parsing method are also possessed by the computer storage medium, which will not be described in detail here.

[0088] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for rapid parsing of log files, characterized in that: Applied to distributed systems, it includes the following steps: Obtain and store a collection of regular expression parsing objects, and obtain the log file to be parsed; Cut the log file into multiple sub-files, and read and load each of the sub-files; Distribute the read and loaded sub-files to multiple processing units in parallel; Each processing unit matches the regular parsing object of the corresponding sub-file in the regular parsing object set, and performs parsing of the corresponding sub-file; The analysis results of each processing unit are stored in the database; Get and store a collection of regular expression parsing objects, specifically: Set up a dedicated storage space, receive the regular parsing object set sent by the java component through the activeMq message queue, and store it in the dedicated storage space, wherein the parsing object set is configured in the java component, and the association and binding relationship between the regular configuration file and the manufacturer's log format is implemented through the risk component; Among them, the implementation process includes: the mapping relationship setting unit in the risk component is used to set the mapping relationship between the device IP and the regular configuration parsing file to realize the binding of the device IP and the regular configuration parsing file; the regular parsing object set loading unit in the risk component is used to load all regular configuration parsing files and convert them into a regular parsing object set.

2. The log file rapid parsing method according to claim 1, characterized in that: Also includes: After completing the parsing process of all sub-files of a log file, the system sleeps for a set time and then performs the parsing process of the next log file.

3. The log file rapid parsing method according to claim 1, characterized in that: The log file is cut into a plurality of sub-files, and before reading and loading each of the sub-files, the method further includes: Determine whether the log file is a compressed file. If so, decompress the log file and then move the decompressed log file to a temporary parsing directory. Otherwise, directly move the log file to a temporary parsing directory.

4. The log file rapid parsing method according to claim 1, characterized in that: The log file is cut into multiple sub-files, and each sub-file is read and loaded, specifically: Determine whether the total number of lines of the log file is greater than a set number of lines, if so, cut the log file into multiple sub-files with the set number of lines, otherwise do not cut; Obtain the path address of each sub-file; The sub-files are read in parallel or serially based on the path address and loaded into a memory buffer.

5. The log file rapid parsing method according to claim 4, characterized in that: The set number of rows and the size of the memory buffer are set according to the configuration parameters of the distributed system.

6. The log file rapid parsing method according to claim 1, characterized in that: Distribute the read and loaded sub-files to multiple processing units in parallel, and also include: Each of the processing units distributes the log data in the corresponding sub-file to a processing unit at the same level or a processing unit at the next level.

7. The log file rapid parsing method according to claim 1, characterized in that: Each processing unit matches the regular parsing object of the corresponding sub-file in the regular parsing object set, and performs parsing of the corresponding sub-file, specifically: Each processing unit preliminarily parses the corresponding sub-file, obtains the log header IP, and matches the corresponding regular parsing object in the regular parsing object set through the log header IP; Each processing unit performs parsing field matching on the corresponding sub-file according to the matched regular parsing object, and if the match is successful, extracts the parsing field data of the corresponding sub-file; The parsing field data obtained by matching each processing unit is combined to form a parsing result.

8. A log file rapid analysis device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method for rapid parsing of log files according to any one of claims 1 to 7 is implemented.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the log file rapid parsing method according to any one of claims 1 to 7 is implemented.

Images