Log file compression package parsing method, device, equipment and medium

Abstract

The embodiment of the present invention discloses a method, device, equipment and medium for parsing a log file compression package. The method comprises: reading a log file compression package through a file reading class pre-created based on Flume; wherein the log file compression package includes multiple bcp files; parsing the row data of the log file compression package through a custom deserializer in the file reading class, and storing the parsing result in a message queue. The above technical solution realizes the parsing of the log file compression package based on Flume.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method, device, equipment and medium for parsing an encrypted compressed package. Background Art

[0002] The log collection system (Flume) is a distributed, highly reliable, and highly available system for collecting, aggregating, and transmitting massive logs. It supports the customization of various data senders in the log system for collecting data. At the same time, Flume also has the ability to perform simple processing and analysis on data.

[0003] At present, the log files analyzed by Flume are usually uncompressed original log files, and log files in the form of compressed packages cannot be directly parsed by Flume. Therefore, how to implement the parsing of log file compressed packages based on Flume is an urgent problem to be solved. Summary of the invention

[0004] The embodiment of the present invention provides a log file compressed package parsing method, device, equipment and medium, so as to implement the parsing of the log file compressed package based on Flume.

[0005] In a first aspect, an embodiment of the present invention provides a method for parsing a log file compression package, comprising:

[0006] Reading a log file compression package through a file reading class pre-created based on Flume; wherein the log file compression package includes multiple bcp files;

[0007] The row data of the log file compression package is parsed through the custom deserializer in the file reading class, and the parsing result is stored in the message queue.

[0008] In a second aspect, an embodiment of the present invention further provides a log file compression package parsing device, comprising:

[0009] A log file compression package reading module is configured to read the log file compression package through a file reading class pre-created based on Flume; wherein the log file compression package includes multiple bcp files;

[0010] The log file compression package line data parsing module is configured to perform line data parsing on the log file compression package through a custom deserializer in the file reading class, and store the parsing results in a message queue.

[0011] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for parsing a compressed log file package as described in any embodiment of the present invention is implemented.

[0012] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for parsing a compressed log file package as described in any embodiment of the present invention.

[0013] In the technical solution provided by the embodiment of the present invention, a log file compression package including multiple bcp files is read by a file reading class pre-created based on Flume, and then the row data of the log file compression package is parsed by a custom deserializer in the file reading class, and the parsing result is stored in a message queue, thereby realizing the parsing of the log file compression package based on Flume. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of a method for parsing a log file compression package in the first embodiment of the present invention;

[0015] Figure 2a It is a flow chart of a method for parsing a log file compression package in Embodiment 2 of the present invention;

[0016] Figure 2b This is a schematic diagram of a process of real-time analysis of a log file compression package in Embodiment 2 of the present invention;

[0017] Figure 3 It is a structural schematic diagram of a log file compression package parsing device in Embodiment 3 of the present invention;

[0018] Figure 4 It is a schematic diagram of the hardware structure of a computer device in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0020] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0021] Embodiment 1

[0022] Figure 1 It is a flowchart of a log file compression package parsing method provided in the first embodiment of the present invention, which can be applicable to the case of parsing a log file compression package including multiple bcp files based on Flume. The method can be executed by the log file compression package parsing device provided in the embodiment of the present invention. The device can be implemented in software and / or hardware and can generally be integrated in a computer device.

[0023] like Figure 1 As shown, the method for parsing a log file compression package provided in this embodiment specifically includes:

[0024] S110. Read a compressed log file package through a file reading class pre-created based on Flume; wherein the compressed log file package includes multiple bcp files.

[0025] Flume refers to a highly available, highly reliable, distributed system for collecting, aggregating, and transmitting massive logs. It supports the customization of various types of data senders in the log system to collect data. In addition, Flume can simply process the data and write it to various types of data receivers.

[0026] File reading class refers to the subclass of java.io.InputStreamReader, which is a convenient class for reading character files and character streams. Among them, InputStreamReader is a subclass of java.io.Reader, which is a bridge from byte stream (file) to character stream. It can read bytes using the specified character set and decode them into characters. Reader is an abstract class (abstract class refers to a class with abstract methods, and a class containing one or more abstract methods must be declared as abstract), and its function is to read character streams.

[0027] In this application, the pre-created file reading class is used to read the log file compression package with a set compression format. For example, if the compression format of the log file compression package is zip, a file reading class (ZipFileEventReader) corresponding to the zip compression format can be created in advance. After creating the file reading class ZipFileEventReader, implement the file reading interface (ReliableEventReader) in Flume.

[0028] Log files refer to the recording files or file collections used to record system operation events. For example, the operating system contains operating system log files, the database system contains database system log files, and so on. Different log files can record different information. For example, some are default system log files, and some are log files that record specific tasks.

[0029] A log file compression package refers to a compressed data package containing multiple log files pushed by a third-party system. The log file compression package can exist in zip format, and the log file can be a bcp file. The log file compression package can include multiple bcp files. Among them, a bcp file is a file that uses a "\t" tab character to separate attributes and a "\r" separator for line breaks. This file is named with a ".bcp" extension.

[0030] The file reading class pre-created based on Flume can be used to read log file compression packages that include multiple bcp files.

[0031] Optionally, reading the compressed log file package by using a file reading class pre-created based on Flume may include: obtaining a compressed package storage path in a Flume configuration file by using the file reading class, and reading the compressed log file package according to the compressed package storage path.

[0032] Flume configuration files (such as conf.properties) refer to files used to store configuration information or configuration data in Flume.

[0033] The compressed package storage path refers to the storage location or storage address in the system where the compressed package of log files pushed by the third-party system is stored.

[0034] When reading a compressed log file package through a file reading class pre-created based on Flume, the compressed package storage path in the Flume configuration file can be obtained through the file reading class, and the compressed log file package can be read according to the compressed package storage path.

[0035] S120. The row data of the log file compression package is parsed through the custom deserializer in the file reading class, and the parsing result is stored in the message queue.

[0036] Deserializer refers to the application program interface corresponding to the serializer. The deserialization process is the reverse process of the serialization process. The serialization process is the process of converting the data type of data into a uniform data type. The deserialization process is the process of restoring data from a uniform data type to the original data type of the data. In the embodiment of the present invention, the so-called custom deserializer (such as ZipEventDeserializer) refers to an interface in the file reading class pre-created based on Flume that can convert a log file compression package into row data, so as to parse the row data of the log file compression package.

[0037] The parsing result refers to the row data result obtained after parsing multiple bcp files included in the log file compression package line by line.

[0038] A message queue refers to a container that can store data during the data transmission process, such as ActiveMQ message queue, Kafka message queue, etc.

[0039] After reading the log file compression package including multiple bcp files through the file reading class pre-created based on Flume, the row data of the log file compression package can be parsed through the custom deserializer in the file reading class, and the parsing results are stored in the message queue.

[0040] Optionally, the message queue may be a Kafka message queue; after storing the parsing result in the message queue, the method may further include: performing real-time analysis on the parsing result in the Kafka message queue through a Kafka consumer program.

[0041] Kafka message queue is a distributed message queue that can process active streaming data, such as page views (PV), unique visitors (UV), etc.

[0042] Kafka consumer program refers to the program called by the Kafka message queue when processing and analyzing data in real time.

[0043] After parsing the row data of the log file compression package through the custom deserializer in the file reading class, the parsing results can be stored in the Kafka message queue, and then the parsing results in the Kafka message queue can be analyzed in real time through the Kafka consumer program.

[0044] Furthermore, storing the parsing result in the message queue through the custom deserializer may include: obtaining the Kafka message queue topic name in the Flume configuration file through the custom deserializer, and storing the parsing result in the Kafka message queue according to the Kafka message queue topic name.

[0045] The Kafka message queue topic name (Topic) refers to the name of the collection of similar data in the Kafka message queue. By specifying the Kafka message queue topic name, the parsing results can be written to the collection of the corresponding topic name, and the data in the collection of the corresponding topic name can also be read according to the Kafka message queue topic name.

[0046] The Kafka message queue topic name in the Flume configuration file is obtained through a custom deserializer. Then, the parsing results obtained after parsing the row data of the log file compression package can be stored in the specified location in the Kafka message queue according to the Kafka message queue topic name. Then, the parsing results in the Kafka message queue can be analyzed in real time through the Kafka consumer program.

[0047] The technical solution provided by the embodiment of the present invention reads a log file compression package including multiple bcp files through a file reading class pre-created based on Flume, and then parses the row data of the log file compression package through a custom deserializer in the file reading class, and stores the parsing results in a message queue, thereby realizing the parsing of the log file compression package based on Flume.

[0048] Embodiment 2

[0049] Figure 2a This is a flow chart of a log file compression package parsing method provided by the second embodiment of the present invention. This embodiment is specific based on the above embodiment, wherein the row data of the log file compression package can be parsed by a custom deserializer in the file reading class, specifically:

[0050] The log file compression package is passed from the file reading interface of the file reading class to the deserializer interface of the custom deserializer through the constructor of the custom deserializer;

[0051] After obtaining the log file compression package from the deserializer interface through the custom deserializer, the log file compression package is decompressed to obtain multiple bcp files, and the row data of the multiple bcp files are parsed.

[0052] Furthermore, before decompressing the compressed log file package to obtain multiple bcp files through the custom deserializer, the following steps may also be included:

[0053] If the log file compression package is an encrypted compression package, the decompression password in the Flume configuration file is obtained through a custom deserializer, and the log file compression package is decrypted according to the decompression password.

[0054] like Figure 2a As shown, the present embodiment provides a method for parsing a log file compression package, which specifically includes:

[0055] S210. Read a compressed log file package through a file reading class pre-created based on Flume; wherein the compressed log file package includes multiple bcp files.

[0056] S220. Transfer the log file compression package from the file reading interface of the file reading class to the deserializer interface of the custom deserializer through the constructor of the custom deserializer.

[0057] A constructor refers to a function with initialized variables or a series of operation methods, which can be used to initialize an object or specify the operation process of an object when creating an object. In an embodiment of the present invention, the constructor can be used to specify that the log file compression package is passed to the deserializer interface of the custom deserializer to parse the row data of the log file compression package after the log file compression package is read from the file reading interface of the file reading class.

[0058] A file reading interface (such as ReliableEventReader) refers to an interface in Flume that can be used to read a compressed package of log files, which is implemented by a file reading class pre-created based on Flume.

[0059] The deserializer interface (such as EventDeserializer) refers to an interface provided by Flume that can be used to parse the compressed package of log files through a custom deserializer.

[0060] The log file compression package is passed from the file reading interface of the file reading class to the deserializer interface of the custom deserializer through the constructor of the custom deserializer to parse the read log file compression package.

[0061] S230. After obtaining the log file compression package from the deserializer interface through the custom deserializer, determine whether the log file compression package is an encrypted compression package. If yes, execute S240; otherwise, execute S250.

[0062] An encrypted compressed package refers to a compressed package obtained by encrypting a compressed package of a log file.

[0063] After obtaining the log file compression package from the deserializer interface through the custom deserializer, further determine whether the log file compression package is an encrypted compression package. If the log file compression package is an encrypted compression package, execute S240, that is, decrypt the log file compression package and then parse the row data. If the log file compression package is not an encrypted compression package, execute S250, that is, directly parse the row data of the log file compression package.

[0064] S240. Obtain the decompression password in the Flume configuration file through the custom deserializer, and decrypt the log file compression package according to the decompression password.

[0065] The decompression password refers to the password required to decrypt an encrypted compressed package.

[0066] Decryption refers to canceling the security and confidentiality of the log file compression package and decrypting the log file compression package based on the decompression password. The log file compression package can be decrypted using any open source decryption tool in the prior art. For example, the net.lingala.zip4j.ZipFile decryption tool can be used to decrypt the log file compression package in zip format.

[0067] When the log file compression package is an encrypted compression package, the decompression password in the Flume configuration file is obtained through a custom deserializer, and the log file compression package is decrypted according to the decompression password to obtain the decrypted log file compression package.

[0068] It should be pointed out that when decrypting a log file compression package, if a decryption error occurs, the custom deserializer can record an error log for the log file compression package that failed to be decrypted, and then the error log record can be manually viewed to analyze the cause of the decryption error, such as decompression password configuration problems, problems with the log file compression package, etc.

[0069] S250. Decompress the log file compression package through a custom deserializer to obtain multiple bcp files, parse the row data of the multiple bcp files, and store the parsing results in a message queue.

[0070] After obtaining the decompression password in the Flume configuration file through the custom deserializer and decrypting the encrypted compressed log file package according to the decompression password, or after obtaining the compressed log file package that is not an encrypted compressed log file package from the deserializer interface through the custom deserializer, the compressed log file package is decompressed through the custom deserializer to obtain multiple bcp files included in the compressed log file package, and then the multiple bcp files are parsed for row data, and the parsing results are stored in the message queue.

[0071] Optionally, parsing row data of multiple bcp files by using a custom deserializer may include: traversing the multiple bcp files by using the custom deserializer, and for each bcp file, reading the file content row by row using a file reading object.

[0072] The file reading object refers to the Java built-in file reading object Reader, which is an abstract class. java.io.Reader is the parent class of all character input streams. This class defines the basic method of reading data in units of characters. The file reading object can be used to read the line file content contained in each bcp file obtained by decompressing the log file compression package line by line.

[0073] After decompressing the log file compression package through the custom deserializer to obtain multiple bcp files included in the log file compression package, the multiple bcp files are parsed for row data. When parsing the row data, the custom deserializer can traverse multiple bcp files, obtain each bcp file in the log file compression package in turn, and then use the file reading object to read the content contained in the bcp file line by line for each bcp file to obtain all the row contents of each bcp file.

[0074] As an optional implementation, the message queue is a Kafka message queue; after storing the parsing results in the message queue, the method may further include: performing real-time analysis on the parsing results in the Kafka message queue through a Kafka consumer program.

[0075] Furthermore, storing the parsing result in the message queue through the custom deserializer may include: obtaining the Kafka message queue topic name in the Flume configuration file through the custom deserializer, and storing the parsing result in the Kafka message queue according to the Kafka message queue topic name.

[0076] Figure 2b is a flow chart of a real-time analysis of a log file compression package provided by an embodiment of the present invention. In a specific example, Figure 2bAs shown, after obtaining the log file compression package from the deserializer interface through the custom deserializer, the log file compression package is decompressed by the custom deserializer to obtain multiple bcp files included in the log file compression package, and the multiple bcp files are parsed for row data. Then, the Kafka message queue topic name in the Flume configuration file is obtained through the custom deserializer. According to the Kafka message queue topic name, the parsing result obtained after parsing the log file compression package for row data can be stored in a specified location in the Kafka message queue. Then, the Kafka consumer program can be used to perform real-time consumption analysis on the parsing result stored in the Kafka message queue, thereby realizing the parsing of the log file compression package based on Flume, and real-time detection and feedback on the effect of pushing data to the third-party system can be performed.

[0077] For details not explained in this embodiment, please refer to the aforementioned embodiments and will not be repeated here.

[0078] The above technical solution reads a log file compression package including multiple bcp files through a file reading class pre-created based on Flume, and then passes the log file compression package from the file reading interface of the file reading class to the deserializer interface of the custom deserializer through the constructor of the custom deserializer, and determines whether the log file compression package is an encrypted compression package through the custom deserializer. If the log file compression package is an encrypted compression package, it is also necessary to obtain the decompression password in the Flume configuration file through the custom deserializer, and first decrypt the log file compression package according to the decompression password, and then decompress the log file compression package through the custom deserializer to obtain multiple bcp files. If the log file compression package is not an encrypted compression package, the log file compression package can be directly decompressed through the custom deserializer to obtain multiple bcp files. Secondly, the multiple bcp files obtained after the log file compression package is decompressed by the custom deserializer are parsed for row data, and the parsing results are stored in the message queue, thereby realizing automatic decryption and parsing of multiple log file compression packages under the specified path based on Flume, and real-time detection and feedback on the data push effect of the third-party system can be performed.

[0079] Embodiment 3

[0080] Figure 3 It is a structural diagram of a log file compression package parsing device provided in Example 3 of the present invention, which can be applicable to the situation of parsing a log file compression package including multiple bcp files based on Flume. The device can be implemented in software and / or hardware and can generally be integrated in a computer device.

[0081] like Figure 3As shown, the log file compressed package parsing device specifically includes: a log file compressed package reading module 310 and a log file compressed package line data parsing module 320. Among them,

[0082] The log file compression package reading module 310 is configured to read the log file compression package through a file reading class pre-created based on Flume; wherein the log file compression package includes multiple bcp files;

[0083] The log file compressed package line data parsing module 320 is configured to parse the line data of the log file compressed package through the custom deserializer in the file reading class, and store the parsing result in the message queue.

[0084] The technical solution provided by the embodiment of the present invention reads a log file compression package including multiple bcp files through a file reading class pre-created based on Flume, and then parses the row data of the log file compression package through a custom deserializer in the file reading class, and stores the parsing results in a message queue, thereby realizing the parsing of the log file compression package based on Flume.

[0085] Optionally, the log file compression package line data parsing module 320 includes: a log file compression package transmission unit and a line data parsing unit, wherein:

[0086] A log file compression package delivery unit, configured to deliver the log file compression package from the file reading interface of the file reading class to the deserializer interface of the custom deserializer through the constructor of the custom deserializer;

[0087] The row data parsing unit is configured to, after obtaining the log file compression package from the deserializer interface through the custom deserializer, decompress the log file compression package to obtain the multiple bcp files, and perform row data parsing on the multiple bcp files.

[0088] Optional, row data parsing unit, specifically set to:

[0089] The multiple bcp files are traversed by the custom deserializer, and for each bcp file, the file content is read line by line using a file reading object.

[0090] Optionally, the log file compression package line data parsing module 320 further includes: a log file compression package decryption unit, wherein the log file compression package decryption unit is configured to:

[0091] Before the log file compression package is decompressed by the custom deserializer to obtain the multiple bcp files, if the log file compression package is an encrypted compression package, the decompression password in the Flume configuration file is obtained by the custom deserializer, and the log file compression package is decrypted according to the decompression password.

[0092] Optionally, the log file compression package reading module 310 is specifically configured as follows:

[0093] The compressed package storage path in the Flume configuration file is obtained through the file reading class, and the log file compressed package is read according to the compressed package storage path.

[0094] Optionally, the message queue is a Kafka message queue;

[0095] The above-mentioned device also includes: a real-time analysis module for analysis results, wherein the real-time analysis module for analysis results is configured to: after storing the analysis results in the message queue, perform real-time analysis on the analysis results in the Kafka message queue through a Kafka consumer program.

[0096] Optionally, the log file compression package line data parsing module 320 is specifically configured as follows:

[0097] The row data of the log file compression package is parsed, the Kafka message queue topic name in the Flume configuration file is obtained through the custom deserializer, and the parsing result is stored in the Kafka message queue according to the Kafka message queue topic name.

[0098] The above-mentioned log file compressed package parsing device can execute the log file compressed package parsing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the log file compressed package parsing method.

[0099] Embodiment 4

[0100] Figure 4 A schematic diagram of the hardware structure of a computer device provided in Embodiment 4 of the present invention. Figure 4 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0101] like Figure 4As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0102] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0103] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0104] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0105] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0106] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0107] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a log file compression package parsing method provided in an embodiment of the present invention. That is, when the processing unit executes the program, the following is implemented:

[0108] Reading a log file compression package through a file reading class pre-created based on Flume; wherein the log file compression package includes multiple bcp files;

[0109] The row data of the log file compression package is parsed through the custom deserializer in the file reading class, and the parsing result is stored in the message queue.

[0110] Embodiment 5

[0111] Embodiment 5 of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a method for parsing a log file compression package provided in all embodiments of the present invention is implemented: that is, when the program is executed by a processor, the following is implemented:

[0112] Reading a log file compression package through a file reading class pre-created based on Flume; wherein the log file compression package includes multiple bcp files;

[0113] The row data of the log file compression package is parsed through the custom deserializer in the file reading class, and the parsing result is stored in the message queue.

[0114] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0115] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0116] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0117] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or combinations thereof, including object-oriented programming languages ​​(such as Java, Smalltalk, C++), and conventional procedural programming languages ​​(such as "C" language or similar programming languages). The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0118] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for parsing a compressed log file, characterized in that: include: Reading a log file compression package through a file reading class pre-created based on the log collection system Flume; wherein the log file compression package includes multiple bcp files; The row data of the log file compression package is parsed through the custom deserializer in the file reading class, and the parsing result is stored in the message queue; The row data of the log file compression package is parsed through the custom deserializer in the file reading class, including: The log file compression package is transferred from the file reading interface of the file reading class to the deserializer interface of the custom deserializer through the constructor of the custom deserializer; After the log file compression package is obtained from the deserializer interface through the custom deserializer, the log file compression package is decompressed to obtain the multiple bcp files, and row data parsing is performed on the multiple bcp files.

2. The method according to claim 1, characterized in that The row data of the multiple bcp files are parsed by the custom deserializer, including: The multiple bcp files are traversed by the custom deserializer, and for each bcp file, the file content is read line by line using a file reading object.

3. The method according to claim 1, characterized in that Before decompressing the log file compression package by the custom deserializer to obtain the multiple bcp files, the method further includes: If the log file compression package is an encrypted compression package, the decompression password in the Flume configuration file is obtained through the custom deserializer, and the log file compression package is decrypted according to the decompression password.

4. The method according to claim 1, characterized in that: Read the compressed log file package through the file reading class pre-created based on Flume, including: The compressed package storage path in the Flume configuration file is obtained through the file reading class, and the log file compressed package is read according to the compressed package storage path.

5. The method according to claim 1, characterized in that The message queue is a Kafka message queue; after storing the parsing result in the message queue, it also includes: The parsing results in the Kafka message queue are analyzed in real time through the Kafka consumer program.

6. The method according to claim 5, characterized in that The parsing result is stored in the message queue through the custom deserializer, including: The Kafka message queue topic name in the Flume configuration file is obtained through the custom deserializer, and the parsing result is stored in the Kafka message queue according to the Kafka message queue topic name.

7. A log file compression package parsing device, characterized in that: include: A log file compression package reading module is configured to read the log file compression package through a file reading class pre-created based on Flume; wherein the log file compression package includes multiple bcp files; A log file compressed package line data parsing module is configured to parse the line data of the log file compressed package through a custom deserializer in the file reading class, and store the parsing result in a message queue; The log file compression package line data parsing module includes: a log file compression package transmission unit and a line data parsing unit, wherein: The log file compression package delivery unit is configured to deliver the log file compression package from the file reading interface of the file reading class to the deserializer interface of the custom deserializer through the constructor of the custom deserializer; The row data parsing unit is configured to, after obtaining the log file compression package from the deserializer interface through the custom deserializer, decompress the log file compression package to obtain the multiple bcp files, and perform row data parsing on the multiple bcp files.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images