Method and information technology system for compressing and decompressing log files

Abstract

The present invention relates to a method for compressing and decompressing log files using a customizable dictionary (1), characterized by the following method steps: monitoring runtime components (2.1, 4) by a monitoring unit (2) and generating a source log file (U-Log); reading the source log file (U-Log) into a log encoder (2.2); initializing a coding table (5) from the versioned dictionary (1) or the source log file (U-Log); applying the coding table (5) to the source log file (U-Log) to generate an encoded log file (KOD-Log); reading the encoded log file (KOD-Log) from the monitoring unit (2). The decoded log file (KOD-Log) is transmitted to the decode unit (3); the decode unit (3) reads the encoded log file (KOD-Log) into the log decoder (3.1) and receives from the monitoring unit (2) a version indication of the version-controlled dictionary (1) or coding table (5) used by the log encoder (2.2); the dictionary database (6.1) is accessed and the dictionary (1) corresponding to the version indication is provided to the log decoder (3.1), and the log decoder (3.1) initializes the coding table from the version-controlled dictionary (1); and the coding table (5) is applied to the encoded log file (KOD-Log) to generate a decoded log file (DEK-Log).

Description

[Technical Field]

[0001] The present invention relates to a method for compressing and decompressing log files using a customizable dictionary, and to an information technology system for implementing the method. [Background technology]

[0002] Information technology systems, such as PCs or embedded systems, consist of hardware and software components. During the development of such systems, it may be necessary to record and analyze the behavior of these hardware and software components during operation in order to check their proper functioning or to detect errors. This also includes so-called "debugging." The behavior of the corresponding hardware and / or software components is recorded in the form of so-called log messages or log files. Log files can be generated by the relevant hardware or software components themselves or by a unit monitoring the corresponding components. Subsequent analysis of the corresponding log files can be achieved by various programs, for example by manual analysis using an editor.

[0003] Depending on the complexity of the hardware and / or software components to be monitored and the complexity of the functions provided by those components, the corresponding log files may be of different sizes, i.e., the storage capacity required on the storage medium may be relatively small or relatively large. Therefore, if the data transmission speed is limited, it may take a long time to transmit the log files between systems and require a large amount of storage capacity on the respective storage medium. Therefore, it is desirable to reduce the size of the corresponding log files.

[0004] Various compression and decompression algorithms or methods are known in the IT field. For example, zlib is a free program library for compressing and decompressing data using the Deflate algorithm. This algorithm is lossless, as opposed to lossy, compression. The Deflate algorithm is a so-called dictionary method that converts the contents of a log file—that is, a series of characters, typically strings—into shorter strings by taking into account the contents of a respective dictionary. Such dictionaries contain entries for sequences of characters that typically and particularly frequently occur in log files. These entries define which strings should be used to replace each of these sequences. For example, a sequence of characters like "ABCDE" can be replaced with the string "a." This reduces the content of the log file and, accordingly, the memory requirements.

[0005] Another data compression algorithm is "Brotli", which is based on a predefined dictionary of 120 KiB in size. However, this data compression algorithm has the disadvantage that due to the fixed dictionary, only certain predefined sequences of characters can be shortened in length.

[0006] Unlike dictionaries with fixed, predetermined sizes and contents, customizable dictionaries are also known. A data compression method based on this is known, for example, from German Patent No. 4340591. This publication describes a data compression initialization method that can be adapted to various types of input data and various data structures in order to improve the data compression rate and reduce the storage capacity required in a dictionary-based compression / decompression system. In this data compression initialization method, entries in the corresponding dictionary are generated depending on the input data to be compressed. This means that the dictionary contents are completed, so that input data that was initially unknown to the dictionary can also be compressed. The corresponding contents can also be coded multiple times.

[0007] Furthermore, U.S. Patent Application Publication No. 2015 / 0066881 discloses a method for compressing log files, which enables efficient collection of data from network nodes. In this method, a table-based compression algorithm is used to compress the log files. The underlying tables are generated using special training data, which allows for the construction of an extensive translation database.

[0008] Additionally, U.S. Patent Application Publication No. 2021 / 0407220 discloses a system, method, and apparatus for managing data collection in a vehicle, which describes the generation of a log file in a computing unit of the vehicle and the processing of the log file in a computing unit within or external to the vehicle. Summary of the Invention [Problem to be solved by the invention]

[0009] It is an object of the present invention to provide an improved method for compressing and decompressing log files, which reduces the effort involved in processing the log files and allows for relatively significant reductions in memory requirements or file sizes for various output files. [Means for solving the problem]

[0010] According to the present invention, in a method for compressing and decompressing log files using a customizable dictionary, the following method steps are performed: - monitoring the runtime components by a monitoring unit and generating a source log file, the source log file recording the behavior of the runtime components described during the monitoring; -The monitoring unit reads the source log file into the log encoder; -Log encoder initializes the coding table from a versioned dictionary or source log file; - a log encoder generates an encoded log file by applying a coding table to the source log file; - transmitting the encoded log file from the monitoring unit to the deployment unit; - by the decompression unit, causing the encoded log file to be read into the log decoder, and receiving from the monitoring unit a version indication of the versioned dictionary or coding table used by the log encoder; accessing a dictionary database containing different version-controlled dictionaries by an expansion unit, providing a dictionary corresponding to the version indication to a log decoder, and initializing a coding table from the version-controlled dictionaries by the log decoder; A log decoder applies a coding table to the encoded log file to generate a decoded log file.

[0011] Because customizable dictionaries are used, the method of the present invention allows for very significant data reduction. That is, each dictionary can be supplemented with entries to enable compression of previously unknown entries from source log files. The versioning of dictionaries significantly reduces processing effort. Depending on the content of each source log file, which depends on the monitored or monitoring hardware and software configuration and the features provided by the runtime environment, different versions of the dictionaries or the coding tables based on them are created. They are thus "customized" for each source log file. Thus, to compress a source log file or decompress a compressed source log file, it is sufficient to derive a coding table from the associated dictionary and apply that coding table to the respective log file for "translation." The use of version designations allows for unambiguous, reliable, and fast reference between each log file and the dictionary or coding table (for compressing the source log file or decompressing the encoded log file) that should be applied to that log file. A versioned dictionary and the coding table based on that dictionary have the same version.

[0012] The runtime component is formed by a hardware component and / or a software component. The runtime component can be implemented inside or outside the monitoring unit and is monitored by the monitoring unit. The runtime component may be, for example, a program running on a PC or an embedded system (or its hardware / software components) in a vehicle. The monitoring unit generates a source log file or receives a source log file from the runtime component. A log encoder is a component of the monitoring unit that reads the source log file.

[0013] The log encoder or monitoring unit then selects a versioned dictionary or coding table to encode the source log file that it has read in. This decision is made based on various boundary conditions, which are explained in more detail below.

[0014] A dictionary is a binary file, also called a binary file or simply "binary." To generate an encoded log file, the log encoder initializes each coding table from the corresponding versioned dictionary and applies the coding table to the source log file. Thus, each versioned dictionary stores the mapping of which character sequences in the source log file should be replaced with which strings. A string is also a sequence of characters, but it is shorter than the initial sequence. Therefore, the terms "sequence of characters" and "string" are chosen here for easier understanding. Depending on the source log file's content, the source log file may contain entirely different sequences of characters. This is taken into account by a separate version of the dictionary, so each versioned dictionary contains only the mappings of character sequences and strings that primarily appear in the respective source log file. This means that a dictionary (or a coding table initialized from / based on a dictionary) can be prevented from being applied to a source log file that contains mappings for sequences of characters that appear very rarely or not at all in the source log file. This also reduces the memory demands of the dictionary.

[0015] By translating the content of each source log file using a coding table (derived from a dictionary), i.e., by actual compression, the memory demands of the encoded log files are reduced compared to the source log files. Therefore, for the same transmission bandwidth, the encoded log files can be transmitted from the monitoring unit to the extraction unit faster than the transmission of the (uncompressed) source log files. Furthermore, cybersecurity is improved, since an attacker can only read the encoded log files transmitted in this way if he has access to the dictionary used or the coding table initialized from that dictionary.

[0016] The dictionary database may be located in the monitoring unit, the expansion unit, and / or externally. In particular, if the dictionary database is implemented externally to the monitoring unit and the expansion unit, and both the monitoring unit and the expansion unit have read or read and write access to the dictionary database, particularly efficient data transmission between the monitoring unit and the expansion unit is possible, since in this case only the encoded log file needs to be transmitted from the monitoring unit to the expansion unit, and the expansion unit receives the corresponding version-controlled dictionary from the dictionary database via a separate communication channel. In this case, the communication channel is preferably a communication channel with a relatively high data transmission rate.

[0017] The log decoder has information about which versioned dictionary it should use so that it can initialize the appropriate coding tables needed to decode the encoded log file. The version indication can be transmitted in various ways, which are also described below.

[0018] The decoded log file substantially or completely corresponds to the source log file, which allows the source log file to be processed in the deployment unit, for example, to determine whether the runtime component is functioning properly or to enable bug detection.

[0019] In an advantageous development of the method, the monitoring unit reads the source log file, generates a coding table depending on the information contained in the source log file, derives a dictionary from the coding table, provides a version indication for the dictionary, and transmits the versioned dictionary to a dictionary database for storage. This process allows for the generation of a dictionary customized for a particular source log file.

[0020] Thus, a properly versioned dictionary is suitable for encoding or decoding source log files generated depending on the hardware and / or software configuration of a runtime component. This is because the contents of the source log file may appear different depending on the hardware and / or software implementation of the runtime component or the function performed by the runtime component. The same application is often executed on the same hardware with the same software version under different external environment conditions. In this case, the same dictionary can be used. If new software is subsequently installed or a different function is performed, a different versioned dictionary can be used because the substantive contents of the source log file will change accordingly. Similarly, if the hardware of the runtime component is changed, a different versioned dictionary can be used.

[0021] The monitoring unit checks which version, i.e. which version indicator, is associated with each dictionary when it is created. In this way, the contents of the coding table or the dictionary built on it can be compared with dictionaries already stored in the dictionary database. If the same contents are found, the dictionary is of the same version. On the other hand, if no match is found, the dictionary is of a new version. In this case, the version indicators can be assigned according to a predefined pattern, for example consecutive numbers.

[0022] The reading of the source log file, the generation of the coding table, and the derivation of the dictionary from the coding table are performed by the monitoring unit, in particular, after the source log file is read into the log encoder, but before the versioned dictionary is read and the coding table is initialized by the log encoder. In this way, a versioned dictionary that can be read into and initialized in the log encoder can be generated by the monitoring unit before the source log file itself is encoded. This allows source log files with previously unknown content to be encoded if a versioned dictionary for that source log file does not yet exist in the dictionary database. Once the corresponding coding table and the corresponding versioned dictionary have been generated from the source log file, the coding table can be used directly for encoding, and the generated versioned dictionary can be transmitted to the dictionary database for storage.

[0023] In contrast, when source log files with familiar patterns or contents are processed, it is not necessary to first read and inspect the contents of the source log file each time, or to generate and store new coding tables and new versioned dictionaries from the coding tables as needed. Instead, a versioned dictionary appropriate for each source log file can be retrieved from a dictionary database. For example, the configuration of the runtime component being monitored and / or the functionality provided by the runtime component can also determine which versioned dictionary to use. A concrete example would be a specific vehicle control unit with specific firmware that performs a specific task, such as an engine control unit that controls fuel injection during normal operation. In this use case, each source log file has a specific, fixed structure that can be associated with a respective versioned dictionary.

[0024] For this purpose, a corresponding dictionary database can additionally be stored in the monitoring unit.

[0025] Versioned dictionaries can be stored in the dictionary database either as complete files or in an incremental update fashion. That is, an initial version of the dictionary can be stored in its entirety in the dictionary database, and versions built upon it contain only content that complements and / or modifies the initial dictionary. This keeps the memory requirements for storing multiple different versioned dictionaries small.

[0026] According to a further advantageous embodiment of the method, the dictionary database is stored on a central computing device, which allows for particularly efficient and reliable processing of the log files. The central computing device can be, for example, a cloud server, which is accessible by various computing systems. For example, it is conceivable that there are several monitoring units that cooperate to keep the dictionary database stored on the central computing device up to date. Similarly, it is conceivable that there are several deployment units that access the dictionary database on the central computing device in order to load currently required dictionaries as they are needed.

[0027] The deployment unit may be, for example, a PC connected to a central computing device via a network or the Internet. This communication connection is preferably a wired connection. The monitoring unit may be a mobile system connected directly or indirectly to the central computing device via a communication module. The communication module may be connected to the central computing device by mobile radio, for example.

[0028] In a further advantageous embodiment of the method, the log encoder also writes a version indication in the encoded log file when generating the encoded log file depending on the dictionary used, which allows a very simple and reliable transfer of the version indication from the expansion unit.

[0029] The version indication is specifically written to the encoded log file in a non-encoding manner, so that the log decoder or expansion unit can also read the version indication without prior decoding and thereby load the respective dictionary to be used.

[0030] According to a further advantageous embodiment of the method according to the invention, the deployment unit actively queries the monitoring unit or reads the version indication from an encoded log file.

[0031] Actively querying the version indication can improve cybersecurity. For example, a computing unit that wants to decode a given log file not only needs to receive the encoded log file and access the corresponding dictionary database, but also needs information about which dictionary to use to decode the encoded log file. This means that a suitable attacker must forge authentication to the monitoring unit, which presents an additional hurdle.

[0032] Alternatively, the version indication can be written into the encoded log file, thereby eliminating the need for further data transmission between the monitoring unit and the expansion unit, thereby enabling a more efficient use of the bandwidth in the transmission stream between the monitoring unit and the expansion unit. Here, the version indication can also typically be encrypted and then attached to each encoded log file, such that the expansion unit has a suitable key for decrypting the version indication.

[0033] Preferably, the log encoder continues to use the initialized coding table to generate different encoded log files from different source log files, or re-initializes the coding table when reading a new source log file.

[0034] As already explained, the version of the dictionary or coding table is selected depending on the structure of each source log file. If source log files with similar or identical content are processed several times in succession, the log encoder may continue to use the coding table already initialized for this type of source log file. However, if the content or type of the source log file changes, a new coding table needs to be initialized or a corresponding new versioned dictionary needs to be generated.

[0035] Depending on the embodiment and in particular the lifecycle phase of the monitoring unit or runtime component, it may be the case that only source log files of known content and type need to be processed. In this case, the monitoring unit no longer needs to retrieve dictionaries from a dictionary database, and instead continues to use coding tables that were initialized at an early stage, resulting in a highly efficient process. In other words, it is no longer necessary to dynamically generate respective coding tables at runtime of the monitoring unit, but rather to directly use previously initialized coding tables, so that dictionary initialization in the monitoring unit can be completely omitted. This allows the computing time of the processor of the monitoring unit to be used for other functions.

[0036] In an information technology system according to the invention, the system comprises a monitoring unit and a deployment unit, the monitoring unit and the deployment unit being designed to implement the method described above.

[0037] Preferably, the monitoring unit and the deployment unit are integrated into a common device, in particular a vehicle. The method according to the present invention allows for particularly efficient processing of log files in an automotive environment. The monitoring unit is therefore a control unit of a vehicle subsystem, a central vehicle computing unit, etc. Correspondingly, the runtime component is also formed by a hardware or software module of the vehicle. The deployment unit may also be a component of the vehicle. In general, the monitoring unit and the deployment unit may be provided in a common computing unit. However, the monitoring unit and the deployment unit can also be designed as two separate computing units, such as two separate vehicle control devices. The vehicle may be a passenger car, a truck, a bus, a typical road vehicle such as an excavator, a tractor, a rail vehicle, a ship, or an airplane.

[0038] According to a further advantageous embodiment of the information technology system, the monitoring unit and the deployment unit are integrated into or form separate devices, in particular the monitoring unit being integrated into the vehicle and the deployment unit being implemented outside the vehicle. For example, the deployment unit can be designed as a mobile terminal device, such as a laptop or tablet computer, connected to the vehicle for communication. For example, the mobile terminal device can be connected to the vehicle's computing unit via a cable, such as a USB cable or an Ethernet cable. Wireless data transmission, for example, Wi-Fi-based, Bluetooth-based, or NFC-based wireless data transmission, can also be performed. The deployment unit can also be geographically implemented outside the vehicle and, for example, formed by an office desktop computer. A corresponding control device of the vehicle can transmit information via a mobile radio telecommunications module and thus transmit information to the deployment unit, in particular via the Internet. The corresponding data transmission is realized between the monitoring unit and a central computing device or between the deployment unit and a central computing device.

[0039] Further advantageous embodiments of the method according to the invention for compressing and decompressing log files using a customizable dictionary will become apparent from the examples which are described in more detail below with reference to the drawings. [Brief explanation of the drawings]

[0040] [Figure 1] 1 is a schematic diagram of the components involved in the method according to the present invention for compressing and decompressing log files; DETAILED DESCRIPTION OF THE INVENTION

[0041] 1 shows a sequence diagram of a method according to the present invention for compressing and decompressing log files using a customizable dictionary 1. Here, a monitoring unit 2 monitors the behavior of runtime components 2.1 and / or 4. The runtime component 2.1 can be integrated into the monitoring unit 2 or can be implemented outside the monitoring unit 2, like the runtime component 4. The behavior of the runtime components 2.1, 4 is recorded using a source log file U-Log. The source log file U-Log can be generated by the runtime components 2.1, 4 or can also be generated by the monitoring unit 2.

[0042] The source log file U-Log is processed by the monitoring unit 2 using a log encoder 2.2. The monitoring unit 2 inspects the contents of the source log file U-Log, as indicated by the magnifying glass, and derives a coding table 5 therefrom. The coding table 5 describes rules for which contents of the source log file U-Log, e.g., a sequence of characters, should be compressed, i.e., converted into which character string. The coding table 5 can be flexibly generated depending on the contents of the source log file U-Log, thereby enabling the generation of a customizable dictionary 1.

[0043] When previously unknown content, i.e., a specific sequence of characters, is discovered for the first time in the source log file U-Log, new compression rules are created detailing how it should be compressed, i.e., translated into a shorter string. The coding table 5 thus generated is versioned by the monitoring unit 2 and transmitted as a version-controlled dictionary 1 to a dictionary database 6.1 for storage. In the embodiment shown in FIG. 1, the dictionary database 6.1 is implemented externally to the monitoring unit 2. Here, the dictionary database 6.1 is integrated into a central computing device 6, e.g., a cloud server. The coding table 5 thus generated and / or the version-controlled dictionary 1 thus generated can also be retained in the monitoring unit 2 for further use, as indicated by the dashed lines.

[0044] The source log file U-Log (whose content has already been inspected) is applied with either the coding table 5 just initialized according to its content, or with the coding table 5 initialized from the already generated versioned dictionary 1 (see dashed path). A log encoder 2.2 generates an encoded log file KOD-Log from it.

[0045] The encoded log file KOD-Log contains fewer character strings than the source log file U-Log and therefore has lower memory requirements. The encoded log file KOD-Log is then transmitted to the expansion unit 3. Due to the reduced memory size, a relatively fast transmission is possible here, even if the available data transmission rate is low. The encoded log file KOD-Log is then decoded in the expansion unit 3 for further processing.

[0046] For this purpose, the expansion unit 3 first needs information about which version-controlled dictionaries 1 are available for decoding. The expansion unit 3 can actively query the monitoring unit 2 for this information in conjunction with the corresponding version designation, or, if necessary, can also retrieve this information from the encoded log file KOD-Log, if this information has been added to the encoded log file KOD-Log. The expansion unit 3 then accesses the dictionary database 6.1 of the central computing device 6 and retrieves the appropriate version-controlled dictionary 1.

[0047] The appropriately versioned dictionary 1 and the encoded log file KOD-Log are read into the log decoder 3.1, which initializes the corresponding coding table 5 from the respective versioned dictionary 1 and subsequently applies the coding table 5 to the encoded log file KOD-Log to generate a decoded log file DEK-Log. The corresponding decoded log file DEK-Log can then be used to evaluate the system behavior of the runtime component 2.1 or 4. For this purpose, the decoded log file DEK-Log can be processed within the decompression unit 3 itself and / or transmitted externally, for example to a central computing device 6.

[0048] The method according to the present invention allows even very extensive source log files U-Log to be compressed in a space-efficient manner, while requiring minimal compression and decompression effort, resulting in particularly efficient processing of the log files. This means that the number of possible forms of the source log file U-Log, for which an appropriate versioned dictionary 1 is provided, is limited. With further development of the corresponding hardware and / or software components, a corresponding new version of the dictionary 1 is generated and stored in the dictionary database 6.1 for further use. In this case, the expansion unit 3 does not need to build the corresponding dictionary 1 anew; it only needs to receive a version instruction from the monitoring unit 2 to retrieve the appropriate dictionary 1 from the dictionary database 6.1 each time. Since each dictionary 1 does not need to be transmitted from the monitoring unit 2 to the expansion unit 3, the amount of data transmitted from the monitoring unit 2 to the expansion unit 3 is further reduced. In particular, if the monitoring unit 2 is integrated into a vehicle and the encoded log file KOD-Log is transmitted via a mobile data connection, the limited available transmission bandwidth can be utilized efficiently in this way. [Prior art documents] [Patent documents]

[0049] [Patent Document 1] German Patent No. 4340591 [Patent Document 2] US Patent Application Publication No. 2015 / 0066881 [Patent Document 3] U.S. Patent Application Publication No. 2021 / 0407220

Claims

1. A method for compressing and decompressing log files using a customizable dictionary (1), comprising: The following method steps are performed: - monitoring the runtime components (2.1, 4) by a monitoring unit (2) and generating a source log file (U-Log), said source log file (U-Log) recording the behavior of said runtime components (2.1, 4) described during said monitoring; - reading said source log file (U-Log) into a log encoder (2.2) by said monitoring unit (2); - initializing a coding table (5) from the versioned dictionary (1) or the source log file (U-Log) by the log encoder (2.2); - generating an encoded log file (KOD-Log) by applying said coding table (5) to said source log file (U-Log) by said log encoder (2.2); - transmitting said encoded log file (KOD-Log) from said monitoring unit (2) to a deployment unit (3); - said decompression unit (3) causes said encoded log file (KOD-Log) to be read by a log decoder (3.1) and receives from said monitoring unit (2) a version indication of said version-controlled dictionary (1) or of said coding table (5) used by said log encoder (2.2); - accessing a dictionary database (6.1) containing different version-controlled dictionaries (1) by the expansion unit (3), providing the dictionary (1) corresponding to the version indication to the log decoder (3.1), and initializing a coding table (5) from the version-controlled dictionaries (1) by the log decoder (3.1); - applying said coding table (5) to said encoded log file (KOD-Log) by said log decoder (3.1) to generate a decoded log file (DEK-Log); method.

2. The monitoring unit (2) reads the source log file (U-Log), generates a coding table (5) according to the information contained in the source log file (U-Log), derives a dictionary (1) from the coding table (5), provides a version indication in the dictionary (1), and transmits the dictionary (1) thus versioned to the dictionary database (6.1) for storage.

2. The method of claim 1.

3. The dictionary database (6.1) is stored in a central computing device (6).

3. The method according to claim 1 or 2.

4. The log encoder (2.2) writes the version indication in the encoded log file (KOD-Log) when generating the encoded log file (KOD-Log) depending on the dictionary (1) used. The method according to any one of claims 1 to 3, characterized in that

5. The deployment unit (3) actively queries the monitoring unit (2) for the version indication or reads it from the encoded log file (KOD-Log). The method according to any one of claims 1 to 4, characterized in that

6. The log encoder (2.2) continues to use the initialized coding table (5) to generate different encoded log files (KOD-Log) from different source log files (U-Log), or reinitializes the coding table (5) when reading a new source log file (U-Log). The method according to any one of claims 1 to 5, characterized in that

7. An information technology system comprising a monitoring unit (2) and a deployment unit (3), The monitoring unit (2) and the deployment unit (3) are designed to implement the method according to any one of claims 1 to 6.

1. An information technology system comprising:

8. The monitoring unit (2) and the deployment unit (3) are integrated into a common device, in particular a vehicle.

8. An information technology system according to claim 7.

9. The monitoring unit (2) and the deployment unit (3) are integrated into or form separate devices, in particular the monitoring unit (2) is integrated into the vehicle and the deployment unit (3) is mounted outside the vehicle.

8. An information technology system according to claim 7.

Images