Method for compressing and decompressing log file and information technology system

Abstract

The invention relates to a method for compressing and decompressing log files using a customizable dictionary (1), characterized by the following method steps: monitoring runtime components (2.1, 4) by means of a monitoring unit (2) and generating a source log file (U-Log); -reading the source log file (U-Log) into a log encoder (2.2); initializing an encoding table (5) from the versioned dictionary (1) or the source log file (U-Log); -generating an encoded log file (KOD-Log) by applying the encoding table (5) to the source log file (U-Log); -transmitting the encoded log file (KOD-Log) from the monitoring unit (2) to a development unit (3); reading the encoded log file (KOD-Log) into a log decoder (3.1) by means of the development unit (3) and retrieving a version description from the monitoring unit (2) in relation to a versioned dictionary (1) used by the log encoder (2.2) or in relation to an encoding table (5); accessing a dictionary database (6.1), providing the dictionary (1) corresponding to the version description to the log decoder (3.1), and initializing a coding table (5) from the versioned dictionary (1) by means of the log decoder (3.1); and-generating a decoded log file (DEK-Log) by applying the encoding table (5) to the encoded log file (KOD-Log).

Description

Technical Field

[0001] The present invention relates to a method for compressing and decompressing log files using a customizable dictionary, and an information technology system for performing the method. Background Art

[0002] Information technology systems, such as personal computers or embedded systems, consist of hardware and software components. When developing such systems, it may be necessary to record the behavior of the hardware and software components in operation and to analyze this in order to check the correct function or to find errors. This also includes so-called "debugging". The behavior of the corresponding hardware and / or software components is recorded in the form of so-called log messages or log files. The log files can be generated by the relevant hardware or software components themselves or by a unit that monitors the corresponding components. It is possible to analyze the corresponding log files at a later stage using different programs, for example manually with the aid of an editor.

[0003] Depending on the complexity of the hardware and / or software components to be monitored, and on the functionality provided by the components, the size of the corresponding log files may vary, which means that relatively less or more storage space is occupied on the storage medium. Therefore, in the case of limited data transfer rates, the transfer of log files between systems requires a lot of time and occupies a lot of storage space on the corresponding storage medium. Therefore, it is desirable to reduce the size of the corresponding log files.

[0004] In the IT field, various compression and decompression algorithms or methods are known. For example, zlib is a free library for compressing and decompressing data using the Deflate algorithm. In contrast to lossy compression, this is lossless compression. The Deflate algorithm belongs to the so-called dictionary method, in which the content of the log file, that is, usually a character sequence, such as a string (String), is converted into a shorter string taking into account the content of the corresponding dictionary. In such a dictionary, entries of typical and particularly frequently occurring character sequences in the log file are included, which define which strings these character sequences are replaced by. Character sequences such as "ABCDE" can be replaced by the string "a", for example. The content of the log file is thereby reduced, and the required storage requirements are correspondingly reduced.

[0005] Another data compression algorithm is "Brotli", which is based on a predefined dictionary of 120KiB size. However, its disadvantage is that, since the dictionary is fixed and predetermined, only the length of specific, predefined character sequences can be reduced.

[0006] In contrast to dictionaries with a fixed predetermined size and predetermined content, customizable dictionaries are also known. A data compression method based on this is known, for example, from DE 43 40 591 C2. This patent document describes data compression initialization methods that can be adjusted to different types of input data and different data structures in order to increase the data compression ratio and reduce the storage space required in a dictionary-based compression / decompression system. Here, corresponding entries in the dictionary are generated according to the input data to be compressed, i.e. the contents of the dictionary are supplemented so that input data that were originally unknown in the dictionary can also be compressed. The corresponding content can also be encoded multiple times here.

[0007] US2015 / 0066881 A1 also discloses a method for compressing log files, which allows for efficient collection of data from network nodes. A table-based compression algorithm is used to compress log files. The underlying table is generated with the aid of special training data. This makes it possible to build a large translation database.

[0008] In addition, US2021 / 0407220 A1 discloses a system, method, and device for managing data collection in a vehicle. The patent document describes generating a log file on a computing unit of the vehicle and processing the log file on a computing unit in the vehicle or outside the vehicle. Summary of the invention

[0009] The object of the present invention is to provide an improved method for compressing and decompressing log files, by which the workload of processing log files is reduced, while it is still possible to relatively greatly reduce the storage requirements or reduce the file size when the source files are different.

[0010] In a method for compressing and decompressing log files using a customizable dictionary, the following method steps are implemented according to the present invention:

[0011] - monitoring the runtime component by the monitoring unit and generating a source log file, wherein the source log file records the behavior of the runtime component written during the monitoring period;

[0012] -Read the source log file into the log encoder through the monitoring unit;

[0013] - Initialize the code table from the versioned dictionary or source log file through the log encoder;

[0014] - generating an encoded log file by applying the encoding table to the source log file with the log encoder;

[0015] -Transfer the encoded log files from the monitoring unit to the development unit;

[0016] - reading the encoded log file into the log decoder by the development unit and obtaining from the monitoring unit a version description of the versioned dictionary or the encoding table used by the log encoder;

[0017] - accessing a dictionary database by the development unit, the dictionary database including different versioned dictionaries, providing the dictionary corresponding to the version description to the log decoder, and initializing the coding table from the versioned dictionary by the log decoder; and

[0018] -Generate a decoded log file by applying the encoding table to the encoded log file with the log decoder.

[0019] By means of the method according to the invention, a particularly comprehensive data reduction can be achieved, since customizable dictionaries are used. In this way, entries can be supplemented in the corresponding dictionaries so that previously unknown entries of the source log files can also be compressed. The processing workload is particularly low here due to the versioning of the dictionaries. That is, different versions of dictionaries or coding tables based thereon are created according to the content of the corresponding source log files (which in turn depends on the hardware and software configuration to be monitored and / or monitored and the functionality provided by the operating environment), so that these dictionaries or coding tables are "tailor-made" according to the corresponding source log files. Subsequently, in order to compress the source log files or decompress the compressed source log files, it is only necessary to derive the coding tables from the respective relevant dictionaries and apply them to the corresponding log files for "translation". By means of the version description, it is possible to make clear, reliable and fast references between the corresponding log files and the dictionaries or coding tables to be applied to the log files (for compressing the source log files or decompressing the encoded log files). The versioned dictionaries and the coding tables based thereon have the same version here.

[0020] The runtime component consists of hardware components and / or software components. The runtime component can be implemented inside or outside the monitoring unit and monitored by the monitoring unit. The runtime component can be, for example, a program running on a personal computer or an embedded system in a vehicle (or its hardware / software components). The monitoring unit generates a source log file or receives the source log file / source log files from the runtime component. The log encoder is a component of the monitoring unit and reads in the source log file.

[0021] The log encoder or monitoring unit then selects which versioned dictionary or encoding table to set for encoding the source log file read in. This decision is made based on various boundary conditions, which will be discussed in detail later.

[0022] The dictionary is a binary file, also called Binary-File or simply "Binary". From the corresponding versioned dictionary, the log encoder initializes the corresponding encoding table and applies it to the source log file to generate an encoded log file. Therefore, the mapping relationship of which character sequence of the source log file should be replaced by which string is stored in the corresponding versioned dictionary. The string is also a character sequence, but shorter than the original character sequence. The two terms "character sequence" and "string" are selected here for ease of understanding. Depending on the content of the source log file, it contains completely different character sequences. This situation has been taken into account by different versions of dictionaries, so that the corresponding versioned dictionary only contains the mapping relationship between the character sequence that mainly appears in the corresponding source log file and the string. This can prevent such a dictionary (or the encoding table initialized / based on it) from being applied to the source log file: the dictionary contains the mapping relationship of the character sequence that only appears very rarely or hardly appears in the source log file. This can also reduce the storage requirements of the dictionary.

[0023] By translating the contents of the corresponding source log files with the aid of the coding table (derived from the dictionary), i.e. by actual compression, the storage space requirement of the coded log files is reduced compared to the source log files. This allows the monitoring unit to transmit the coded log files to the development unit faster than the (uncompressed) source log files given the same transmission bandwidth. Furthermore, the network security is increased thereby, since an attacker can only read the coded log files transmitted in this way if he has access to the dictionary used or the coding table initialized therefrom.

[0024] The dictionary database can be provided by the monitoring unit, the development unit and / or externally. Particularly when the dictionary database is external to the monitoring unit and the development unit and both the monitoring unit and the development unit have read access or read-write access rights to the dictionary database, a particularly efficient data transmission between the monitoring unit and the development unit is possible, since in this case only the encoded log file has to be transferred from the monitoring unit to the development unit and the development unit receives the corresponding versioned dictionary from the dictionary database via a separate communication channel. This is preferably a communication channel with a relatively high data transmission rate.

[0025] The log decoder has information about which versioned dictionary it must use to initialize the corresponding encoding table when decoding the encoded log file. The version information can be transmitted in different ways, which will be discussed later.

[0026] The decoded log file substantially or completely corresponds to the source log file. This makes it possible to process the source log file on the development unit in order to assess, for example, the correct functioning of the runtime component or to discover errors.

[0027] An advantageous improvement of the method provides that the monitoring unit reads the source log file, generates a coding table based on the information contained in the source log file, derives a dictionary from the coding table and sets a version description for the dictionary, and transmits the versioned dictionary to a dictionary database for storage. With this process, a dictionary adapted to the corresponding source log file can be generated.

[0028] The corresponding versioned dictionary is therefore suitable for encoding or decoding the source log file generated according to the design of the runtime component in hardware and / or software. In this way, the content of the source log file will be different, depending on the hardware and / or software implementation method of the runtime component, or depending on the functionality implemented by the runtime component. It is usually the case that the same application runs under different external boundary conditions on the same hardware and under the same software version. For this case, the same dictionary can be used here. When running new software and / or implementing other functionality, different versioned dictionaries can be used because the substantive content of the source log file changes accordingly. Similarly, when the hardware of the runtime component changes, different versioned dictionaries can be used.

[0029] The monitoring unit checks here which version, i.e. which version specification, is to be assigned to the corresponding dictionary during generation. In this way, the coding table or the dictionary based thereon can be compared in terms of content with the dictionaries already in the dictionary database. If identical contents are found, it is the same version. However, if no match is found, it is a new version. The assignment of the version specifications can be carried out here according to a predefined pattern, for example consecutive numbering.

[0030] The process of reading the source log file, generating the coding table and deriving the dictionary from the coding table is performed by the monitoring unit, in particular after the source log file is read into the log encoder, and before the versioned dictionary is possibly read and the coding table is initialized by the log encoder. In this way, a versioned dictionary that can be read in the log encoder and initialized there can be generated by the monitoring unit before the source log file is encoded. This makes it possible to encode a source log file with previously unknown content if a versioned dictionary does not exist for the source log file in the dictionary database. After the corresponding coding table and the corresponding versioned dictionary are generated from the source log file, the coding table can be used directly for encoding, and the generated versioned dictionary can be transferred to the dictionary database for storage.

[0031] On the other hand, if the source log file is processed in a familiar pattern (i.e., content), it is not necessarily required to first read out and check the content of the source log file each time and to generate and store a new coding table and a new versioned dictionary derived therefrom when appropriate. That is, instead, a versioned dictionary suitable for the corresponding source log file can be read out from a dictionary database. For example, it is also possible to determine which versioned dictionary to use from the configuration of the monitored runtime component and / or from the functionality provided by the runtime component. A specific example may be a specific vehicle control unit with specific firmware that performs a specific task, such as an engine control unit that controls fuel injection during normal operation. For this application scenario, the corresponding source log file will have a specific fixed structure, which can be assigned here to the corresponding versioned dictionary.

[0032] For this purpose, a corresponding dictionary database can also be stored in the monitoring unit in addition.

[0033] The versioned dictionaries can be stored in the dictionary database as complete files, or they can be stored as differential updates. In this way, the original version of the dictionary can be stored in its entirety in the dictionary database, while the versions based on it only contain additions and / or changes to the initial dictionary. This makes it possible to keep the storage requirements for storing multiple different versioned dictionaries low.

[0034] According to a further advantageous embodiment of the method, the dictionary database is stored on a central computing device. This enables particularly efficient and reliable processing of log files. The central computing device can be, for example, a cloud server. The cloud server can be accessed by different computing systems. For example, there can be a plurality of monitoring units which together keep the dictionary database stored on the central computing device up to date. Similarly, there can be a plurality of development units which access the dictionary database on the central computing device in order to load the respective currently required dictionaries.

[0035] The development unit may be a personal computer connected to the central computing device, for example, via a network or the Internet. This communication connection is preferably wired. The monitoring unit may be a mobile system, which is in communication connection with the central computing device directly or indirectly via the communication module. The communication module may be connected to the central computing device, for example, via mobile communication.

[0036] Another advantageous embodiment of the method provides that when generating the encoded log file, the log encoder writes the version description into the encoded log file according to the dictionary used. This makes it possible to communicate the version description from the development unit in a particularly simple and reliable manner.

[0037] The version description is written in an uncoded manner into the coded log file, so that a log decoder or a development unit can also read the version description without prior decoding and can accordingly load the corresponding dictionary to be used.

[0038] According to a further advantageous embodiment of the method according to the invention, the development unit actively requests the version information from the monitoring unit or reads the version information from an encoded log file.

[0039] By actively querying the version description, network security can be improved. In this way, the computing unit that wants to decode the log file not only needs to receive the encoded log file, but also needs to access the corresponding dictionary database and also needs information about which dictionary must be used in order to decode the encoded log file. In this way, the corresponding attacker must pretend to be authorized to the monitoring unit. This is an additional obstacle.

[0040] However, the version description can also be written to the coded log file, thereby making further data transmission between the monitoring unit and the development unit superfluous. This enables a more efficient bandwidth utilization in the transmission flow between the monitoring unit and the development unit. The version description can also be attached to the corresponding coded log file in an encrypted manner. In this case, a suitable key is stored on the development unit for decrypting the version description.

[0041] Preferably, the log encoder uses the initialized encoding table to generate different encoding log files from different source log files, or reinitializes the encoding table when reading a new source log file.

[0042] As mentioned above, the version of the dictionary or code table is selected according to the structure of the corresponding source log file. If it is necessary to process source log files with similar or identical content multiple times in succession, the log encoder can also continue to use the code table that has been initialized for this type of source log file. However, if the content or type of the source log file changes, a new code table must be initialized or a corresponding new versioned dictionary must be generated.

[0043] Depending on the implementation and in particular the lifecycle phase of the monitoring unit or runtime component, it may only be necessary to process source log files of known content and type. In this case, a completely particularly efficient process is possible, in which the monitoring unit no longer needs to read the dictionary from the dictionary database, but instead continues to use the code table initialized at the initial moment. In other words, it is no longer necessary to dynamically generate the corresponding code table during the runtime of the monitoring unit, but the step of initializing the dictionary on the monitoring unit can be completely omitted, because the pre-initialized code table is used directly. As a result, the processor's computing time can be made available for other functions of the monitoring unit.

[0044] In an information technology system having a monitoring unit and a development unit, the monitoring unit and the development unit are adapted according to the invention to carry out the above-described method.

[0045] Preferably, the monitoring unit and the development unit are integrated in a common device, in particular in a vehicle. By means of the method according to the invention, log files can be processed in an automotive environment in a particularly efficient manner. The monitoring unit is therefore a control device for a vehicle subsystem, a central vehicle computing unit or the like. Correspondingly, a runtime component is also formed by hardware or software modules of the vehicle. The development unit can also be a component of the vehicle here. In general, the monitoring unit and the development unit can also be provided on a common computing unit. However, the monitoring unit and the development unit can also be formed by two independent computing units, for example two independent vehicle control devices. As vehicles, all common road vehicles, such as cars, trucks, buses, excavators, tractors, etc., as well as rail vehicles, water vehicles or aircraft are taken into consideration.

[0046] According to another advantageous design of the information technology system, the monitoring unit and the development unit are integrated in a separate device or constitute separate devices, in particular, the monitoring unit is integrated in the vehicle and the development unit is implemented outside the vehicle. In this way, the development unit can be implemented, for example, by a mobile terminal, such as a laptop or a tablet, which is in communication connection with the vehicle. For example, the mobile terminal can be connected to the computing unit of the vehicle in a wired manner, such as via a USB cable or an Ethernet cable. Wireless data transmission can also be carried out, such as WiFi-based, Bluetooth-based or NFC-based wireless data transmission. The development unit can also be implemented geographically outside the vehicle, for example, it can be constructed by a desktop computer located in an office. The corresponding control device of the vehicle can send information about the telecommunication module via mobile communication, in particular via the Internet to the development unit. Corresponding data transmission can be carried out between the monitoring unit and the central computing device or between the development unit and the central computing device. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Further advantageous embodiments of the method for compressing and decompressing log files using a customizable dictionary according to the invention are also derived from the embodiments, which will be described in more detail below with reference to the drawings.

[0048] Figure 1 A schematic diagram showing components involved in a method for compressing and decompressing log files according to the present invention. DETAILED DESCRIPTION

[0049] Figure 1The flow of a method according to the invention for compressing and decompressing log files using a customizable dictionary 1 is shown. In this case, a monitoring unit 2 monitors the behavior of a runtime component 2.1 and / or 4. The runtime component 2.1 can be integrated in the monitoring unit 2 or can be implemented externally thereto like the runtime component 4. The behavior of the runtime components 2.1, 4 is recorded with the aid of a source log file U-Log. The source log file U-Log can be generated by the runtime components 2.1, 4 or the monitoring unit 2.

[0050] The source log file U-Log is processed by the monitoring unit 2 with the aid of a log encoder 2.2. The monitoring unit 2 checks (as indicated by a magnifying glass in the figure) the content of the source log file U-Log and derives a coding table 5 therefrom. The coding table 5 describes here the rules according to which the content (e.g., a sequence of characters) of the source log file U-Log should be compressed, i.e., converted into which strings. It is feasible to flexibly generate the coding table 5 according to the content of the source log file U-Log, so that a customizable dictionary 1 can be generated.

[0051] Therefore, if previously unknown content, i.e. a specific character sequence, is encountered for the first time in the source log file U-Log, a new compression rule is created specifically for how to compress it, i.e. how to translate it into a shorter string. Next, the monitoring unit 2 sets a version description / version information / version identification for the coding table 5 thus generated, and transmits the coding table in the form of a versioned dictionary 1 to the dictionary database 6.1 for storage. Figure 1 In the example shown, the dictionary database 6.1 is implemented outside the monitoring unit 2. Here, the dictionary database 6.1 is integrated in a central computing device 6, for example a cloud server. The coding table 5 generated in this way and / or the versioned dictionary 1 generated in this way can also be stored in the monitoring unit 2 for further use, as indicated by the dashed line.

[0052] For the source log file U-Log (whose content has been reviewed), either a newly initialized code table 5 according to the content or a code table 5 initialized from an already generated versioned dictionary 1 is applied (see dotted line path). The log encoder 2.2 thus generates the encoded log file KOD-Log therefrom.

[0053] The storage requirement of the encoded log file KOD-Log is lower than that of the source log file U-Log, since it contains a smaller number of character strings. The encoded log file KOD-Log is then transmitted to the development unit 3. Due to the reduced storage volume, relatively fast transmission is possible here even at lower data transmission rates. The encoded log file KOD-Log now needs to be decoded on the development unit 3 in order to be further processed.

[0054] To this end, the development unit 3 first requires information about which version of the dictionary 1 is available for decoding. The development unit 3 can actively request information about the corresponding version description from the monitoring unit 2 or read this information from the encoded log file KOD-Log if appropriate, if this information has been attached to this file. The development unit 3 then accesses the dictionary database 6.1 of the central computing device 6 and reads out the corresponding, appropriate version of the dictionary 1.

[0055] The appropriately versioned dictionary 1 and the encoded log file KOD-Log are read into the log decoder 3.1. The log decoder 3.1 initializes the corresponding coding table 5 from the corresponding versioned dictionary 1 and then applies the coding table 5 to the encoded log file KOD-Log in order to generate a decoded log file DEK-Log therefrom. The corresponding decoded log file DEK-Log can then be evaluated in order to evaluate the system behavior of the runtime component 2.1 or 4. For this purpose, the decoded log file DEK-Log can be processed on the development unit 3 itself and / or transmitted to the outside, for example to the central computing device 6.

[0056] The method according to the invention makes it possible to process log files particularly efficiently, since even particularly large source log files U-Log can be compressed in a storage-saving manner, with a low amount of work required for compression and decompression. In this way, the number of possible implementations of the source log file U-Log is limited, and a matching versioned dictionary 1 is retained for each implementation. If the corresponding hardware and / or software components are further developed, a corresponding new version of the dictionary 1 is generated and retained in the dictionary database 6.1 for subsequent use. The development unit 3 does not have to reconstruct the corresponding dictionary 1, but only needs to obtain the version information from the monitoring unit 2 in order to read the respective matching dictionary 1 from the dictionary database 6.1. The corresponding dictionary 1 does not have to be transmitted from the monitoring unit 2 to the development unit 3, thereby reducing the amount of data that is additionally transmitted from the monitoring unit 2 to the development unit 3. In particular, when the monitoring unit 2 is integrated in the vehicle and sends the encoded log file KOD-Log via a mobile data connection, the limited transmission bandwidth can be used effectively.

Claims

1. A method for compressing and decompressing log files using a customizable dictionary (1), The following method steps are implemented: - monitoring the runtime component (2.1, 4) by means of the monitoring unit (2) and generating a source log file (U-Log), wherein the source log file (U-Log) records the behavior of the runtime component (2.1, 4) written down during the monitoring; - reading the source log file (U-Log) into a log encoder (2.2) via the monitoring unit (2); - initializing a coding table (5) from a versioned dictionary (1) or the source log file (U-Log) by means of the log encoder (2.2); - applying the coding table (5) to the source log file (U-Log) by the log encoder (2.2) to generate a coded log file (KOD-Log); - transmitting the coded log file (KOD-Log) from the monitoring unit (2) to the development unit (3); - reading the coded log file (KOD-Log) into a log decoder (3.1) via the development unit (3) and obtaining from the monitoring unit (2) information about the versioned dictionary (1) or about the coding table (5) used by the log encoder (2.2); - accessing a dictionary database (6.1) by means of the development unit (3), the dictionary database comprising dictionaries (1) of different versions, providing the dictionary (1) corresponding to the version description to the log decoder (3.1), and initializing a coding table (5) from the versioned dictionary (1) by means of the log decoder (3.1); and - Applying the coding table (5) to the coded log file (KOD-Log) by the log decoder (3.1) to generate a decoded log file (DEK-Log).

2. The method according to claim 1, It is characterized in that The monitoring unit (2) reads the source log file (U-Log), generates a coding table (5) based on the information contained in the source log file (U-Log), derives a dictionary (1) from the coding table (5) and sets a version description for the dictionary, and transmits the versioned dictionary (1) to the dictionary database (6.1) for storage.

3. The method according to claim 1 or 2, It is characterized in that The dictionary database (6.1) is stored on a central computing device (6).

4. The method according to claim 1, It is characterized in that When generating the encoded log file (KOD-Log), the log encoder (2.2) writes the version description into the encoded log file (KOD-Log) according to the dictionary (1) used.

5. The method according to one of claims 1 to 4, It is characterized in that The development unit (3) actively requests the version description from the monitoring unit (2), or reads the version description from the encoded log file (KOD-Log).

6. The method according to one of claims 1 to 5, It is characterized in that The log encoder (2.2) continues to use the initialized encoding table (5) to generate different encoded log files (KOD-Log) from different source log files (U-Log), or reinitializes the encoding table (5) when reading a new source log file (U-Log).

7. An information technology system having a monitoring unit (2) and an exploitation unit (3), It is characterized in that The monitoring unit (2) and the development unit (3) are adapted to perform the method according to one of claims 1 to 6.

8. The information technology system according to claim 7, It is characterized in that The monitoring unit (2) and the development unit (3) are integrated in a common device, in particular a vehicle.

9. The information technology system according to claim 7, It is characterized in that The monitoring unit (2) and the development unit (3) are integrated in a separate device or form a separate device. In particular, the monitoring unit (2) is integrated in a vehicle and the development unit (3) is implemented outside the vehicle.

Images