Log compression method and device, computer readable storage medium and electronic equipment

Abstract

The invention provides a log compression method and device, a computer readable storage medium and electronic equipment, and the method comprises the steps: obtaining to-be-compressed log data, classifying the to-be-compressed log data, and obtaining a plurality of target categories, the to-be-compressed log data being original log data which is not compressed; determining reference to-be-compressed log data of each target category, and calculating a difference value between each to-be-compressed log data in each target category and the reference to-be-compressed log data to obtain difference value data, the reference to-be-compressed log data being used for encoding the to-be-compressed log data; and encoding the reference to-be-compressed log data and the difference data to generate simplified log data corresponding to each target category, and compressing the multiple pieces of simplified log data. Through the method and the device, the problem of low compression efficiency of a log compression method in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the technical field of log data compression, and in particular to a log compression method, device, computer-readable storage medium and electronic device. Background Art

[0002] In the current era of high digitalization and networking, the amount of log data generated by information systems and network devices is exploding. Log data is an important record of system operations and events, and is essential for monitoring system health, fault diagnosis, and performance analysis. However, the massive generation and storage requirements of log data have brought significant cost and efficiency challenges to information systems. Traditional log compression methods mainly rely on general compression algorithms such as Gzip and LZ4. Although these algorithms can provide a certain data compression ratio, their efficiency and compression effect are not ideal when processing log data with highly redundant and repeated characteristics.

[0003] Gzip, as a compression tool based on Lempel-Ziv encoding, can effectively compress text data, but it does not make full use of repeated patterns and contextual information in log data, resulting in a low compression ratio. Similarly, LZ4, known for its fast compression and decompression speed, has a significant drop in compression efficiency when faced with a large amount of repeated data. Log data usually contains a large number of repeated timestamps, log levels (such as INFO, ERROR, DEBUG), and similar event descriptions, which makes traditional general-purpose compression algorithms incapable of processing such data.

[0004] Therefore, how to utilize the contextual information of log data (such as timestamp, log level, etc.) for more efficient compression has become an urgent problem to be solved. Summary of the invention

[0005] The main purpose of the present application is to provide a log compression method, device, computer-readable storage medium and electronic device to at least solve the problem of low compression efficiency of the log compression method in the prior art.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a log compression method is provided, including: obtaining log data to be compressed, classifying the log data to be compressed, and obtaining multiple target categories, wherein the log data to be compressed is the original log data that has not been compressed; determining the baseline log data to be compressed for each of the target categories, and calculating the difference between each of the log data to be compressed in each of the target categories and the baseline log data to be compressed to obtain difference data, wherein the baseline log data to be compressed is used for encoding the log data to be compressed; encoding the baseline log data to be compressed and the difference data to generate simplified log data corresponding to each of the target categories, and compressing multiple simplified log data.

[0007] Optionally, the log data to be compressed are classified to obtain multiple target categories, including: classifying the log data to be compressed whose timestamp difference is less than a preset threshold into a time category; classifying the log data to be compressed with the same log level into a level category; and classifying the log data to be compressed with the same log content into a content category, to obtain multiple target categories.

[0008] Optionally, determining the baseline log data to be compressed for each target category includes: when the target category is the time category, determining the log data to be compressed with the smallest timestamp as the baseline log data to be compressed corresponding to the time category; when the target category is the level category, using numbers to represent each level category to obtain the baseline log data to be compressed corresponding to the level category; when the target category is the content category, extracting the same content from multiple log data to be compressed, and determining the same content as the baseline log data to be compressed corresponding to the content category.

[0009] Optionally, encoding the baseline log data to be compressed and the difference data includes: when the target category is the time category, encoding the baseline log data to be compressed and the difference data using a differential encoding method, wherein the differential encoding method is a method of encoding the difference data; when the target category is the level category, encoding the baseline log data to be compressed and the difference data using a dictionary encoding method, wherein the dictionary encoding method is a method of encoding using numbers; when the target category is the content category, encoding the baseline log data to be compressed and the difference data using a pattern matching method, wherein the pattern matching method is a method of encoding using different contents.

[0010] Optionally, obtaining the log data to be compressed includes: obtaining a timestamp, a log level, and log content of the log data to be compressed.

[0011] Optionally, after compressing the plurality of simplified log data, the method further comprises: storing the compressed simplified log data in a predetermined storage format, wherein the predetermined storage format includes the simplified log data corresponding to each of the target categories.

[0012] Optionally, after compressing the plurality of simplified log data, the method further comprises: upon receiving a decompression instruction, calculating the sum of the reference log data to be compressed and the difference data to obtain decompressed data.

[0013] According to another aspect of the present application, a log compression device is provided, including: a classification unit, used to obtain log data to be compressed, classify the log data to be compressed, and obtain multiple target categories, wherein the log data to be compressed is the original log data that has not been compressed; a calculation unit, used to determine the baseline log data to be compressed for each of the target categories, and calculate the difference between each of the log data to be compressed in each of the target categories and the baseline log data to be compressed to obtain difference data, wherein the baseline log data to be compressed is used for encoding the log data to be compressed; a compression unit, used to encode the baseline log data to be compressed and the difference data, generate simplified log data corresponding to each of the target categories, and compress multiple simplified log data.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the log compression methods.

[0015] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the log compression methods.

[0016] Applying the technical solution of the present application, log data to be compressed is obtained, and the log data to be compressed is classified to obtain multiple target categories, wherein the log data to be compressed is the original log data that has not been compressed; the benchmark log data to be compressed of each target category is determined, and the difference between each log data to be compressed and the benchmark log data to be compressed in each target category is calculated to obtain the difference data, wherein the benchmark log data to be compressed is the log data to be compressed as a benchmark; the benchmark log data to be compressed and the difference data are encoded to generate simplified log data corresponding to each target category, and multiple simplified log data are compressed. Compared with the low compression efficiency of the log compression method in the prior art, the present application reduces the storage space of the log data and improves the compression efficiency by classifying the data to be compressed and encoding the benchmark data and difference data for each category. Therefore, it can solve the problem of low compression efficiency of the log compression method in the prior art and improve the compression efficiency of the log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a mobile terminal for executing a log compression method provided in an embodiment of the present application is shown;

[0019] Figure 2 A flow chart of a log compression method provided in an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of a flow chart of a specific log compression method provided in an embodiment of the present application is shown;

[0021] Figure 4 A structural block diagram of a log compression device provided in an embodiment of the present application is shown.

[0022] The above drawings include the following reference numerals:

[0023] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] As introduced in the background technology, the compression efficiency of the log compression method in the prior art is low. In order to solve the problem of low compression efficiency of the log compression method, the embodiments of the present application provide a log compression method, device, computer-readable storage medium and electronic device.

[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0029] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a log compression method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0030] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the log compression method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned network specific examples may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] In this embodiment, a log compression method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] Figure 2 : is a flow chart of a log compression method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0033] Step S201, obtaining log data to be compressed, classifying the log data to be compressed, and obtaining multiple target categories, wherein the log data to be compressed is original log data that has not been compressed;

[0034] Specifically, in modern information systems, log data is continuously generated, recording various aspects of system operation, including but not limited to user activities, system events, error reports, performance indicators, etc. These log data are usually stored in text or binary format, which takes up a lot of storage space when not compressed. The first step to obtain the log data to be compressed is to ensure the integrity of the data, that is, to collect all relevant raw log data to avoid losing important information during the data collection stage. This usually involves grabbing log files from multiple source data points such as servers, network devices, and applications, and then storing these files in a unified manner to prepare for subsequent processing steps. Classified log data can reveal the inherent structure and regularity of the data, providing a more targeted processing basis for compression algorithms. Based on the various characteristics of log data, it can be classified into different target categories. For example, according to the timestamp of the log entry, the data can be grouped in chronological order or time interval, which helps to identify the time pattern and trend of the data and provides a basis for differential encoding. According to the log level, such as INFO, ERROR, DEBUG, etc., the log data can be divided into different severity categories, which is of great significance for optimizing log storage and fast retrieval. In addition, based on the similarity of log contents, repeated or similar event descriptions can be identified, which helps to further reduce data redundancy.

[0035] Step S202, determining the reference log data to be compressed of each target category, and calculating the difference between each log data to be compressed in each target category and the reference log data to be compressed to obtain difference data, wherein the reference log data to be compressed is used for encoding the log data to be compressed;

[0036] Specifically, one or more entries are selected from the classified log data as the benchmark log data to be compressed, and these benchmark data will be used for subsequent difference calculations. After the benchmark log data to be compressed is determined, a difference calculation is performed between each log data to be compressed in each target category and the benchmark data. The purpose of the difference calculation is to identify and quantify the differences between the data, rather than storing the complete data content, thereby reducing the storage space occupied. The calculation of difference data can be performed on different parts of the data, including timestamps, log levels, and log content.

[0037] Step S203: Encode the reference log data to be compressed and the difference data to generate simplified log data corresponding to each target category, and compress a plurality of simplified log data.

[0038] Specifically, the purpose of encoding the benchmark data is to convert it into a more compact form that is easier to store and transmit. Time difference encoding: record the difference between each log and the benchmark timestamp instead of the full date and time. For example, if the timestamp of the benchmark log is 1609459200 seconds, then the subsequent data with a timestamp of 1609459205 seconds only needs to be encoded as "5 seconds". Log level difference encoding: for entries with the same level as the benchmark log, the encoding can be omitted; for entries of different levels, the corresponding encoding value can be found in the log level dictionary through the index to store the encoding difference. Content difference encoding: identify the difference or supplementary information between each log and the benchmark content, and only store the changed part. For example, if the benchmark log records that user A visited a certain web page, and the subsequent entry records that user A visited another web page, it only needs to be encoded as "user ID = A, access URL = changed URL". Generate simplified log data: combine the encoded benchmark data and difference data to generate simplified log data corresponding to each target category. Simplified log data contains the key features of the original data, but is stored in a more compact form, reducing unnecessary duplication of information. Through the above steps, the storage and transmission costs of log data can be effectively reduced while maintaining the integrity and retrievability of the data, providing strong support for efficient data management and analysis.

[0039] Through this embodiment, the log data to be compressed is obtained, and the log data to be compressed is classified to obtain a plurality of target categories, wherein the log data to be compressed is the original log data that has not been compressed; the benchmark log data to be compressed of each target category is determined, and the difference between each log data to be compressed and the benchmark log data to be compressed in each target category is calculated to obtain the difference data, wherein the benchmark log data to be compressed is the log data to be compressed as a benchmark; the benchmark log data to be compressed and the difference data are encoded to generate the simplified log data corresponding to each target category, and the plurality of simplified log data are compressed. Compared with the low compression efficiency of the log compression method in the prior art, the present application reduces the storage space of the log data and improves the compression efficiency by classifying the data to be compressed and encoding the benchmark data and the difference data for each category. Therefore, the problem of low compression efficiency of the log compression method in the prior art can be solved, and the compression efficiency of the log data can be improved.

[0040] In the specific implementation process, the above step S201 classifies the log data to be compressed to obtain multiple target categories, which can be achieved by the following steps: Step S2011: classify the log data to be compressed whose timestamp difference is less than a preset threshold into time categories; Step S2012: classify the log data to be compressed with the same log level into level categories; Step S2013: classify the log data to be compressed with the same log content into content categories to obtain multiple target categories. The method classifies logs through the above steps to obtain multiple target categories, effectively reducing the storage and transmission costs of log data, while maintaining the integrity and retrievability of the data, providing strong support for efficient data management and analysis.

[0041] Specifically, the time category division is based on the timestamp information of the log data. The timestamp is one of the key attributes in the log data, which records the specific time when each log is generated. By classifying the log data to be compressed whose timestamp difference is less than the preset threshold into the same time category, efficient compression of time series data can be achieved. For example, if the preset threshold is set to 10 seconds, all log entries with a timestamp interval of less than 10 seconds will be classified into the same time category. The log level is a standard for measuring the importance and urgency of log information. Common log levels include INFO, ERROR, DEBUG, WARNING, etc. Classifying log entries with the same log level into level categories can make the compression method more intelligent and efficient. Extract the log level information from each log entry as the basis for classification. Level category division: Log entries with the same log level are classified into the same category. For example, all ERROR level logs will be classified into the ERROR category. This helps to eliminate the repeated storage of log level information, especially when the log levels are consistent in the same category, the storage structure can be further optimized. Content parsing: Parse the text description of each log to extract key information, such as user ID, operation type, error information, etc. Content pattern recognition: Identify repeated or similar log content patterns. For example, multiple logs record the event of user login failure, but the specific reasons for failure may be different. Content classification: Log entries with the same pattern or similar content are classified into the same content category. For example, all logs that record user login failures will be classified into the "Login Failure" category. Through content classification, more efficient pattern matching and compression algorithms can be applied to reduce redundant storage. After completing the classification of time, level, and content, the system can further integrate these categories to generate a more detailed and structured classification of log data. For example, all log entries belonging to the same time category, the same level category, and the same content category can be combined into a specific target category. Each target category contains highly related and repeated features, which provides optimization opportunities for context-aware compression algorithms. Ultimately, the log data in each target category will be processed and compressed independently, thereby significantly reducing storage and transmission costs, while also ensuring data integrity and retrievability, which facilitates subsequent data analysis and troubleshooting. In this way, the system can manage log data more intelligently and improve overall operational efficiency and resource utilization.

[0042] In some optional implementations, the above step S202 determines the benchmark log data to be compressed for each target category, which can be achieved by the following steps: Step S2021: When the target category is the time category, the log data to be compressed with the smallest timestamp is determined as the benchmark log data to be compressed corresponding to the time category; Step S2022: When the target category is the level category, each level category is represented by a number to obtain the benchmark log data to be compressed corresponding to the level category; Step S2023: When the target category is the content category, the same content in multiple log data to be compressed is extracted, and the same content is determined as the benchmark log data to be compressed corresponding to the content category. This method can convert the log data under each target category into a more compact and structured form, laying the foundation for subsequent context-aware compression and fast retrieval.

[0043] In the specific implementation process, when dividing the log data into time categories, the log entry with the smallest timestamp (i.e., the earliest generated) is usually selected as the benchmark log data to be compressed. The log data with the smallest timestamp is usually the starting point of the time series, and all subsequent log data are calculated based on this time point to ensure the continuity and integrity of the time information. Therefore, by determining the log data with the smallest timestamp as the benchmark, the system can calculate the difference between all subsequent log entries and the benchmark, and then only store these difference information, which greatly reduces the storage space requirements. In the log level classification, the determination of the benchmark can be achieved by creating a level encoding dictionary, in which the name of each level is mapped to a digital code. For example, the dictionary can be defined as {'INFO':1,'ERROR':2,'DEBUG':3,'WARNING':4}. The process of determining the benchmark is actually to build this dictionary and map log entries of different levels to corresponding digital codes. Determining the benchmark data of the content category involves identifying duplicate content in multiple log data to be compressed. The data of each log is parsed to extract key information, such as user ID, access URL, error code, etc. Using pattern matching algorithms, log entries with the same or very similar content are identified. These entries may record different instances of the same event. For example, multiple logs record user login failures, but the specific reasons for failure may be different. Select one entry from the log entries with the same content as the baseline log data to be compressed. The selection criteria may include the typicality of the data, the frequency of occurrence, or the completeness of the information. For example, a log entry with the most complete error information can be selected as the baseline, because this helps to quickly locate the problem after decompression. For each log entry with the same content as the baseline, the difference information between it and the baseline is calculated and stored, such as the difference in user ID, the change in access time, etc. Through the above steps, the log data under each target category can be converted into a more compact and structured form, laying the foundation for subsequent context-aware compression and fast retrieval. In this process, the selection of baseline data is crucial, which affects the compression efficiency and the accuracy of the decompressed data. By utilizing the context information of the log data, the system can manage massive log data more intelligently and efficiently, while ensuring the integrity of the data and the convenience of retrieval.

[0044] In some optional embodiments, the above-mentioned step S203 encodes the benchmark log data to be compressed and the difference data, which can be achieved by the following steps: Step S2031: When the target category is the time category, the benchmark log data to be compressed and the difference data are encoded by a differential encoding method, wherein the differential encoding method is a method of encoding the difference data; Step S2032: When the target category is the level category, the benchmark log data to be compressed and the difference data are encoded by a dictionary encoding method, wherein the dictionary encoding method is a method of encoding by numbers; Step S2033: When the target category is the content category, the benchmark log data to be compressed and the difference data are encoded by a pattern matching method, wherein the pattern matching method is a method of encoding by different contents.

[0045] Specifically, when the target category is the time category, the system uses differential encoding to compress the timestamp information. Differential encoding is an effective data compression technology that reduces storage requirements by recording the change value of a data point relative to the previous data point rather than its complete value. In log data processing, the specific steps of differential encoding are as described above. The log data with the smallest timestamp is selected as the benchmark for subsequent time difference calculations. For each subsequent log data, the difference between its timestamp and the benchmark timestamp is calculated. For example, if the benchmark timestamp is 1609459200 seconds, then the log entry with a timestamp of 1609459205 seconds only needs to store the difference information of "5 seconds". Differential encoding: Encode the calculated timestamp differences and store these differences instead of the complete timestamp value. Differential encoding can significantly reduce the storage requirements of timestamp information, especially in data-intensive time series. When processing log levels, dictionary encoding is an effective method to convert the text representation of log levels into digital encodings, thereby reducing storage space. The steps of dictionary encoding are as follows: Log level dictionary establishment: Create a log level dictionary to map the names of different levels to digital codes. For example, {'INFO':1,'ERROR':2,'WARNING':3,'DEBUG':4}. Determine the baseline log data to be compressed corresponding to each level category, that is, the first log level that appears, or select the most frequently occurring level as the baseline based on the frequency of use, and perform digital encoding. Level difference encoding: For log entries that are different from the baseline log level, search through the dictionary and replace them with the corresponding digital code. If the log level is the same as the baseline, the encoding can be omitted, and only a flag bit is stored to indicate that the level has not changed. Store the encoded log level information for subsequent compression processing and data retrieval. Select an entry from the log entries with the same content as the baseline log data to be compressed. The baseline content should contain the most common or important information in the data of this category. For each log entry with the same content as the baseline, analyze the differences between it and the baseline, such as user ID, access time, etc. Based on the content difference, a pattern matching algorithm is used to encode the difference information. For example, repeated strings or fixed-format information can be replaced by storing a pattern ID and the changed part instead of the complete log content. The pattern matching encoded data is stored, and these data only contain the difference from the baseline content, which significantly reduces the storage space occupied. By adopting the above encoding method, the system can not only effectively compress log data and reduce storage and transmission costs, but also maintain data integrity and retrieval convenience. These encoding strategies make full use of the internal structure and context information of log data, and are an important part of implementing context-aware log compression methods.

[0046] In some optional implementations, the above step S201 of obtaining the log data to be compressed can be implemented by the following steps: obtaining the timestamp, log level and log content of the log data to be compressed.

[0047] In the specific implementation process, the timestamp is a key attribute in the log entry, which records the exact time when each log is generated. The extraction and processing of timestamps are the basis for time series analysis and context-aware compression. Log parsing: Use log parsing tools or custom parsing logic to accurately extract timestamp information from the text content of the log entry. This may involve the use of regular expressions to match timestamps in different formats. The log level reflects the severity and type of the log information. Common levels include INFO, ERROR, WARNING, DEBUG, etc. The log content contains a detailed description of the system operation status, user behavior, error information, etc. Through the above steps, the key attributes of the log data to be compressed can be fully and accurately obtained, providing the necessary information for the subsequent context-aware compression scheme. This information is not only used for compression, but also for building indexes, so that the compressed log data can be quickly retrieved and decompressed when needed, maintaining its integrity and availability.

[0048] In other optional implementations, after compressing the plurality of simplified log data, the method further includes: storing the compressed simplified log data in a predetermined storage format, wherein the predetermined storage format includes the simplified log data corresponding to each of the target categories.

[0049] Specifically, for each target category (time, level, content), the compressed simplified log data is divided into several storage blocks. Each storage block contains one or more simplified log data entries, as well as the context information of the block, such as timestamp range, log level, content mode, etc. The beginning of each storage block should contain block header information for quickly locating and identifying the block content. The block header information usually includes the start and end timestamps of the block, the log level encoding dictionary, the content mode ID, etc. In order to support fast retrieval, it is crucial to design a multi-level index structure. The index should include time index, level index, and content index to find specific log records on demand. By indexing the storage blocks by time range, users can quickly locate the log data within a certain time window. By classifying and indexing the storage blocks by log level, users can filter the log information of a specific level. Indexing is based on content mode ID or keywords to support content-related log queries. In addition to indexing, metadata needs to be managed, including compression algorithm, compression ratio, block size, number of blocks, etc. This information is crucial for data management and performance optimization.

[0050] In some optional implementations, after compressing the plurality of simplified log data, the method further comprises: upon receiving a decompression instruction, calculating the sum of the reference log data to be compressed and the difference data to obtain decompressed data.

[0051] In the specific implementation process, according to the query conditions in the decompression instruction, such as time range, log level, etc., the pre-built index structure is used to find the storage block containing the relevant simplified log data. After locating the storage block, the block header information and compressed data are read. The block header information contains the context information used during compression, such as timestamp range, log level encoding dictionary, content mode ID, etc., which are necessary references for decompression. The benchmark log data to be compressed is obtained from the context information of the storage block. For example, in the time category, the benchmark data is the log entry with the smallest timestamp; in the level category, the benchmark data is the encoding of the corresponding log level of the category; in the content category, the benchmark data is the log entry with the same content mode. The difference data stored in the block is read. These data are simplified during compression, and only the change information relative to the benchmark data is retained, such as the difference in timestamps, the encoding of log level changes, and the difference description of content modes. According to the data type and encoding method, the difference data is calculated with the benchmark data. For the time category, add the timestamp difference to the base timestamp to restore the original timestamp; for the level category, convert the digital code back to the original log level according to the encoding dictionary; for the content category, restore the complete log content according to the pattern difference description and the base content. Assume that the base timestamp is 1609459200 seconds and the difference data is [5,10,20]. The process of calculating the sum is as follows: For the first difference 5, calculate 1609459200+5=1609459205 seconds. For the second difference 10, calculate 1609459205 (previous recovery timestamp)+10=1609459215 seconds. For the third difference 20, calculate 1609459215+20=1609459235 seconds.

[0052] Log level decompression example: Assume that the base log level is INFO, the code is 1, and the difference data is [0,1,2], which respectively indicates that the level remains unchanged, changes to ERROR, and changes to WARNING. Convert back to the original level according to the dictionary code: For the first difference value 0, the level remains unchanged and is INFO. For the second difference value 1, according to the code dictionary, INFO(1)+1=ERROR(2). For the third difference value 2, INFO(1)+2=WARNING(3).

[0053] Content category decompression example: Assume that the baseline content is "User login successful", and the difference data is [{"user_id":1001,"login_time":"2023-01-01 00:00:05"}, {"user_id":1002,"login_time":"2023-01-0100:00:10"}]. Restore the complete log content: For the first difference data, the complete log is "User ID 1001 successfully logged in at 2023-01-01 00:00:05". For the second difference data, the complete log is "User ID 1002 successfully logged in at 2023-01-01 00:00:10".

[0054] Through the above steps, the system can quickly and accurately restore the original log data without storing the complete original information, thus greatly saving storage space. The decompression process combines the context information during compression to ensure the integrity and consistency of data recovery while maintaining high efficiency, meeting the needs of log management and analysis in a big data environment.

[0055] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the log compression method of the present application will be described in detail below in conjunction with specific embodiments.

[0056] This embodiment relates to a specific log compression method, such as Figure 3 As shown, the following steps are included:

[0057] Step S1: The user collects log data to the system;

[0058] Step S2: the system pre-processes log data;

[0059] Step S3: timestamp standardization;

[0060] Step S4: Log level extraction;

[0061] Step S5: content analysis;

[0062] Step S6: context-aware compression;

[0063] Step S7: timestamp differential decoding;

[0064] Step S8: Log level dictionary encoding;

[0065] Step S9: content pattern matching;

[0066] Step S10: storing compressed data;

[0067] Step S11: The user requests log data from the system;

[0068] Step S12: the system establishes an index structure;

[0069] Step S13: query index;

[0070] Step S14: Return data location;

[0071] Step S15: The system can read the compressed data from the storage area;

[0072] Step S16: the storage area returns compressed data;

[0073] Step S17: the system requests to decompress the data;

[0074] Step S18: The compression module restores the original data;

[0075] Step S19: The system returns compressed data to the user.

[0076] The embodiments of the present application also provide a log compression device. It should be noted that the log compression device of the embodiments of the present application can be used to execute the log compression method provided by the embodiments of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0077] The following is an introduction to the log compression device provided in the embodiment of the present application.

[0078] Figure 4 Schematic diagram of a log compression device according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0079] The classification unit 10 is used to obtain the log data to be compressed, classify the log data to be compressed, and obtain multiple target categories, wherein the log data to be compressed is the original log data that has not been compressed;

[0080] Specifically, in modern information systems, log data is continuously generated, recording various aspects of system operation, including but not limited to user activities, system events, error reports, performance indicators, etc. These log data are usually stored in text or binary format, which takes up a lot of storage space when not compressed. The first step to obtain the log data to be compressed is to ensure the integrity of the data, that is, to collect all relevant raw log data to avoid losing important information during the data collection stage. This usually involves grabbing log files from multiple source data points such as servers, network devices, and applications, and then storing these files in a unified manner to prepare for subsequent processing steps. Classified log data can reveal the inherent structure and regularity of the data, providing a more targeted processing basis for compression algorithms. Based on the various characteristics of log data, it can be classified into different target categories. For example, according to the timestamp of the log entry, the data can be grouped in chronological order or time interval, which helps to identify the time pattern and trend of the data and provides a basis for differential encoding. According to the log level, such as INFO, ERROR, DEBUG, etc., the log data can be divided into different severity categories, which is of great significance for optimizing log storage and fast retrieval. In addition, based on the similarity of log contents, repeated or similar event descriptions can be identified, which helps to further reduce data redundancy.

[0081] The calculation unit 20 is used to determine the reference log data to be compressed of each target category, and calculate the difference between each log data to be compressed in each target category and the reference log data to be compressed to obtain difference data, wherein the reference log data to be compressed is used for encoding the log data to be compressed;

[0082] Specifically, one or more entries are selected from the classified log data as the benchmark log data to be compressed, and these benchmark data will be used for subsequent difference calculations. After the benchmark log data to be compressed is determined, a difference calculation is performed between each log data to be compressed in each target category and the benchmark data. The purpose of the difference calculation is to identify and quantify the differences between the data, rather than storing the complete data content, thereby reducing the storage space occupied. The calculation of difference data can be performed on different parts of the data, including timestamps, log levels, and log content.

[0083] The compression unit 30 is used to encode the reference log data to be compressed and the difference data, generate simplified log data corresponding to each target category, and compress a plurality of simplified log data.

[0084] Specifically, the purpose of encoding the benchmark data is to convert it into a more compact form that is easier to store and transmit. Time difference encoding: record the difference between each log and the benchmark timestamp instead of the full date and time. For example, if the timestamp of the benchmark log is 1609459200 seconds, then the subsequent data with a timestamp of 1609459205 seconds only needs to be encoded as "5 seconds". Log level difference encoding: for entries with the same level as the benchmark log, the encoding can be omitted; for entries of different levels, the corresponding encoding value can be found in the log level dictionary through the index to store the encoding difference. Content difference encoding: identify the difference or supplementary information between each log and the benchmark content, and only store the changed part. For example, if the benchmark log records that user A visited a certain web page, and the subsequent entry records that user A visited another web page, it only needs to be encoded as "user ID = A, access URL = changed URL". Generate simplified log data: combine the encoded benchmark data and difference data to generate simplified log data corresponding to each target category. Simplified log data contains the key features of the original data, but is stored in a more compact form, reducing unnecessary duplication of information. Through the above steps, the storage and transmission costs of log data can be effectively reduced while maintaining the integrity and retrievability of the data, providing strong support for efficient data management and analysis.

[0085] Through this embodiment, the log data to be compressed is obtained, and the log data to be compressed is classified to obtain a plurality of target categories, wherein the log data to be compressed is the original log data that has not been compressed; the benchmark log data to be compressed of each target category is determined, and the difference between each log data to be compressed and the benchmark log data to be compressed in each target category is calculated to obtain the difference data, wherein the benchmark log data to be compressed is the log data to be compressed as a benchmark; the benchmark log data to be compressed and the difference data are encoded to generate the simplified log data corresponding to each target category, and the plurality of simplified log data are compressed. Compared with the prior art, in which the compression efficiency of the log compression device is low, the present application reduces the storage space of the log data and improves the compression efficiency by classifying the data to be compressed and encoding the benchmark data and the difference data for each category. Therefore, the problem of low compression efficiency of the log compression device in the prior art can be solved, and the compression efficiency of the log data can be improved.

[0086] In the specific implementation process, the classification unit includes a first division module, a second division module and a third division module. The first division module is used to divide the log data to be compressed whose timestamp difference is less than a preset threshold into time categories; the second division module is used to divide the log data to be compressed with the same log level into level categories; the third division module is used to divide the log data to be compressed with the same log content into content categories, and obtain a plurality of target categories. The device classifies logs through the above steps to obtain a plurality of target categories, effectively reducing the storage and transmission costs of log data, while maintaining the integrity and retrievability of the data, and providing strong support for efficient data management and analysis.

[0087] Specifically, the time category division is based on the timestamp information of the log data. The timestamp is one of the key attributes in the log data, which records the specific time when each log is generated. By dividing the log data to be compressed whose timestamp difference is less than the preset threshold into the same time category, efficient compression of time series data can be achieved. For example, if the preset threshold is set to 10 seconds, all log entries with a timestamp interval of less than 10 seconds will be classified into the same time category. The log level is a standard for measuring the importance and urgency of log information. Common log levels include INFO, ERROR, DEBUG, WARNING, etc. Dividing log entries with the same log level into level categories can make the compression device more intelligent and efficient. Extract log level information from each log entry as the basis for classification. Level category division: Log entries with the same log level are classified into the same category. For example, all ERROR level logs will be classified into the ERROR category. This helps to eliminate the repeated storage of log level information, especially when the log levels are consistent in the same category, the storage structure can be further optimized. Content parsing: Parse the text description of each log to extract key information, such as user ID, operation type, error information, etc. Content pattern recognition: Identify repeated or similar log content patterns. For example, multiple logs record the event of user login failure, but the specific reasons for failure may be different. Content classification: Log entries with the same pattern or similar content are classified into the same content category. For example, all logs that record user login failures will be classified into the "Login Failure" category. Through content classification, more efficient pattern matching and compression algorithms can be applied to reduce redundant storage. After completing the classification of time, level, and content, the system can further integrate these categories to generate a more detailed and structured classification of log data. For example, all log entries belonging to the same time category, the same level category, and the same content category can be combined into a specific target category. Each target category contains highly related and repeated features, which provides optimization opportunities for context-aware compression algorithms. Ultimately, the log data in each target category will be processed and compressed independently, thereby significantly reducing storage and transmission costs, while also ensuring data integrity and retrievability, which facilitates subsequent data analysis and troubleshooting. In this way, the system can manage log data more intelligently and improve overall operational efficiency and resource utilization.

[0088] In some optional implementations, the calculation unit includes a first determination module, a second determination module, and a third determination module. The first determination module is used to determine the log data to be compressed with the smallest timestamp as the reference log data to be compressed corresponding to the time category when the target category is the time category; the second determination module is used to use a number to represent each level category when the target category is the level category, and obtain the reference log data to be compressed corresponding to the level category; the third determination module is used to extract the same content from multiple log data to be compressed when the target category is the content category, and determine the same content as the reference log data to be compressed corresponding to the content category. The device can convert the log data under each target category into a more compact and structured form, laying a foundation for subsequent context-aware compression and fast retrieval.

[0089] In the specific implementation process, when dividing the log data into time categories, the log entry with the smallest timestamp (i.e., the earliest generated) is usually selected as the benchmark log data to be compressed. The log data with the smallest timestamp is usually the starting point of the time series, and all subsequent log data are calculated based on this time point to ensure the continuity and integrity of the time information. Therefore, by determining the log data with the smallest timestamp as the benchmark, the system can calculate the difference between all subsequent log entries and the benchmark, and then only store these difference information, which greatly reduces the storage space requirements. In the log level classification, the determination of the benchmark can be achieved by creating a level encoding dictionary, in which the name of each level is mapped to a digital code. For example, the dictionary can be defined as {'INFO':1,'ERROR':2,'DEBUG':3,'WARNING':4}. The process of determining the benchmark is actually to build this dictionary and map log entries of different levels to corresponding digital codes. Determining the benchmark data of the content category involves identifying duplicate content in multiple log data to be compressed. The data of each log is parsed to extract key information, such as user ID, access URL, error code, etc. Using pattern matching algorithms, log entries with the same or very similar content are identified. These entries may record different instances of the same event. For example, multiple logs record user login failures, but the specific reasons for failure may be different. Select one entry from the log entries with the same content as the baseline log data to be compressed. The selection criteria may include the typicality of the data, the frequency of occurrence, or the completeness of the information. For example, a log entry with the most complete error information can be selected as the baseline, because this helps to quickly locate the problem after decompression. For each log entry with the same content as the baseline, the difference information between it and the baseline is calculated and stored, such as the difference in user ID, the change in access time, etc. Through the above steps, the log data under each target category can be converted into a more compact and structured form, laying the foundation for subsequent context-aware compression and fast retrieval. In this process, the selection of baseline data is crucial, which affects the compression efficiency and the accuracy of the decompressed data. By utilizing the context information of the log data, the system can manage massive log data more intelligently and efficiently, while ensuring the integrity of the data and the convenience of retrieval.

[0090] In some optional embodiments, the compression unit includes a first encoding module, a second encoding module and a third encoding module. The first encoding module is used to encode the benchmark log data to be compressed and the difference data by using a differential encoding method when the target category is the time category, wherein the differential encoding method is a method of encoding the difference data; the second encoding module is used to encode the benchmark log data to be compressed and the difference data by using a dictionary encoding method when the target category is the level category, wherein the dictionary encoding method is a method of encoding by using numbers; the third encoding module is used to encode the benchmark log data to be compressed and the difference data by using a pattern matching method when the target category is the content category, wherein the pattern matching method is a method of encoding by using different contents.

[0091] Specifically, when the target category is the time category, the system uses differential encoding to compress the timestamp information. Differential encoding is an effective data compression technology that reduces storage requirements by recording the change value of a data point relative to the previous data point rather than its complete value. In log data processing, the specific steps of differential encoding are as described above. The log data with the smallest timestamp is selected as the benchmark for subsequent time difference calculations. For each subsequent log data, the difference between its timestamp and the benchmark timestamp is calculated. For example, if the benchmark timestamp is 1609459200 seconds, then the log entry with a timestamp of 1609459205 seconds only needs to store the difference information of "5 seconds". Differential encoding: Encode the calculated timestamp differences and store these differences instead of the complete timestamp value. Differential encoding can significantly reduce the storage requirements of timestamp information, especially in data-intensive time series. When processing log levels, dictionary encoding is an effective device that can convert the text representation of log levels into digital encodings, thereby reducing storage space. The steps of dictionary encoding are as follows: Log level dictionary establishment: Create a log level dictionary to map the names of different levels to digital codes. For example, {'INFO':1,'ERROR':2,'WARNING':3,'DEBUG':4}. Determine the baseline log data to be compressed corresponding to each level category, that is, the first log level that appears, or select the most frequently occurring level as the baseline based on the frequency of use, and perform digital encoding. Level difference encoding: For log entries that are different from the baseline log level, search through the dictionary and replace them with the corresponding digital code. If the log level is the same as the baseline, the encoding can be omitted, and only a flag bit is stored to indicate that the level has not changed. Store the encoded log level information for subsequent compression processing and data retrieval. Select an entry from the log entries with the same content as the baseline log data to be compressed. The baseline content should contain the most common or important information in the data of this category. For each log entry with the same content as the baseline, analyze the differences between it and the baseline, such as user ID, access time, etc. Based on the content difference, a pattern matching algorithm is used to encode the difference information. For example, repeated strings or information in a fixed format can be replaced by storing a pattern ID and the changed part instead of the complete log content. The pattern matching encoded data is stored, and these data only contain the difference from the baseline content, thereby significantly reducing the storage space occupied. By adopting the above encoding method, the system can not only effectively compress log data and reduce storage and transmission costs, but also maintain data integrity and retrieval convenience. These encoding strategies make full use of the internal structure and context information of log data, and are an important component of realizing context-aware log compression devices.

[0092] In some optional implementations, the classification unit includes an acquisition module for acquiring the timestamp, log level, and log content of the log data to be compressed.

[0093] In the specific implementation process, the timestamp is a key attribute in the log entry, which records the exact time when each log is generated. The extraction and processing of timestamps are the basis for time series analysis and context-aware compression. Log parsing: Use log parsing tools or custom parsing logic to accurately extract timestamp information from the text content of the log entry. This may involve the use of regular expressions to match timestamps in different formats. The log level reflects the severity and type of the log information. Common levels include INFO, ERROR, WARNING, DEBUG, etc. The log content contains a detailed description of the system operation status, user behavior, error information, etc. Through the above steps, the key attributes of the log data to be compressed can be fully and accurately obtained, providing the necessary information for the subsequent context-aware compression scheme. This information is not only used for compression, but also for building indexes, so that the compressed log data can be quickly retrieved and decompressed when needed, maintaining its integrity and availability.

[0094] In other optional embodiments, the device also includes a storage unit for storing the compressed simplified log data in a predetermined storage format after compressing the multiple simplified log data, wherein the predetermined storage format includes the simplified log data corresponding to each of the target categories.

[0095] Specifically, for each target category (time, level, content), the compressed simplified log data is divided into several storage blocks. Each storage block contains one or more simplified log data entries, as well as the context information of the block, such as timestamp range, log level, content mode, etc. The beginning of each storage block should contain block header information for quickly locating and identifying the block content. The block header information usually includes the start and end timestamps of the block, the log level encoding dictionary, the content mode ID, etc. In order to support fast retrieval, it is crucial to design a multi-level index structure. The index should include time index, level index, and content index to find specific log records on demand. By indexing the storage blocks by time range, users can quickly locate the log data within a certain time window. By classifying and indexing the storage blocks by log level, users can filter the log information of a specific level. Indexing is based on content mode ID or keywords to support content-related log queries. In addition to indexing, metadata needs to be managed, including compression algorithm, compression ratio, block size, number of blocks, etc. This information is crucial for data management and performance optimization.

[0096] In some optional embodiments, the device further includes a calculation unit for calculating the sum of the reference log data to be compressed and the difference data to obtain decompressed data after compressing the plurality of simplified log data and upon receiving a decompression instruction.

[0097] In the specific implementation process, according to the query conditions in the decompression instruction, such as time range, log level, etc., the pre-built index structure is used to find the storage block containing the relevant simplified log data. After locating the storage block, the block header information and compressed data are read. The block header information contains the context information used during compression, such as timestamp range, log level encoding dictionary, content mode ID, etc., which are necessary references for decompression. The benchmark log data to be compressed is obtained from the context information of the storage block. For example, in the time category, the benchmark data is the log entry with the smallest timestamp; in the level category, the benchmark data is the encoding of the corresponding log level of the category; in the content category, the benchmark data is the log entry with the same content mode. The difference data stored in the block is read. These data are simplified during compression, and only the change information relative to the benchmark data is retained, such as the difference in timestamps, the encoding of log level changes, and the difference description of content modes. According to the data type and encoding method, the difference data is calculated with the benchmark data. For the time category, add the timestamp difference to the base timestamp to restore the original timestamp; for the level category, convert the digital code back to the original log level according to the encoding dictionary; for the content category, restore the complete log content according to the pattern difference description and the base content. Assume that the base timestamp is 1609459200 seconds and the difference data is [5,10,20]. The process of calculating the sum is as follows: For the first difference 5, calculate 1609459200+5=1609459205 seconds. For the second difference 10, calculate 1609459205 (previous recovery timestamp)+10=1609459215 seconds. For the third difference 20, calculate 1609459215+20=1609459235 seconds.

[0098] Log level decompression example: Assume that the base log level is INFO, the code is 1, and the difference data is [0,1,2], which respectively indicates that the level remains unchanged, changes to ERROR, and changes to WARNING. Convert back to the original level according to the dictionary code: For the first difference value 0, the level remains unchanged and is INFO. For the second difference value 1, according to the code dictionary, INFO(1)+1=ERROR(2). For the third difference value 2, INFO(1)+2=WARNING(3).

[0099] Content category decompression example: Assume that the baseline content is "User login successful", and the difference data is [{"user_id":1001,"login_time":"2023-01-01 00:00:05"}, {"user_id":1002,"login_time":"2023-01-0100:00:10"}]. Restore the complete log content: For the first difference data, the complete log is "User ID 1001 successfully logged in at 2023-01-01 00:00:05". For the second difference data, the complete log is "User ID 1002 successfully logged in at 2023-01-01 00:00:10".

[0100] Through the above steps, the system can quickly and accurately restore the original log data without storing the complete original information, thus greatly saving storage space. The decompression process combines the context information during compression to ensure the integrity and consistency of data recovery while maintaining high efficiency, meeting the needs of log management and analysis in a big data environment.

[0101] The log compression device includes a processor and a memory. The classification unit, calculation unit, compression unit, etc. are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.

[0102] The processor contains a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the log compression efficiency can be improved by adjusting the kernel parameters.

[0103] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0104] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the log compression method.

[0105] Specifically, the log compression method includes:

[0106] Step S201, obtaining log data to be compressed, classifying the log data to be compressed, and obtaining multiple target categories, wherein the log data to be compressed is original log data that has not been compressed;

[0107] Specifically, in modern information systems, log data is continuously generated, recording various aspects of system operation, including but not limited to user activities, system events, error reports, performance indicators, etc. These log data are usually stored in text or binary format, which takes up a lot of storage space when not compressed. The first step to obtain the log data to be compressed is to ensure the integrity of the data, that is, to collect all relevant raw log data to avoid losing important information during the data collection stage. This usually involves grabbing log files from multiple source data points such as servers, network devices, and applications, and then storing these files in a unified manner to prepare for subsequent processing steps. Classified log data can reveal the inherent structure and regularity of the data, providing a more targeted processing basis for compression algorithms. Based on the various characteristics of log data, it can be classified into different target categories. For example, according to the timestamp of the log entry, the data can be grouped in chronological order or time interval, which helps to identify the time pattern and trend of the data and provides a basis for differential encoding. According to the log level, such as INFO, ERROR, DEBUG, etc., the log data can be divided into different severity categories, which is of great significance for optimizing log storage and fast retrieval. In addition, based on the similarity of log contents, repeated or similar event descriptions can be identified, which helps to further reduce data redundancy.

[0108] Step S202, determining the reference log data to be compressed of each target category, and calculating the difference between each log data to be compressed in each target category and the reference log data to be compressed to obtain difference data, wherein the reference log data to be compressed is used for encoding the log data to be compressed;

[0109] Specifically, one or more entries are selected from the classified log data as the benchmark log data to be compressed, and these benchmark data will be used for subsequent difference calculations. After the benchmark log data to be compressed is determined, a difference calculation is performed between each log data to be compressed in each target category and the benchmark data. The purpose of the difference calculation is to identify and quantify the differences between the data, rather than storing the complete data content, thereby reducing the storage space occupied. The calculation of difference data can be performed on different parts of the data, including timestamps, log levels, and log content.

[0110] Step S203: Encode the reference log data to be compressed and the difference data to generate simplified log data corresponding to each target category, and compress a plurality of simplified log data.

[0111] Specifically, the purpose of encoding the benchmark data is to convert it into a more compact form that is easier to store and transmit. Time difference encoding: record the difference between each log and the benchmark timestamp instead of the full date and time. For example, if the timestamp of the benchmark log is 1609459200 seconds, then the subsequent data with a timestamp of 1609459205 seconds only needs to be encoded as "5 seconds". Log level difference encoding: for entries with the same level as the benchmark log, the encoding can be omitted; for entries of different levels, the corresponding encoding value can be found in the log level dictionary through the index to store the encoding difference. Content difference encoding: identify the difference or supplementary information between each log and the benchmark content, and only store the changed part. For example, if the benchmark log records that user A visited a certain web page, and the subsequent entry records that user A visited another web page, it only needs to be encoded as "user ID = A, access URL = changed URL". Generate simplified log data: combine the encoded benchmark data and difference data to generate simplified log data corresponding to each target category. Simplified log data contains the key features of the original data, but is stored in a more compact form, reducing unnecessary duplication of information. Through the above steps, the storage and transmission costs of log data can be effectively reduced while maintaining the integrity and retrievability of the data, providing strong support for efficient data management and analysis.

[0112] An embodiment of the present invention provides an electronic device, including a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, at least the following steps are implemented:

[0113] Step S201, obtaining log data to be compressed, classifying the log data to be compressed, and obtaining multiple target categories, wherein the log data to be compressed is original log data that has not been compressed;

[0114] Specifically, in modern information systems, log data is continuously generated, recording various aspects of system operation, including but not limited to user activities, system events, error reports, performance indicators, etc. These log data are usually stored in text or binary format, which takes up a lot of storage space when not compressed. The first step to obtain the log data to be compressed is to ensure the integrity of the data, that is, to collect all relevant raw log data to avoid losing important information during the data collection stage. This usually involves grabbing log files from multiple source data points such as servers, network devices, and applications, and then storing these files in a unified manner to prepare for subsequent processing steps. Classified log data can reveal the inherent structure and regularity of the data, providing a more targeted processing basis for compression algorithms. Based on the various characteristics of log data, it can be classified into different target categories. For example, according to the timestamp of the log entry, the data can be grouped in chronological order or time interval, which helps to identify the time pattern and trend of the data and provides a basis for differential encoding. According to the log level, such as INFO, ERROR, DEBUG, etc., the log data can be divided into different severity categories, which is of great significance for optimizing log storage and fast retrieval. In addition, based on the similarity of log contents, repeated or similar event descriptions can be identified, which helps to further reduce data redundancy.

[0115] Step S202, determining the reference log data to be compressed of each target category, and calculating the difference between each log data to be compressed in each target category and the reference log data to be compressed to obtain difference data, wherein the reference log data to be compressed is used for encoding the log data to be compressed;

[0116] Specifically, one or more entries are selected from the classified log data as the benchmark log data to be compressed, and these benchmark data will be used for subsequent difference calculations. After the benchmark log data to be compressed is determined, a difference calculation is performed between each log data to be compressed in each target category and the benchmark data. The purpose of the difference calculation is to identify and quantify the differences between the data, rather than storing the complete data content, thereby reducing the storage space occupied. The calculation of difference data can be performed on different parts of the data, including timestamps, log levels, and log content.

[0117] Step S203: Encode the reference log data to be compressed and the difference data to generate simplified log data corresponding to each target category, and compress a plurality of simplified log data.

[0118] Specifically, the purpose of encoding the benchmark data is to convert it into a more compact form that is easier to store and transmit. Time difference encoding: record the difference between each log and the benchmark timestamp instead of the full date and time. For example, if the timestamp of the benchmark log is 1609459200 seconds, then the subsequent data with a timestamp of 1609459205 seconds only needs to be encoded as "5 seconds". Log level difference encoding: for entries with the same level as the benchmark log, the encoding can be omitted; for entries of different levels, the corresponding encoding value can be found in the log level dictionary through the index to store the encoding difference. Content difference encoding: identify the difference or supplementary information between each log and the benchmark content, and only store the changed part. For example, if the benchmark log records that user A visited a certain web page, and the subsequent entry records that user A visited another web page, it only needs to be encoded as "user ID = A, access URL = changed URL". Generate simplified log data: combine the encoded benchmark data and difference data to generate simplified log data corresponding to each target category. Simplified log data contains the key features of the original data, but is stored in a more compact form, reducing unnecessary duplication of information. Through the above steps, the storage and transmission costs of log data can be effectively reduced while maintaining the integrity and retrievability of the data, providing strong support for efficient data management and analysis.

[0119] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0120] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in each embodiment of the present application:

[0121] Step S201, obtaining log data to be compressed, classifying the log data to be compressed, and obtaining multiple target categories, wherein the log data to be compressed is original log data that has not been compressed;

[0122] Specifically, in modern information systems, log data is continuously generated, recording various aspects of system operation, including but not limited to user activities, system events, error reports, performance indicators, etc. These log data are usually stored in text or binary format, which takes up a lot of storage space when not compressed. The first step to obtain the log data to be compressed is to ensure the integrity of the data, that is, to collect all relevant raw log data to avoid losing important information during the data collection stage. This usually involves grabbing log files from multiple source data points such as servers, network devices, and applications, and then storing these files in a unified manner to prepare for subsequent processing steps. Classified log data can reveal the inherent structure and regularity of the data, providing a more targeted processing basis for compression algorithms. Based on the various characteristics of log data, it can be classified into different target categories. For example, according to the timestamp of the log entry, the data can be grouped in chronological order or time interval, which helps to identify the time pattern and trend of the data and provides a basis for differential encoding. According to the log level, such as INFO, ERROR, DEBUG, etc., the log data can be divided into different severity categories, which is of great significance for optimizing log storage and fast retrieval. In addition, based on the similarity of log contents, repeated or similar event descriptions can be identified, which helps to further reduce data redundancy.

[0123] Step S202, determining the reference log data to be compressed of each target category, and calculating the difference between each log data to be compressed in each target category and the reference log data to be compressed to obtain difference data, wherein the reference log data to be compressed is used for encoding the log data to be compressed;

[0124] Specifically, one or more entries are selected from the classified log data as the benchmark log data to be compressed, and these benchmark data will be used for subsequent difference calculations. After the benchmark log data to be compressed is determined, a difference calculation is performed between each log data to be compressed in each target category and the benchmark data. The purpose of the difference calculation is to identify and quantify the differences between the data, rather than storing the complete data content, thereby reducing the storage space occupied. The calculation of difference data can be performed on different parts of the data, including timestamps, log levels, and log content.

[0125] Step S203: Encode the reference log data to be compressed and the difference data to generate simplified log data corresponding to each target category, and compress a plurality of simplified log data.

[0126] Specifically, the purpose of encoding the benchmark data is to convert it into a more compact form that is easier to store and transmit. Time difference encoding: record the difference between each log and the benchmark timestamp instead of the full date and time. For example, if the timestamp of the benchmark log is 1609459200 seconds, then the subsequent data with a timestamp of 1609459205 seconds only needs to be encoded as "5 seconds". Log level difference encoding: for entries with the same level as the benchmark log, the encoding can be omitted; for entries of different levels, the corresponding encoding value can be found in the log level dictionary through the index to store the encoding difference. Content difference encoding: identify the difference or supplementary information between each log and the benchmark content, and only store the changed part. For example, if the benchmark log records that user A visited a certain web page, and the subsequent entry records that user A visited another web page, it only needs to be encoded as "user ID = A, access URL = changed URL". Generate simplified log data: combine the encoded benchmark data and difference data to generate simplified log data corresponding to each target category. Simplified log data contains the key features of the original data, but is stored in a more compact form, reducing unnecessary duplication of information. Through the above steps, the storage and transmission costs of log data can be effectively reduced while maintaining the integrity and retrievability of the data, providing strong support for efficient data management and analysis.

[0127] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0128] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0129] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0133] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0134] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0135] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0136] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0137] 1) In the log compression method of the present application, the log data to be compressed is obtained, and the log data to be compressed is classified to obtain a plurality of target categories, wherein the log data to be compressed is the original log data that has not been compressed; the benchmark log data to be compressed of each target category is determined, and the difference between each log data to be compressed and the benchmark log data to be compressed in each target category is calculated to obtain the difference data, wherein the benchmark log data to be compressed is the log data to be compressed as a benchmark; the benchmark log data to be compressed and the difference data are encoded to generate simplified log data corresponding to each target category, and the plurality of simplified log data are compressed. Compared with the low compression efficiency of the log compression method in the prior art, the present application reduces the storage space of the log data and improves the compression efficiency by classifying the data to be compressed and encoding the benchmark data and the difference data for each category. Therefore, the problem of low compression efficiency of the log compression method in the prior art can be solved, and the compression efficiency of the log data can be improved.

[0138] 2) In the log compression device of the present application, the log data to be compressed is obtained, and the log data to be compressed is classified to obtain a plurality of target categories, wherein the log data to be compressed is the original log data that has not been compressed; the benchmark log data to be compressed of each target category is determined, and the difference between each log data to be compressed and the benchmark log data to be compressed in each target category is calculated to obtain the difference data, wherein the benchmark log data to be compressed is the log data to be compressed as a benchmark; the benchmark log data to be compressed and the difference data are encoded to generate simplified log data corresponding to each target category, and the plurality of simplified log data are compressed. Compared with the low compression efficiency of the log compression method in the prior art, the present application reduces the storage space of the log data and improves the compression efficiency by classifying the data to be compressed and encoding the benchmark data and the difference data for each category. Therefore, the problem of low compression efficiency of the log compression method in the prior art can be solved, and the compression efficiency of the log data can be improved.

[0139] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A log compression method, characterized in that: include: Obtaining log data to be compressed, and classifying the log data to be compressed to obtain multiple target categories, wherein the log data to be compressed is original log data that has not been compressed; Determine the reference log data to be compressed of each target category, and calculate the difference between each log data to be compressed in each target category and the reference log data to be compressed to obtain difference data, wherein the reference log data to be compressed is used for encoding the log data to be compressed; The reference log data to be compressed and the difference data are encoded to generate simplified log data corresponding to each target category, and a plurality of simplified log data are compressed.

2. The log compression method according to claim 1, characterized in that: The log data to be compressed is classified to obtain multiple target categories, including: Classify the log data to be compressed whose timestamp difference is less than a preset threshold into time categories; Dividing the to-be-compressed log data having the same log level into level categories; The to-be-compressed log data with the same log content are divided into content categories to obtain a plurality of the target categories.

3. The log compression method according to claim 2, characterized in that: Determine the baseline log data to be compressed for each target category, including: In a case where the target category is the time category, determining the log data to be compressed with the smallest timestamp as the reference log data to be compressed corresponding to the time category; In the case where the target category is the level category, each level category is represented by a number to obtain the reference log data to be compressed corresponding to the level category; In the case where the target category is the content category, identical contents are extracted from a plurality of the log data to be compressed, and the identical contents are determined as the reference log data to be compressed corresponding to the content category.

4. The log compression method according to claim 2, characterized in that: Encoding the reference log data to be compressed and the difference data includes: In the case where the target category is the time category, the reference log data to be compressed and the difference data are encoded using a differential encoding method, wherein the differential encoding method is a method of encoding the difference data; In the case where the target category is the level category, the reference log data to be compressed and the difference data are encoded by using a dictionary encoding method, wherein the dictionary encoding method is a method of encoding by using numbers; In the case where the target category is the content category, the reference log data to be compressed and the difference data are encoded in a pattern matching manner, wherein the pattern matching manner is a manner of encoding with different contents.

5. The log compression method according to claim 1, characterized in that: Get the log data to be compressed, including: Obtain the timestamp, log level, and log content of the log data to be compressed.

6. The log compression method according to claim 1, characterized in that: After compressing the plurality of simplified log data, the method further comprises: The compressed simplified log data is stored in a predetermined storage format, wherein the predetermined storage format includes the simplified log data corresponding to each target category.

7. The log compression method according to claim 1, characterized in that: After compressing the plurality of simplified log data, the method further comprises: When a decompression instruction is received, the sum of the reference log data to be compressed and the difference data is calculated to obtain decompressed data.

8. A log compression device, characterized in that: include: a classification unit, configured to obtain log data to be compressed, classify the log data to be compressed, and obtain a plurality of target categories, wherein the log data to be compressed is original log data that has not been compressed; a calculation unit, configured to determine a reference log data to be compressed of each target category, and calculate a difference between each log data to be compressed in each target category and the reference log data to be compressed to obtain difference data, wherein the reference log data to be compressed is used for encoding the log data to be compressed; The compression unit is used to encode the reference log data to be compressed and the difference data, generate simplified log data corresponding to each target category, and compress a plurality of simplified log data.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the log compression method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the log compression method described in any one of claims 1 to 7.

Images