Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Log compression method and device, log decompression method and device and storage medium

A compression method and decompression technology, applied in special data processing applications, instruments, file systems, etc., can solve the problems of manpower and material resources, and achieve the effect of saving the use of storage space, increasing the compression ratio, and saving storage costs.

Inactive Publication Date: 2020-02-28
NANJING TRANSWARP INTELLIGENCE CO LTD
View PDF9 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In addition, operation and maintenance personnel are required for maintenance, which consumes a lot of manpower and material resources

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log compression method and device, log decompression method and device and storage medium
  • Log compression method and device, log decompression method and device and storage medium
  • Log compression method and device, log decompression method and device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0052] figure 1 It is a flowchart of a log compression method provided by Embodiment 1 of the present invention. This embodiment is applicable to the case where a clustered enterprise compresses a large number of daily log files. The method can be executed by a log compression device. It can be realized by means of software and / or hardware, and the device can be integrated in a processor of a computer or a server, such as figure 1 As shown, the method specifically includes:

[0053] Step 110, analyzing the original log file to obtain multiple column data of the log file;

[0054]Among them, the original log file records logs in a specific format by default. The data in the original log file may be in various forms or types of data. The original log file can complete the split analysis of the original log file according to the "log parsing rules" in the configuration file, for example, the data of the original log file is separated by the end of each column of data such as a...

Embodiment 2

[0069] image 3 It is a flow chart of a log compression method provided in Embodiment 2 of the present invention. This embodiment further refines the above technical solution. This embodiment can be combined with each optional solution in one or more of the above embodiments. .

[0070] Such as image 3 As shown, the method specifically includes:

[0071] Step 200, use the log file storing the cold data as the original log file, so as to compress the original log file.

[0072] Step 210, analyzing the original log file to obtain multiple column data of the log file.

[0073] Step 220, determine the information entropy of each column data.

[0074] Step 230, if the information entropy of the column data is less than or equal to the set threshold, replace all the data corresponding to the column name of the column data with character codes.

[0075] Optionally, replacing all data corresponding to the column name of the column data with a character encoding for compression i...

Embodiment 3

[0089] Figure 4 It is a flow chart of a log compression method provided by Embodiment 3 of the present invention. This embodiment further refines the above technical solution, and this embodiment can be combined with each optional solution in one or more of the above embodiments.

[0090] Such as Figure 4 As shown, the method specifically includes:

[0091] Step 300, use the log file storing the cold data as the original log file, so as to compress the original log file.

[0092] Step 310, analyzing the original log file to obtain multiple column data of the log file.

[0093] Step 320, determine the information entropy of each column data.

[0094] Step 330: If the information entropy of the column data is less than or equal to the set threshold, at least two rows of data in the column data are replaced in parallel with corresponding Huffman codes.

[0095] Step 340, if the information entropy of the column data is greater than the set threshold, keep the column data un...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention discloses a log compression method and device, a log decompression method and device and a storage medium. The log compression method comprises the following steps: analyzing an original log file to obtain multiple column data of the log file; determining the information entropy of each column of data; and if the information entropy of the column data is smaller than or equal to a set threshold, replacing all data corresponding to the column name of the column data with character codes for compression to obtain a compressed file of the log file. According to themethod, the storage space of the log file can be greatly saved, and the storage cost can be saved.

Description

technical field [0001] The embodiment of the present invention relates to the technical field of log compression and decompression, and in particular to a log compression and decompression method, device and storage medium. Background technique [0002] Generally speaking, a clustered enterprise-level application will generate hundreds of GB (capacity unit) or even terabytes (1TB = 1024GB) of log file data (for example: apache-access, tomcat-action logs) every day. For requirements such as auditing, level protection, and bug location, generally at least 1 to 6 months of historical log data should be backed up and stored. In this way, only to meet the log backup storage requirements, the enterprise will need to purchase multiple log servers and PB (1PB=1024TB) hard disks for storage. [0003] Common algorithms such as zip, tar, and gz are usually used to compress and store log files. Generally, the compression ratio can reach 10 to 20 times at most. That is to say, compressi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/174G06F16/18
CPCG06F16/1744G06F16/1815
Inventor 张晨黄南溪郭建新
Owner NANJING TRANSWARP INTELLIGENCE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com