A device log compression method, device, medium and product

Abstract

The application discloses a device log compression method, device, medium and product, relates to the technical field of big data and data compression, and can be applied to the field of financial technology. The method comprises the following steps: when a preset compression time is reached, checking the IO load of a current system; in the case that the IO load is identified to reach a safety threshold, identifying and classifying at least one to-be-compressed log file respectively to obtain a target log type; according to a compression mode corresponding to the target log type, preliminarily compressing the to-be-compressed log to obtain a preliminarily compressed log; and based on an encryption algorithm, encrypting the preliminarily compressed log to obtain a target compressed log. Through the technical scheme, the log compression efficiency can be improved.

Description

Technical Field

[0001] This invention relates to the fields of big data and data compression technology, and can be used in the financial technology field. In particular, it relates to a device log compression method, device, medium, and product. Background Technology

[0002] With the large-scale deployment of cloud computing, distributed services, databases, and network devices, massive amounts of log data are continuously generated during system operation, business interactions, and equipment maintenance. These logs carry critical information such as system operating status, fault information, and operational behaviors, serving as the core basis for online fault diagnosis, system monitoring, and security auditing. Log text exhibits highly structured redundancy, with many log lines sharing fixed template statements, differing only in a few fields such as timestamps, process IDs, and business variables. Directly storing raw logs would consume massive amounts of disk space, significantly increasing storage hardware costs and log transmission bandwidth overhead.

[0003] Currently, the industry commonly uses general lossless compression algorithms such as gzip, LZ4, and Zstandard to compress logs as a whole. These general algorithms only rely on sliding window mining of local character repetition relationships and cannot identify the globally unified template structure of logs. They are difficult to fully eliminate large-scale template redundancy in logs, resulting in low compression ratios. Existing dedicated compression solutions for logs mostly rely on offline log parsing, manual template configuration, or complex clustering to extract fixed templates. In real-time log stream scenarios, the parsing overhead is high, the compression throughput is insufficient, and the template library update lag can easily cause compression failure. At the same time, log retrieval after compression requires full decompression, resulting in high query latency. It is difficult to balance compression efficiency, compression rate, and retrieval performance. Therefore, there is an urgent need for a log compression solution that adapts to real-time log streams, makes full use of log template redundancy, and balances compression speed and query efficiency. Summary of the Invention

[0004] This invention provides a device log compression method, device, medium, and product to improve log compression efficiency.

[0005] According to one aspect of the present invention, a device log compression method is provided, the method comprising: When the preset compression time is reached, check the current system I / O load; If the IO load is found to have reached a safe threshold, at least one log file to be compressed is identified and classified to obtain the target log type. Based on the compression method corresponding to the target log type, the log to be compressed is initially compressed to obtain a pre-compressed log; The initial compressed log is encrypted using an encryption algorithm to obtain the target compressed log.

[0006] According to another aspect of the present invention, a device log compression apparatus is provided, the apparatus comprising: The load check module is used to check the current system's I / O load when the preset compression time is reached; The log type determination module is used to identify and classify at least one log file to be compressed to obtain the target log type when the IO load is found to have reached a safety threshold. The log preliminary compression module is used to perform preliminary compression on the log to be compressed according to the compression method corresponding to the target log type, so as to obtain a preliminary compressed log; The log encryption module is used to encrypt the preliminary compressed log based on an encryption algorithm to obtain the target compressed log.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the device log compression method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the device log compression method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the device log compression method according to any embodiment of the present invention.

[0010] The technical solution of this invention involves checking the current system's IO load when a preset compression time is reached; if the IO load reaches a safety threshold, identifying and classifying at least one log file to be compressed to obtain a target log type; performing preliminary compression on the log file according to the compression method corresponding to the target log type to obtain a preliminary compressed log; and encrypting the preliminary compressed log based on an encryption algorithm to obtain the target compressed log. This technical solution, by real-time monitoring of the system's IO load, can distribute compression tasks to idle periods, effectively avoiding IO load spikes and ensuring the performance stability and response speed of digital certificate devices during peak business periods; simultaneously, classifying and specifically compressing the log file improves compression efficiency.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a device log compression method according to an embodiment of the present invention; Figure 2 This is a flowchart of a device log compression method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a device log compression device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the device log compression method of this invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Furthermore, it should be noted that the information collected in the technical solution of this invention is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data all comply with the relevant laws, regulations and standards of relevant countries and regions, necessary confidentiality measures have been taken, and public order and good morals are not violated. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] Currently, general compression algorithms such as gzip are not optimized for the structured characteristics of logs. They treat logs as ordinary byte streams and cannot effectively identify and utilize the large number of semantic repetition patterns contained in log messages. For example, for massive amounts of similar operation records that differ only in timestamps and certificate IDs, gzip still needs to repeatedly store a large amount of descriptive text, making it difficult to further improve the compression ratio. Furthermore, centralized compression timing (such as triggering at midnight) can lead to resource contention and IO load spikes. When multiple containers or service instances compress simultaneously during peak periods, it can drastically increase disk IO and CPU load, potentially affecting the performance of core business operations and even causing service delays in extreme cases. Existing solutions lack awareness of log file heterogeneity. Different log files (such as a few MB system snapshot sys_snap and a 1-3 GB system core dump app_core) have significant differences in content characteristics, importance, and compression potential. A uniform "one-size-fits-all" compression process cannot achieve fine-grained and optimized compression for different logs. Traditional streaming compression, while generating high compression ratio files, also sacrifices log searchability. To query a specific event, it is usually necessary to decrypt and decompress the entire file first, which is a cumbersome and time-consuming process that makes it difficult to quickly locate and audit faults.

[0018] Figure 1 This is a flowchart of a device log compression method according to an embodiment of the present invention. This embodiment is applicable to situations requiring improved device log compression efficiency. The method can be executed by a device log compression device, which can be implemented in hardware and / or software. This device can be configured in an electronic device carrying device log compression functionality, such as a server. Figure 1 As shown, the method includes: S110. When the preset compression time is reached, check the current system IO load.

[0019] In this embodiment, the preset compression time can be, for example, a system idle period. IO load refers to the busyness of the system's input / output interfaces when the compression task is triggered, typically measured by the ratio of non-empty IO queues or the number of read / write operations per second.

[0020] Specifically, when the preset compression time is reached, the system monitors the IO load indicators of the system in real time and checks the current IO load of the system.

[0021] S120. When the IO load is found to have reached the safety threshold, at least one log file to be compressed is identified and classified to obtain the target log type.

[0022] In this embodiment, the security threshold refers to a preset load threshold for IO load. The log to be compressed refers to the log that needs to be compressed. The target log type refers to the log type of the log to be compressed; optionally, the target log type includes system debug files or business log files.

[0023] An optional approach involves identifying and classifying the log files to be compressed to obtain the target log type, including: performing a rapid scan of the log files to be compressed to obtain the inherent structural characteristics of the logs to be compressed; and determining the target log type based on the inherent structural characteristics; wherein the target log type includes system debugging files or business log files.

[0024] Specifically, a rapid scan is performed on the logs to be compressed to analyze their internal structural characteristics. For example, if the logs to be compressed are relatively small (a few MB to 50 MB) and structurally rich files, such as system snapshots (sys_snap) and system core dumps (sys_core), then the target log type is determined to be a system debug file. If the logs to be compressed are large (e.g., 1 GB to 3 GB) and contain a large number of business operation records, such as system logs (sys_log) and application core logs (app_core), then the target log type is determined to be a large business log file.

[0025] Understandably, by analyzing the internal structure of the logs to be compressed and obtaining the corresponding target log type, a basis can be provided for subsequent log compression strategies.

[0026] Another alternative approach is to identify and classify at least one log file to be compressed to obtain the target log type, including: using a low-priority process to asynchronously identify and classify at least one log file to be compressed in the background to obtain the target log type.

[0027] Specifically, the compression task is executed asynchronously in the background by a low-priority process. This means that at least one log file to be compressed is identified and classified asynchronously to obtain the target log type. Resource usage limits are also set.

[0028] Understandably, it's crucial to ensure that even during large-scale compression, system resources aren't exhausted, impacting the performance of core functions like certificate loading / unloading. By using a load-aware scheduling mechanism to free compression tasks from fixed "midnight peak" periods and smoothly distribute them to idle times, I / O load spikes can be effectively avoided, ensuring the performance stability and response speed of digital certificate devices during peak business periods.

[0029] It should be noted that if the IO load is too high, the compression task will be delayed until the next detection cycle or the system load will reach a safe level.

[0030] S130. Based on the compression method corresponding to the target log type, perform preliminary compression on the log to be compressed to obtain the preliminary compressed log.

[0031] Specifically, based on the log compression method corresponding to the target log type, the log to be compressed is initially compressed to obtain the initially compressed log.

[0032] S140. Based on the encryption algorithm, encrypt the initial compressed log to obtain the target compressed log.

[0033] In this embodiment, the target compressed log refers to the final compressed log file.

[0034] Specifically, the initial compressed log is encrypted using an encryption algorithm such as AES-256 to obtain the target compressed log.

[0035] It should be noted that the encryption key is managed centrally by the device's security module. Simultaneously, the system maintains metadata generated during the compression process (such as a log template library and a feature mapping table). This metadata is the basis for subsequent log decompression and retrieval and needs to be securely stored along with the compressed data.

[0036] The technical solution of this invention involves checking the current system's IO load when a preset compression time is reached; if the IO load reaches a safety threshold, identifying and classifying at least one log file to be compressed to obtain a target log type; performing preliminary compression on the log file according to the compression method corresponding to the target log type to obtain a preliminary compressed log; and encrypting the preliminary compressed log based on an encryption algorithm to obtain the target compressed log. This technical solution, by real-time monitoring of the system's IO load, can distribute compression tasks to idle periods, effectively avoiding IO load spikes and ensuring the performance stability and response speed of digital certificate devices during peak business periods; simultaneously, classifying and specifically compressing the log file improves compression efficiency.

[0037] It should be noted that this invention avoids resource contention and ensures the smooth operation of the device's core business by using intelligent compressed task scheduling.

[0038] Figure 2 This is a flowchart of a device log compression method according to an embodiment of the present invention. Based on the above embodiments, this embodiment further optimizes the step of "performing preliminary compression of the log to be compressed according to the compression method corresponding to the target log type to obtain a preliminary compressed log," providing an optional solution. For example... Figure 2 As shown, the method includes: S210. When the preset compression time is reached, check the current system IO load.

[0039] S220. When the IO load is found to have reached the safety threshold, at least one log file to be compressed is identified and classified to obtain the target log type.

[0040] S230. Based on the compression method corresponding to the target log type, perform preliminary compression on the log to be compressed to obtain the preliminary compressed log.

[0041] S240. Based on the encryption algorithm, encrypt the preliminary compressed log to obtain the target compressed log.

[0042] An optional approach involves performing preliminary compression on the log to be compressed based on the compression method corresponding to the target log type, resulting in preliminary compressed logs. This includes: if the target log type is a system debug file, then a templated compression engine based on the parser tree is used to compress the log file to be compressed, resulting in preliminary compressed logs; if the target log type is a business log file, then a deep compression engine based on feature selection is used to compress the log file to be compressed, resulting in preliminary compressed logs.

[0043] Specifically, if the target log type to be compressed is a system debug file, the templated compression engine based on the parser tree compresses the log to be compressed to obtain a preliminary compressed log; if the target log type to be compressed is a business log file, the feature-selected deep compression engine compresses the log file to be compressed to obtain a preliminary compressed log.

[0044] Understandably, differentiated processing strategies for different log types directly eliminate structural redundancy in log data through a combination of templated compression and feature selection, achieving refined resource utilization, improving compression rates, and thus significantly reducing storage space and network bandwidth consumption.

[0045] For example, a template-based compression engine based on a parser tree compresses a log file to be compressed to obtain a preliminary compressed log. This includes: matching the log file to be compressed with the parser tree in the log template library to obtain a matching result; if the matching result is successful, replacing the template identifier in the constant part of the log file to be compressed to obtain a target template identifier; extracting the variable part of the log file to be compressed and encoding the variable part to obtain variable encoding; and performing lossless encoding on the target template identifier and variable encoding to obtain the preliminary compressed log.

[0046] In the log template library, templates are essentially nodes in the parser tree, used to describe the structure of a specific log message type. For example, the template: "User [UserID] performed [OperationType] operation at [Timestamp], certificate serial number is [CertSN], result [ResultCode]". In other words, a log template refers to a structured pattern extracted from a log message, containing constant strings and variable placeholders. For instance, the log line "User 1234 loaded certificate 6789 at 2025-06-15 10:00:00" can be abstracted into the template "User [*] loaded certificate [*] at [*]", where [*] is a variable placeholder.

[0047] Specifically, if the log file to be compressed is a system debugging file, it is matched against the parser tree in the log template library. This involves comparing constant strings and verifying the consistency of variable positions and types. If the match is successful, the constant portion of the log file is replaced with a short template identifier to obtain the target template identifier. The variable portion of the log file is then extracted and encoded to obtain variable codes. The log file is then converted into a tuple composed of the target template identifier and the variable codes. This tuple is further losslessly encoded. For example, since the target template identifier itself has a high degree of repetition, lightweight dictionary encoding can be used. The variable codes are then specifically encoded based on their data progression, such as timestamp differential encoding or variable-length integer encoding, to obtain the initial compressed log.

[0048] It is understandable that template-based compression can improve compression efficiency, and further lossless compression can reduce redundancy.

[0049] Furthermore, it also includes: if the matching result fails, performing word frequency and position analysis on the log to be compressed to obtain constant strings and variable strings; generating a new log template based on the constant strings and variable strings; and updating the log template library using the new log template.

[0050] Specifically, for log files to be compressed that do not match in the log template library, the frequency and position information of words in the log files to be compressed are analyzed, constant strings and variable strings are automatically distinguished, new log templates are generated, and the parser tree and log template library are updated.

[0051] Understandably, the dynamic update mechanism ensures that the compression system can adapt to the evolution of log formats.

[0052] For example, if the target log type is a business log file, the feature-based deep compression engine compresses the log file to be compressed to obtain a preliminary compressed log, including: identifying fields in the log file to be compressed to obtain key feature fields and non-key feature fields; filtering or aggregating the non-key feature fields; and performing sequence encoding on the key feature fields to obtain the preliminary compressed log.

[0053] Key feature fields include operation type, result code, error code, and certificate identifier. Non-key feature fields include highly refined auxiliary fields with extremely low information entropy.

[0054] Specifically, if the log file to be compressed is a business log file, then based on statistical methods or business rules, each field in the log file is analyzed to obtain key and non-key feature fields. Non-key feature fields are then filtered or aggregated; for example, detailed debugging information is summarized into statistical indicators. Key feature fields are then sequence encoded. For example, for numerical sequences within key feature fields, such as consecutive operation timestamps, more efficient specialized encoding schemes, such as inter-frame differential coding or run-length encoding, are used to obtain preliminary compressed logs. This eliminates redundancy in the time-series data.

[0055] It is understandable that filtering or aggregating non-critical feature fields can significantly reduce the amount of data that needs to be stored, thus achieving data dimensionality reduction; numerical sequence encoding of critical feature fields can eliminate redundancy in time series data. This deep compression using feature selection technology can maximize the compression ratio while retaining audit value.

[0056] The technical solution of this invention involves checking the current system's IO load when a preset compression time is reached; if the IO load reaches a safety threshold, identifying and classifying at least one log file to be compressed to obtain a target log type; performing preliminary compression on the log file according to the compression method corresponding to the target log type to obtain a preliminary compressed log; and encrypting the preliminary compressed log based on an encryption algorithm to obtain the target compressed log. This technical solution, by real-time monitoring of the system's IO load, can distribute compression tasks to idle periods, effectively avoiding IO load spikes and ensuring the performance stability and response speed of digital certificate devices during peak business periods; simultaneously, classifying and specifically compressing the log file improves compression efficiency.

[0057] It should be noted that the compressed log data in this invention is not a single byte stream, but retains structured index information. When it is necessary to retrieve a specific event (such as querying all operation records for a certain certificate serial number), it is not necessary to decompress the entire file. Quick queries and filtering can be performed directly on the compressed format, which can greatly improve operational efficiency and auditing convenience.

[0058] Figure 3 This is a schematic diagram of a device log compression device according to an embodiment of the present invention. This embodiment is applicable to situations requiring improved device log compression efficiency. The device log compression device can be implemented in hardware and / or software, and can be configured in an electronic device that carries the device log compression function, such as a server. Figure 3 As shown, the device includes: The load check module 310 is used to check the current system's I / O load when a preset compression time is reached; The log type determination module 320 is used to identify and classify at least one log file to be compressed to obtain the target log type when the IO load is found to have reached a safe threshold. The log preliminary compression module 330 is used to perform preliminary compression on the log to be compressed according to the compression method corresponding to the target log type, so as to obtain the preliminary compressed log. The log encryption module 340 is used to encrypt the preliminary compressed log based on the encryption algorithm to obtain the target compressed log.

[0059] The technical solution of this invention involves checking the current system's IO load when a preset compression time is reached; if the IO load reaches a safety threshold, identifying and classifying at least one log file to be compressed to obtain a target log type; performing preliminary compression on the log file according to the compression method corresponding to the target log type to obtain a preliminary compressed log; and encrypting the preliminary compressed log based on an encryption algorithm to obtain the target compressed log. This technical solution, by real-time monitoring of the system's IO load, can distribute compression tasks to idle periods, effectively avoiding IO load spikes and ensuring the performance stability and response speed of digital certificate devices during peak business periods; simultaneously, classifying and specifically compressing the log file improves compression efficiency.

[0060] Optionally, the log preliminary compression module 330 is used for: If the target log type is a system debug file, the templated compression engine based on the parser tree will compress the log file to be compressed to obtain a preliminary compressed log. If the target log type is a business log file, the feature-based deep compression engine compresses the log file to be compressed to obtain a preliminary compressed log.

[0061] Optionally, the log preliminary compression module 330 is specifically used for: The log file to be compressed is matched against the parser tree in the log template library to obtain the matching result; If the matching result is successful, the template identifier is replaced in the constant part of the log file to be compressed to obtain the target template identifier; Extract the variable portion of the log file to be compressed and encode the variable portion to obtain the variable code; Lossless encoding is performed on the target template identifier and variable encoding to obtain a preliminary compressed log.

[0062] Optionally, the device also includes a template library update module for: If the matching result fails, then the term frequency and position analysis are performed on the log to be compressed to obtain constant strings and variable strings; Generate a new log template based on the constant string and the variable string; Update the log template library with the new log template.

[0063] Optionally, the log preliminary compression module 330 is specifically used for: The log file to be compressed is subjected to field identification to obtain key feature fields and non-key feature fields; Filter or aggregate non-critical feature fields; Sequence encoding is performed on key feature fields to obtain preliminary compressed logs.

[0064] Optionally, the log type determination module 320 is used for: Perform a quick scan of the log file to be compressed to obtain its internal structural characteristics; The target log type is determined based on its inherent structural characteristics; the target log type includes system debug files or business log files.

[0065] Optionally, the log type determination module 320 is used for: The target log type is obtained by asynchronously identifying and classifying at least one log file to be compressed in the background using a low-priority process.

[0066] The device log compression device provided in this embodiment of the invention can execute the device log compression method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0067] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0068] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the device log compression method of this invention. Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0069] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0070] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0071] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the device log compression method.

[0072] In some embodiments, the device log compression method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the device log compression method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the device log compression method by any other suitable means (e.g., by means of firmware).

[0073] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0074] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0075] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0076] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0077] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0078] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system. This addresses the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0079] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0080] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for compressing device logs, characterized in that, include: When the preset compression time is reached, check the current system I / O load; If the IO load is found to have reached a safe threshold, at least one log file to be compressed is identified and classified to obtain the target log type. Based on the compression method corresponding to the target log type, the log to be compressed is initially compressed to obtain a pre-compressed log; The initial compressed log is encrypted using an encryption algorithm to obtain the target compressed log.

2. The method according to claim 1, characterized in that, Based on the compression method corresponding to the target log type, the log to be compressed is initially compressed to obtain a pre-compressed log, including: If the target log type is a system debug file, the templated compression engine based on the parser tree compresses the log file to be compressed to obtain a preliminary compressed log. If the target log type is a business log file, then the log file to be compressed is compressed by a feature-based deep compression engine to obtain a preliminary compressed log.

3. The method according to claim 2, characterized in that, The template-based compression engine, based on the parser tree, compresses the log file to be compressed, resulting in a preliminary compressed log, including: The log file to be compressed is matched with the parser tree in the log template library to obtain the matching result; If the matching result is successful, the template identifier is replaced in the constant part of the log file to be compressed to obtain the target template identifier; Extract the variable portion of the log file to be compressed, and encode the variable portion to obtain the variable code; The target template identifier and the variable encoding are losslessly encoded to obtain a preliminary compressed log.

4. The method according to claim 3, characterized in that, Also includes: If the matching result fails, then the term frequency and position analysis are performed on the log to be compressed to obtain constant strings and variable strings; Generate a new log template based on the constant string and the variable string; The log template library is updated using the new log template.

5. The method according to claim 2, characterized in that, If the target log type is a business log file, then the log file to be compressed is compressed using a feature-selective deep compression engine to obtain a preliminary compressed log, including: The log file to be compressed is subjected to field identification to obtain key feature fields and non-key feature fields; The non-critical feature fields are filtered or aggregated. The key feature fields are sequence encoded to obtain a preliminary compressed log.

6. The method according to claim 1, characterized in that, The log files to be compressed are identified and classified to obtain the target log type, including: A rapid scan of the log file to be compressed is performed to obtain the inherent structural features of the log file. The target log type is determined based on the inherent structural characteristics; wherein, the target log type includes system debug files or business log files.

7. The method according to claim 1, characterized in that, At least one log file to be compressed is identified and classified to obtain the target log type, including: The target log type is obtained by asynchronously identifying and classifying at least one log file to be compressed in the background using a low-priority process.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the device log compression method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the device log compression method of any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the device log compression method according to any one of claims 1-7.

Images