A log shard compression storage method and device, electronic equipment and medium

Abstract

The application discloses a log fragment compression storage method and device, electronic equipment and medium. The current task increment log file corresponding to the current pipeline task is acquired in real time; each current standard to-be-fragmented compression log is generated according to each current to-be-fragmented compression log, and log blocking is performed in combination with a dynamic time window strategy to generate at least one current to-be-fragmented compression log block; each current to-be-fragmented compression log block is sequentially subjected to vertical fragmentation extraction through a sliding window algorithm to obtain each vertical log fragment and a remaining log not subjected to vertical fragmentation, and the remaining log not subjected to vertical fragmentation is subjected to horizontal fragmentation extraction in combination with a horizontal log fragmentation processing method to obtain each horizontal log fragment; and encoding processing is performed through a self-adaptive encoding processing method and the log fragment is stored in a log fragment compression storage database. The application solves the problem that current repetitive mode logs occupy a large amount of storage space, improves the utilization rate of storage resources, and reduces storage costs.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a log fragmentation compression storage method, apparatus, electronic device, and medium. Background Technology

[0002] During the execution of data processing pipeline tasks, massive amounts of runtime logs are generated. These logs can come from: logs output by user-written business scripts; logs output by compilers, build tools, or testing frameworks; and logs output by the underlying runtime platform or environment. These logs are typically stored in raw text format on a log server.

[0003] In the process of developing this invention, the inventors discovered the following shortcomings in the existing technology: Currently, for the same type of task, the same toolchain, or the same operating environment, the output log content exhibits a large number of highly similar or even repetitive patterns (such as fixed log frames, error messages, and environment information templates) and dynamic variables (such as timestamps, file paths, and numerical values). These repetitive pattern texts occupy a large amount of storage space, leading to high storage costs, wasted storage resources, and low data retrieval efficiency. Summary of the Invention

[0004] This invention provides a log fragmentation compression storage method, apparatus, electronic device, and medium to improve the utilization rate of storage resources and reduce storage costs.

[0005] According to one aspect of the present invention, a log fragmented compressed storage method is provided, comprising:

[0006] The current task incremental log file corresponding to the current pipeline task is acquired in real time; wherein the current task incremental log file includes at least one current log to be fragmented and compressed.

[0007] Based on each of the current logs to be compressed, the logs are processed by a pre-set log data preprocessing method to generate each current standard log to be compressed. The logs are then divided into blocks using a pre-set dynamic time window strategy to generate at least one current log block to be compressed corresponding to each of the current standard logs to be compressed.

[0008] Using the sliding window algorithm, each current log block to be compressed is sequentially segmented and extracted vertically to obtain each vertical log segment and the remaining log that has not been segmented vertically. Then, combined with the pre-set horizontal log segmentation processing method, the remaining log that has not been segmented vertically is segmented and extracted horizontally to obtain each horizontal log segment.

[0009] The vertical log shards and horizontal log shards are encoded using a pre-set adaptive encoding method and stored in the log shard compression storage database.

[0010] The log shard compression storage database uses different encoding methods based on the frequency of occurrence of different log shards. When the frequency of occurrence of log shards changes, the encoding method of the log shard compression storage database needs to be updated periodically and a new encoding version needs to be generated.

[0011] According to another aspect of the present invention, a log fragmented compressed storage device is provided, comprising:

[0012] The current task incremental log file acquisition module is used to acquire the current task incremental log file corresponding to the current pipeline task in real time; wherein, the current task incremental log file includes at least one current log to be fragmented and compressed;

[0013] The current log block generation module is used to generate current standard logs to be compressed by replacing each current log with a pre-set regular expression matching unified placeholder replacement method, and to divide the logs into blocks in combination with a pre-set dynamic time window strategy, thereby generating at least one current log block to be compressed corresponding to each current standard log.

[0014] The vertical log sharding and horizontal log sharding determination module is used to extract vertical shards from each current log block to be sharded and compressed sequentially using a sliding window algorithm to obtain each vertical log shard and the remaining log that has not been vertically sharded. The module also combines a pre-set horizontal log sharding processing method to extract horizontal shards from the remaining log that has not been vertically sharded to obtain each horizontal log shard.

[0015] The storage module is used to encode each vertical log shard and each horizontal log shard using a pre-set adaptive encoding method, and store the encoded data in the log shard compression storage database.

[0016] The log shard compression storage database uses different encoding methods based on the frequency of occurrence of different log shards. When the frequency of occurrence of log shards changes, the encoding method of the log shard compression storage database needs to be updated periodically and a new encoding version needs to be generated.

[0017] According to another aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the log fragmentation compression storage method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log fragmentation and compression storage method according to any embodiment of the present invention.

[0019] The technical solution of this invention involves acquiring the current task incremental log file corresponding to the current pipeline task in real time; wherein the current task incremental log file includes at least one current log to be sharded and compressed; processing each current log to be sharded and compressed using a pre-set log data preprocessing method to generate each current standard log to be sharded and compressed, and combining it with a pre-set dynamic time window strategy to divide the log into blocks, generating at least one current log block to be sharded and compressed corresponding to each current standard log to be sharded and compressed; using a sliding window algorithm, sequentially performing vertical sharding extraction on each current log block to be sharded and compressed to obtain each vertical log block and the remaining log that has not been vertically sharded, and combining it with a pre-set horizontal log sharding processing method to perform horizontal sharding extraction on the remaining log that has not been vertically sharded, to obtain each horizontal log block; and using a pre-set adaptive encoding processing method to encode each vertical log block and each horizontal log block, and storing them in a log sharding and compressed storage database.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a log fragmentation and compression storage method provided in Embodiment 1 of the present invention;

[0023] Figure 2 This is a detailed flowchart of a log fragmentation and compression storage method according to Embodiment 2 of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a log fragmented compression storage device according to Embodiment 3 of the present invention;

[0025] Figure 4This is a schematic diagram of the structure of an electronic device provided according to Embodiment 4 of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "target," "current," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] Example 1

[0029] Figure 1 The flowchart of a log fragment compression storage method provided in Embodiment 1 of the present invention describes a method for fragment compression storage of logs in a repetitive pattern in the current pipeline task. This method can be executed by a log fragment compression storage device, which can be implemented in hardware and / or software.

[0030] Correspondingly, such as Figure 1 As shown, the method includes:

[0031] S110. Real-time acquisition of the current task incremental log file corresponding to the current pipeline task.

[0032] The current task incremental log file includes at least one currently pending fragmentation and compression log.

[0033] Optionally, the real-time acquisition of the current task incremental log file corresponding to the current pipeline task includes: monitoring the status of the current pipeline task in real time through a log fragmentation and compression storage system; and acquiring the current incremental log file corresponding to the current pipeline task when the status of the current pipeline task is detected to be the end state.

[0034] In this embodiment, a log fragmentation and compression storage system is needed to monitor the status of the current pipeline task in real time. If the status is "continue running," monitoring needs to continue. When the status is "completed," it indicates that the current pipeline task has been completed, generating the corresponding current incremental log file. Data storage operations are required for this current incremental log file, hence the need for log fragmentation and compression storage.

[0035] The incremental log file for the current task can include one or more logs that are currently to be fragmented and compressed, and storage processing operations need to be performed on one or more of these logs.

[0036] S120. Based on each of the current logs to be compressed, process them using a pre-set log data preprocessing method to generate each current standard log to be compressed, and combine them with a pre-set dynamic time window strategy to divide the logs into blocks, generating at least one current log block to be compressed corresponding to each of the current standard logs to be compressed.

[0037] The log data preprocessing method may include normalizing the log data and performing regular expression matching to replace placeholders.

[0038] The current standard log to be fragmented and compressed can be the log obtained after preprocessing the log data. The dynamic time window strategy can be a strategy for fragmenting logs based on the timestamps of log entries.

[0039] Optionally, the step of processing each current unsharded compressed log using a pre-set log data preprocessing method to generate each current standard unsharded compressed log, and then performing log block generation using a pre-set dynamic time window strategy to generate at least one current unsharded compressed log block corresponding to each current standard unsharded compressed log, includes: normalizing each current unsharded compressed log, identifying dynamic variables in each current unsharded compressed log, and replacing them using a pre-set regular expression matching unified placeholder replacement method to generate each current standard unsharded compressed log; sequentially obtaining a target current standard unsharded compressed log from each current standard unsharded compressed log; and obtaining each log entry in the target current standard unsharded compressed log using a dynamic time window strategy. The process involves: determining the timestamp and the log block time threshold; calculating the time interval corresponding to consecutive log entries based on the timestamp of each log entry; determining whether the time interval is greater than the log block time threshold; if so, performing log block segmentation on the target current standard log to be segmented and compressing, continuing to traverse the log entries in the unsegmented log blocks, and returning to execute the operation based on the timestamp of each log entry until all log entries in the target current standard log to be segmented and compressing have been traversed; determining whether all current standard logs to be segmented and compressing have been traversed; if so, generating at least one current log block corresponding to each current standard log to be segmented and compressing; if not, returning to execute the operation of sequentially obtaining a target current standard log to be segmented and compressing from each current standard log to be segmented and compressing.

[0040] In this embodiment, each current log to be compressed into fragments first needs to be normalized to obtain a normalized log. Further, dynamic variables in each current log to be compressed into fragments are identified, and it is determined whether dynamic variables exist. If they exist, the corresponding dynamic variables need to be replaced according to the regular expression matching table corresponding to the unified placeholder replacement method, generating each current standard log to be compressed into fragments. The regular expression matching table can include dynamic variable types and unified placeholders corresponding to the dynamic variable types. Conversely, assuming no dynamic variables exist, each current standard log to be compressed into fragments is directly generated.

[0041] Furthermore, a dynamic time window strategy is needed to obtain the timestamps of each log entry in the target current standard log to be fragmented and compressed, as well as the log block time threshold. The target current standard log to be fragmented and compressed includes multiple log entries, arranged in chronological order. Accordingly, the time interval corresponding to consecutive log entries needs to be calculated based on the timestamps of each log entry.

[0042] For example, assume the target standard log to be fragmented and compressed contains 100 log entries; where log entries 1-10 have a timestamp of 12:12:10 PM on September 1, 2025; log entries 11-30 have a timestamp of 12:12:11 PM on September 1, 2025; log entries 31-41 have a timestamp of 12:12:15 PM on September 1, 2025; and log entries 42-50 have a timestamp of 12:12:30 PM on September 1, 2025. The time interval between log entries 10 and 11 can be calculated to be 1 second; the time interval between log entries 30 and 31 is 4 seconds; and the time interval between log entries 41 and 42 is 15 seconds. Assume the log fragmentation time threshold is 10 seconds.

[0043] Since the time interval between log entries 41 and 42 is 15 seconds, which exceeds the log chunking time threshold, log chunking needs to be performed between log entries 41 and 42 corresponding to the current standard log to be chunked and compressed. Continue iterating through the log entries in the un-chunked (i.e., the remaining 50 log entries) and repeat the above operations.

[0044] Furthermore, it is necessary to perform log block operations on all current standard logs to be compressed, thereby obtaining at least one current log block to be compressed corresponding to each current standard log.

[0045] The advantage of this setup is that by preprocessing log data and dividing each current standard log to be sharded and compressed into blocks, it can better assist in the implementation of log sharding operations, improve the efficiency of log sharding, and enhance the efficiency of log data processing.

[0046] S130. Using the sliding window algorithm, each current log block to be compressed is extracted vertically to obtain each vertical log block and the remaining log that has not been vertically divided. Then, the remaining log that has not been vertically divided is extracted horizontally using the pre-set horizontal log sharding processing method to obtain each horizontal log block.

[0047] Among them, the sliding window algorithm can be an algorithm that can perform vertical log fragment extraction operations by using sliding windows of different lengths.

[0048] Specifically, vertical log sharding can be performed on logs generated by the same type of task running in the same pipeline. These logs are characterized by high commonality, resulting in relatively long shards. Horizontal log sharding, on the other hand, can be performed on logs generated by different types of tasks running in the pipeline. This involves combining the remaining logs from the vertical sharding with the logs from the horizontal sharding. In detail, horizontal logs are characterized by low commonality and fragmentation.

[0049] Optionally, the step of sequentially extracting vertical fragments from each current log block to be compressed using a sliding window algorithm to obtain each vertical log fragment and the remaining logs not yet vertically fragmented includes: obtaining a pre-set minimum fragment length threshold; sequentially classifying each current log block to be compressed using a pre-trained deep learning classification model for log entries to obtain log entry types; obtaining a target current log block to be compressed from each current log block to be compressed; determining the maximum sliding window using a sliding window algorithm, and sliding the target current log block to be compressed according to the maximum sliding window, and determining the current sliding window based on the log entry type. If the log includes a uniform placeholder, extract the placeholder template structure and generate the current placeholder vertical log fragment; otherwise, perform long string pattern matching on the current sliding window log. If a match is found, the current vertical log fragment is identified. Reduce the maximum sliding window and determine if it is greater than or equal to the minimum fragment length threshold. If so, return to execute the operation of sliding the target current fragment compressed log block according to the maximum sliding window; otherwise, determine that the vertical fragment extraction is complete, obtain the remaining log without vertical fragmentation, and obtain each vertical log fragment according to each current vertical log fragment and each current placeholder vertical log fragment.

[0050] The log entry type can be a column type obtained by classifying the current logs to be sharded and compressed. Specifically, log entry types include business logs, platform logs, or test framework logs. Different types of log entry types include different placeholder template structures and different long string patterns.

[0051] In this embodiment, the maximum sliding window is first determined using a sliding window algorithm, assuming the maximum sliding window is L. This sliding window is then used to slide the target currently sharded and compressed log blocks. Depending on the log entry type, it is determined whether the current sliding window log includes a uniform placeholder. If so, the placeholder template structure is extracted first, and then the current placeholder vertical log shard is generated.

[0052] Furthermore, if it does not exist, it is necessary to perform long string pattern matching on the current sliding window log. If a match is found, the current vertical log shard is confirmed to be identified; if no match is found, the maximum sliding window L needs to be reduced, specifically, the length can be reduced to L-1.

[0053] The sliding operation continues to slide the target currently sharded and compressed log blocks using a sliding window of length L-1 until the sliding window length is less than the minimum shard length threshold.

[0054] Correspondingly, each current vertical log fragment and each current placeholder vertical log fragment can be used to obtain each vertical log fragment.

[0055] In this embodiment, the remaining logs that have not been vertically sharded need to be processed using a horizontal log sharding method. Specifically, the horizontal log sharding method includes a longest common subsequence processing method, the specific operations of which are as follows:

[0056] Optionally, the step of performing horizontal log sharding extraction on the remaining logs that have not been vertically sharded, using a pre-set horizontal log sharding processing method, to obtain each horizontal log shard, includes: obtaining a pre-set minimum length allowable threshold and a minimum shard occurrence frequency; obtaining an initial horizontal shard length using the longest common subsequence processing method; and performing horizontal sharding extraction on the remaining logs that have not been vertically sharded; wherein, the initial horizontal shard length is greater than the minimum length allowable threshold; if extraction is successful and the occurrence frequency of the extracted horizontal log shard is not lower than the minimum shard occurrence frequency, then the current horizontal log shard is obtained, and the initial horizontal shard length is reduced. It is then determined whether the length is greater than or equal to the minimum length allowable threshold. If so, the operation of obtaining the initial horizontal shard length using the longest common subsequence processing method and performing horizontal sharding extraction on the remaining logs that have not been vertically sharded is returned; after traversing the remaining logs that have not been vertically sharded, each horizontal log shard is obtained.

[0057] In this embodiment, the initial horizontal shard length needs to be obtained using the longest common subsequence processing method. This initial horizontal shard length is required to be greater than the minimum allowed length threshold. Then, the remaining logs that have not been vertically sharded are horizontally extracted. If extraction is successful, it is necessary to determine whether the occurrence frequency of the horizontal log shard is greater than or equal to the lowest shard occurrence frequency. Only if this condition is met can the current horizontal log shard be obtained. If the occurrence frequency of the extracted horizontal log shard is lower than the lowest shard occurrence frequency, it means that the extracted horizontal log shard cannot achieve significant compression benefits, i.e., it requires a large amount of storage memory and cannot effectively perform data compression operations. Therefore, the occurrence frequency of the extracted horizontal log shard needs to be considered.

[0058] Furthermore, the initial horizontal shard length needs to be reduced, but it cannot be less than the minimum allowed length threshold. Generally, the minimum allowed length threshold can be set to half of the minimum shard length threshold. Then, based on the different initial horizontal shard lengths, the remaining logs that have not been vertically sharded are horizontally extracted to obtain each horizontal log shard.

[0059] In this embodiment, the remaining logs that have not been vertically sharded need to be processed using a horizontal log sharding method. Specifically, the horizontal log sharding method may include a dynamic programming method or a greedy algorithm method, and the specific operation is as follows: First, the remaining logs that have not been vertically sharded need to be vectorized. Combining the pre-set minimum length allowable threshold and the minimum shard occurrence frequency, the similarity is further calculated with templates in a pre-built clustering sharding representative template library using a dynamic programming method or a greedy algorithm method, thereby generating the corresponding horizontal log shards.

[0060] The advantage of this setup is that it allows for better horizontal sharding of the remaining logs that were not vertically sharded, thus enabling more efficient storage of log shards and reducing database storage pressure and costs.

[0061] S140. Using a pre-set adaptive encoding method, each vertical log shard and each horizontal log shard are encoded and stored in the log shard compression storage database.

[0062] The log shard compression storage database uses different encoding methods based on the frequency of occurrence of different log shards. When the frequency of occurrence of log shards changes, the encoding method of the log shard compression storage database needs to be updated periodically and a new encoding version needs to be generated.

[0063] Specifically, the adaptive coding method can be set as the Huffman coding method.

[0064] In this embodiment, after obtaining each vertical log shard and each horizontal log shard, it is necessary to perform reasonable encoding and storage operations according to the adaptive encoding processing method and store them in the log shard compression storage database.

[0065] The technical solution of this invention involves acquiring the current task incremental log file corresponding to the current pipeline task in real time; processing each current log to be compressed using a pre-set log data preprocessing method to generate current standard logs to be compressed; and dividing the logs into blocks using a pre-set dynamic time window strategy to generate at least one current log block to be compressed corresponding to each current standard log to be compressed; using a sliding window algorithm, vertically extracting each current log block to be compressed sequentially to obtain vertical log blocks and the remaining logs not vertically divided; and horizontally extracting the remaining logs not vertically divided using a pre-set horizontal log block processing method to obtain horizontal log blocks; and encoding each vertical log block and each horizontal log block using a pre-set adaptive encoding method, and storing them in a log block compression storage database. This solves the problem of repetitive pattern logs occupying a large amount of storage space, improves the utilization rate of storage resources, reduces storage costs, reduces storage resource waste, and improves data retrieval efficiency.

[0066] Example 2

[0067] Figure 2 This is a detailed flowchart of a log fragment compression and storage method provided in Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiments. Specifically, it further refines the process of encoding each vertical log fragment and each horizontal log fragment using a pre-set adaptive encoding method and storing them in a log fragment compression and storage database.

[0068] S210. Real-time acquisition of the current task incremental log file corresponding to the current pipeline task.

[0069] The current task incremental log file includes at least one currently pending fragmentation and compression log.

[0070] S220. Based on each of the current logs to be compressed, process them using a pre-set log data preprocessing method to generate each current standard log to be compressed, and combine them with a pre-set dynamic time window strategy to divide the logs into blocks, generating at least one current log block to be compressed corresponding to each of the current standard logs to be compressed.

[0071] S230. Using the sliding window algorithm, each current log block to be compressed is extracted vertically to obtain each vertical log block and the remaining log that has not been vertically divided. Then, the remaining log that has not been vertically divided is extracted horizontally using the pre-set horizontal log sharding processing method to obtain each horizontal log block.

[0072] S240. According to the historical version encoding mapping rules, each vertical log fragment and each horizontal log fragment are encoded using an adaptive encoding processing method to obtain each vertical encoded compressed log fragment and each horizontal encoded compressed log fragment.

[0073] The log sharding and compression storage database includes historical version encoding mapping rules.

[0074] Each vertically encoded compressed log shard and each horizontally encoded compressed log shard includes an encoding version, shard fingerprint, shard type, shard encoding, and shard metadata.

[0075] Among them, the adaptive encoding processing method can be to assign shorter binary codes to log fragments that appear more frequently (including identical text segments and parameterized templates) in order to maximize the saving of overall storage space.

[0076] The log sharded compressed storage database includes historical version encoding mapping rules, with each historical version encoding mapping rule corresponding to a specific encoding version. After updating the historical version encoding mapping rules, the encoding version needs to be modified accordingly.

[0077] For example, if the encoding version of a historical encoding mapping rule is 1, after an update, the encoding version can be changed to 2.

[0078] The shard fingerprint can include the hash value of the shard content or a normalized template string. The shard metadata can include parameters such as the first occurrence time of the shard metadata, average length, and frequency statistics.

[0079] S250. Store each vertically encoded compressed log fragment and each horizontally encoded compressed log fragment in the log fragment compressed storage database.

[0080] Optionally, it also includes: periodically counting the occurrence frequency of each log fragment in the log fragment compression storage database; when the occurrence frequency of a log fragment changes, adjusting the encoding method of the log fragment according to a pre-set compression efficiency calculation formula, and updating the historical version encoding mapping rules in the log fragment compression storage database, saving the new version encoding mapping rules and the new encoding version; wherein, the new version encoding mapping rules and the new encoding version are loaded into the cache and marked as the current latest active version; using the new version encoding mapping rules and the new encoding version corresponding to the current latest active version to encode each vertical log fragment and each horizontal log fragment received at the next moment; for each vertical encoded compressed log fragment and each horizontal encoded compressed log fragment at the current moment, decoding is performed according to the corresponding encoding version to generate the original log.

[0081] In this embodiment, encoding needs to be performed according to the historical version encoding mapping rules. The historical version encoding mapping rules can include encoding methods corresponding to different types of log fragments. Specifically, for log fragments that are longer and more frequent, an encoding method with greater compression efficiency is adopted, thereby better realizing the compression encoding processing of log data.

[0082] Furthermore, each vertically coded compressed log fragment and each horizontally coded compressed log fragment can be obtained and stored in a log fragment compression storage database. The coded versions corresponding to historical version coded mapping rules also need to be stored. This allows for log restoration operations based on the coded version during the original log retrieval process.

[0083] The formula for calculating compression benefits can be: Where n is the total number of fragment types; For the first The total number of occurrences of each segment; For the first The average length of the original text fragments (where, for templates, it refers to the template string length plus the estimated length of typical parameters). For the first Z represents the average length of the encoded fragments; Z is the compression benefit. This formula shows that the larger the original fragment length and the more repetitions, the greater the compression contribution.

[0084] Therefore, it is necessary to periodically count the occurrence frequency of each log fragment in the log fragment compression storage database; when the occurrence frequency of log fragments changes, the encoding method of log fragments is adjusted according to the compression benefit calculation formula, that is, log fragments with longer lengths and higher occurrence frequencies are matched with relatively shorter binary codes.

[0085] This allows for updating the historical version encoding mapping rules in the log sharding and compression storage database to generate new version encoding mapping rules and new encoding versions.

[0086] Additionally, the new version encoding mapping rules and the new encoding version are loaded into the cache and marked as the latest active version. The new version encoding mapping rules and the new encoding version corresponding to the latest active version are then used to encode each vertical log shard and each horizontal log shard received in the next moment. This ensures that subsequent log shards are encoded using the latest active version, optimizing data storage and reducing storage pressure and cost. Simultaneously, historical version encoding mapping rules and corresponding encoding versions need to be saved to ensure correct decoding of historical log shards.

[0087] The technical solution of this invention involves: acquiring the current task incremental log file corresponding to the current pipeline task in real time; processing each current log to be compressed using a pre-set log data preprocessing method to generate each current standard log to be compressed; and dividing the log into blocks using a pre-set dynamic time window strategy to generate at least one current log block to be compressed corresponding to each current standard log to be compressed; extracting vertically from each current log block using a sliding window algorithm to obtain vertical log blocks and un-sharded logs; extracting horizontally from the un-sharded logs using a pre-set horizontal log sharding processing method to obtain horizontal log blocks; encoding each vertical log block and each horizontal log block using an adaptive encoding processing method according to the historical version encoding mapping rules to obtain vertical encoded compressed log blocks and each horizontal encoded compressed log block; and storing each vertical encoded compressed log block and each horizontal encoded compressed log block in a log block compression storage database. This ensures that subsequent log shards are encoded using the latest active version, optimizing data storage, reducing storage pressure and costs, improving storage resource utilization, reducing storage waste, and increasing data retrieval efficiency.

[0088] Example 3

[0089] Figure 3 This is a schematic diagram of a log fragment compression and storage device provided in Embodiment 3 of the present invention. The log fragment compression and storage device provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or server to implement a log fragment compression and storage method according to an embodiment of the present invention. Figure 3 As shown, the device includes: a current task incremental log file acquisition module 310, a current log block generation module 320, a vertical log fragment and horizontal log fragment determination module 330, and a storage module 340.

[0090] The current task incremental log file acquisition module 310 is used to acquire the current task incremental log file corresponding to the current pipeline task in real time; wherein, the current task incremental log file includes at least one current log to be fragmented and compressed.

[0091] The current log block generation module 320 is used to generate each current standard log to be compressed by replacing each current log to be compressed by a pre-set regular expression matching unified placeholder replacement method, and to divide the log into blocks in combination with a pre-set dynamic time window strategy, thereby generating at least one current log block to be compressed corresponding to each current standard log to be compressed.

[0092] The vertical log sharding and horizontal log sharding determination module 330 is used to extract vertical sharding from each current log block to be sharded and compressed sequentially using a sliding window algorithm to obtain each vertical log shard and the remaining log that has not been vertically sharded. It also combines a pre-set horizontal log sharding processing method to extract horizontal sharding from the remaining log that has not been vertically sharded to obtain each horizontal log shard.

[0093] Storage module 340 is used to encode each vertical log shard and each horizontal log shard using a pre-set adaptive encoding processing method, and store them in the log shard compression storage database.

[0094] The log shard compression storage database uses different encoding methods based on the frequency of occurrence of different log shards. When the frequency of occurrence of log shards changes, the encoding method of the log shard compression storage database needs to be updated periodically and a new encoding version needs to be generated.

[0095] The technical solution of this invention involves acquiring the current task incremental log file corresponding to the current pipeline task in real time; processing each current log to be compressed using a pre-set log data preprocessing method to generate current standard logs to be compressed; and dividing the logs into blocks using a pre-set dynamic time window strategy to generate at least one current log block to be compressed corresponding to each current standard log to be compressed; using a sliding window algorithm, vertically extracting each current log block to be compressed sequentially to obtain vertical log blocks and the remaining logs not vertically divided; and horizontally extracting the remaining logs not vertically divided using a pre-set horizontal log block processing method to obtain horizontal log blocks; and encoding each vertical log block and each horizontal log block using a pre-set adaptive encoding method, and storing them in a log block compression storage database. This solves the problem of repetitive pattern logs occupying a large amount of storage space, improves the utilization rate of storage resources, reduces storage costs, reduces storage resource waste, and improves data retrieval efficiency.

[0096] Based on the above embodiments, the current task incremental log file acquisition module 310 can be specifically used to monitor the status of the current pipeline task in real time through a log fragmentation and compression storage system; when the status of the current pipeline task is detected to be the end state, the incremental log file corresponding to the current pipeline task is acquired.

[0097] Based on the above embodiments, the current log segmentation generation module 320 can be specifically used to: normalize each current log segmentation log, identify dynamic variables in each current log segmentation log, and replace them using a pre-set regular expression matching unified placeholder replacement method to generate each current standard log segmentation log; sequentially obtain a target current standard log segmentation log from each current standard log segmentation log; obtain the timestamps of each log entry in the target current standard log segmentation log and the log segmentation time threshold using a dynamic time window strategy; and calculate the corresponding log entries based on the timestamps of each log entry. The time interval is determined; it is determined whether the time interval is greater than the log block time threshold; if so, the target current standard log to be compressed is divided into log blocks, and the log entries in the un-blocked logs are traversed. The operation based on the timestamp of each log entry is returned until all log entries in the target current standard log to be compressed are traversed; it is determined whether all current standard logs to be compressed have been traversed. If so, at least one current log block to be compressed is generated corresponding to each current standard log to be compressed. If not, the operation of sequentially obtaining a target current standard log to be compressed from each current standard log to be compressed is returned.

[0098] Based on the above embodiments, the vertical log sharding and horizontal log sharding determination module 330 can be specifically used to: obtain a pre-set minimum sharding length threshold; classify each of the current logs to be sharded and compressed sequentially using a pre-trained log entry deep learning classification model to obtain log entry types; obtain a target current log block to be sharded and compressed from each of the current log blocks to be sharded and compressed; determine the maximum sliding window using a sliding window algorithm, and slide the target current log block to be sharded and compressed according to the maximum sliding window; and determine whether the current sliding window log includes a uniform placeholder based on the log entry type. If yes, then the placeholder template structure is extracted to generate the current placeholder vertical log fragment; if no, then long string pattern matching is performed on the current sliding window log. If a match is found, the current vertical log fragment is identified. The maximum sliding window is reduced, and it is determined whether it is greater than or equal to the minimum fragment length threshold. If yes, then the operation of sliding the target current fragment to be compressed log block according to the maximum sliding window is returned; if no, then the vertical fragment extraction is completed, the remaining logs that have not been vertically fragmented are obtained, and each vertical log fragment is obtained according to each current vertical log fragment and each current placeholder vertical log fragment.

[0099] Based on the above embodiments, the horizontal log sharding processing method includes the longest common subsequence processing method.

[0100] Based on the above embodiments, the vertical log sharding and horizontal log sharding determination module 330 can also be specifically used to: obtain a pre-set minimum length allowable threshold and a minimum sharding occurrence frequency; obtain an initial horizontal sharding length through the longest common subsequence processing method, and perform horizontal sharding extraction on the remaining logs that are not vertically sharded; wherein, the initial horizontal sharding length is greater than the minimum length allowable threshold; if extraction is successful, and the occurrence frequency of the extracted horizontal log shards is not lower than the minimum sharding occurrence frequency, then the current horizontal log sharding is obtained, and the initial horizontal sharding length is reduced, and it is determined whether it is greater than or equal to the minimum length allowable threshold. If so, the operation of obtaining the initial horizontal sharding length through the longest common subsequence processing method and performing horizontal sharding extraction on the remaining logs that are not vertically sharded is returned; after traversing the remaining logs that are not vertically sharded, each horizontal log shard is obtained.

[0101] Based on the above embodiments, the log sharded compressed storage database includes historical version encoding mapping rules.

[0102] Based on the above embodiments, the storage module 340 can be specifically used to: encode each vertical log shard and each horizontal log shard according to the historical version encoding mapping rules, using an adaptive encoding processing method, to obtain each vertical encoded compressed log shard and each horizontal encoded compressed log shard; wherein each vertical encoded compressed log shard and each horizontal encoded compressed log shard includes an encoding version, shard fingerprint, shard type, shard encoding, and shard metadata; and store each vertical encoded compressed log shard and each horizontal encoded compressed log shard in a log shard compressed storage database.

[0103] Based on the above embodiments, it can also be specifically used for: periodically counting the occurrence frequency of each log fragment in the log fragment compression storage database; when the occurrence frequency of a log fragment changes, adjusting the encoding method of the log fragment according to a pre-set compression efficiency calculation formula, and updating the historical version encoding mapping rules in the log fragment compression storage database, saving the new version encoding mapping rules and the new encoding version; wherein, the new version encoding mapping rules and the new encoding version are loaded into the cache and marked as the current latest active version; using the new version encoding mapping rules and the new encoding version corresponding to the current latest active version to encode each vertical log fragment and each horizontal log fragment received at the next moment; for each vertical encoded compressed log fragment and each horizontal encoded compressed log fragment at the current moment, decoding is performed according to the corresponding encoding version to generate the original log.

[0104] The log fragment compression and storage device provided in the embodiments of the present invention can execute the log fragment compression and storage method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0105] Example 4

[0106] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement Embodiment 4 of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0107] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0108] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0109] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the log sharding compressed storage method.

[0110] In some embodiments, the log fragmentation compression storage method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the log fragmentation compression storage method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the log fragmentation compression storage method by any other suitable means (e.g., by means of firmware).

[0111] The method includes: acquiring the current task incremental log file corresponding to the current pipeline task in real time; wherein, the current task incremental log file includes at least one current log to be compressed; processing each current log to be compressed using a pre-set log data preprocessing method to generate each current standard log to be compressed, and performing log segmentation using a pre-set dynamic time window strategy to generate at least one current log segment to be compressed corresponding to each current standard log to be compressed; and extracting each current log segment vertically using a sliding window algorithm to obtain each vertical log segment. The remaining logs that have not been vertically sharded are extracted horizontally using a pre-set horizontal log sharding processing method to obtain horizontal log shards. Each vertical and horizontal log shard is then encoded using a pre-set adaptive encoding method and stored in a log shard compression storage database. The log shard compression storage database uses different encoding methods based on the frequency of occurrence of different log shards. When the frequency of occurrence of a log shard changes, the encoding method of the log shard compression storage database needs to be periodically updated and a new encoding version generated.

[0112] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0116] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0117] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0118] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0120] Example 5

[0121] Embodiment 5 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions, when executed by a computer processor, are used to execute a log fragmentation and compression storage method. The method includes: real-time acquisition of a current task incremental log file corresponding to the current pipeline task; wherein the current task incremental log file includes at least one current log to be fragmented and compressed; processing each of the current logs to be fragmented and compressed using a pre-set log data preprocessing method to generate each current standard log to be fragmented and compressed, and performing log segmentation using a pre-set dynamic time window strategy to generate at least one current log segment to be fragmented and compressed corresponding to each of the current standard logs to be fragmented and compressed; and sequentially processing the logs using a sliding window algorithm. Each current log block to be sharded and compressed is subjected to vertical sharding extraction to obtain each vertical log shard and the remaining logs that have not been vertically sharded. Then, using a pre-set horizontal log sharding processing method, the remaining logs that have not been vertically sharded are subjected to horizontal sharding extraction to obtain each horizontal log shard. Each vertical log shard and each horizontal log shard is encoded using a pre-set adaptive encoding processing method and stored in a log shard compression storage database. The log shard compression storage database uses different encoding methods based on the frequency of occurrence of different log shards. When the frequency of occurrence of a log shard changes, the encoding method of the log shard compression storage database needs to be periodically updated and a new encoding version generated.

[0122] Of course, the computer-executable instructions provided in the embodiments of the present invention, which include a computer-readable storage medium, are not limited to the method operations described above, but can also perform related operations in the log fragmented compressed storage provided in any embodiment of the present invention.

[0123] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0124] It is worth noting that in the above-described log fragmented compression storage embodiments, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0125] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A log fragmentation and compression storage method, characterized in that, include: The current task incremental log file corresponding to the current pipeline task is acquired in real time; wherein the current task incremental log file includes at least one current log to be fragmented and compressed. Based on each of the current logs to be compressed, the logs are processed by a pre-set log data preprocessing method to generate each current standard log to be compressed. The logs are then divided into blocks using a pre-set dynamic time window strategy to generate at least one current log block to be compressed corresponding to each of the current standard logs to be compressed. Using the sliding window algorithm, each current log block to be compressed is sequentially segmented and extracted vertically to obtain each vertical log segment and the remaining log that has not been segmented vertically. Then, combined with the pre-set horizontal log segmentation processing method, the remaining log that has not been segmented vertically is segmented and extracted horizontally to obtain each horizontal log segment. The vertical log shards and horizontal log shards are encoded using a pre-set adaptive encoding method and stored in the log shard compression storage database. The log shard compression storage database uses different encoding methods based on the frequency of occurrence of different log shards. When the frequency of occurrence of log shards changes, the encoding method of the log shard compression storage database needs to be updated periodically and a new encoding version needs to be generated.

2. The method according to claim 1, characterized in that, The real-time acquisition of the current task incremental log file corresponding to the current pipeline task includes: The status of the current pipeline tasks is monitored in real time through a log fragmentation and compression storage system; When the current pipeline task is detected to be in a completed state, the incremental log file corresponding to the current pipeline task is retrieved.

3. The method according to claim 2, characterized in that, The step involves processing each of the current logs to be compressed using a pre-set log data preprocessing method to generate current standard logs to be compressed, and then dividing the logs into blocks using a pre-set dynamic time window strategy to generate at least one current log block to be compressed corresponding to each of the current standard logs to be compressed, including: Normalize each of the current logs to be compressed and fragmented, identify the dynamic variables in each of the current logs to be compressed and fragmented, and replace them using a pre-set regular expression matching unified placeholder replacement method to generate each current standard log to be compressed and fragmented. In each of the current standard fragmented compression logs, a target current standard fragmented compression log is obtained sequentially; By using a dynamic time window strategy, the timestamps of each log entry in the target's current standard log to be fragmented and compressed are obtained, as well as the log block time threshold is obtained; The time interval corresponding to consecutive log entries is calculated based on the timestamps of each log entry. Determine whether the time interval is greater than the log block time threshold; If so, then the target current standard log to be fragmented and compressed is divided into log blocks, and the log entries in the un-fragmented log blocks are traversed. Then, the operation based on the timestamp of each log entry is returned and executed until all log entries in the target current standard log to be fragmented and compressed are traversed. Determine whether all current standard logs to be fragmented and compressed have been traversed. If yes, generate at least one current log block to be fragmented and compressed corresponding to each current standard log to be fragmented and compressed. If no, return to the operation of sequentially obtaining a target current standard log to be fragmented and compressed from each current standard log to be fragmented and compressed.

4. The method according to claim 3, characterized in that, The process involves using a sliding window algorithm to sequentially extract log fragments vertically from each current log block to be compressed, resulting in each vertical log fragment and the remaining logs not yet vertically fragmented, including: Get the pre-set minimum fragment length threshold; By using a pre-trained deep learning classification model for log entries, each of the current log entries to be compressed is classified sequentially to obtain the log entry type. In each of the current log blocks to be compressed, obtain the target current log block to be compressed; The maximum sliding window is determined by the sliding window algorithm, and the target current log block to be compressed is slid according to the maximum sliding window. Combined with the log entry type, it is determined whether the current sliding window log includes a uniform placeholder. If so, the placeholder template structure is extracted and the current placeholder vertical log block is generated. If not, perform long string pattern matching on the current sliding window log. If a match is found, the current vertical log shard is confirmed to be identified. The maximum sliding window is reduced, and it is determined whether it is greater than or equal to the minimum fragment length threshold. If so, the operation of sliding the target current fragmented compressed log block according to the maximum sliding window is returned. If not, then the vertical sharding extraction is completed, the remaining logs that have not been vertically sharded are obtained, and each vertical log shard is obtained based on each current vertical log shard and each current placeholder vertical log shard.

5. The method according to claim 4, characterized in that, The horizontal log sharding processing method includes the longest common subsequence processing method; The method of combining a pre-set horizontal log sharding processing method to perform horizontal sharding and extraction of the remaining logs that have not been vertically sharded, resulting in each horizontal log shard, includes: Get the pre-set minimum length allowable threshold and the minimum fragment occurrence frequency; The initial horizontal shard length is obtained by the longest common subsequence processing method, and the remaining logs that have not been vertically sharded are then sharded and extracted horizontally. Wherein, the initial horizontal segment length is greater than the minimum allowed length threshold; If extraction is successful and the frequency of occurrence of the extracted horizontal log fragment is not lower than the frequency of occurrence of the lowest fragment, then the current horizontal log fragment is obtained, and the initial horizontal fragment length is reduced. It is then determined whether it is greater than or equal to the minimum length allowed threshold. If so, the process returns to the operation of obtaining the initial horizontal fragment length through the longest common subsequence processing method and performing horizontal fragment extraction on the remaining logs that have not been vertically fragmented. After traversing the remaining logs that have not been vertically sharded, we obtain the horizontal log shards.

6. The method according to claim 5, characterized in that, The log sharded compressed storage database includes historical version encoding mapping rules; The process involves encoding each vertical log shard and each horizontal log shard using a pre-set adaptive encoding method, and storing the encoded data in a log shard compression storage database. This includes: Based on the historical version encoding mapping rules, each vertical log shard and each horizontal log shard are encoded using an adaptive encoding processing method to obtain each vertical encoded compressed log shard and each horizontal encoded compressed log shard. Each vertically encoded compressed log shard and each horizontally encoded compressed log shard includes an encoding version, shard fingerprint, shard type, shard encoding, and shard metadata. Each vertically encoded compressed log fragment and each horizontally encoded compressed log fragment are stored in the log fragment compressed storage database.

7. The method according to claim 6, characterized in that, Also includes: In the log sharded compressed storage database, the frequency of occurrence of each log shard is periodically counted; When the frequency and size of log fragments change, the encoding method of the log fragments is adjusted according to the pre-set compression efficiency calculation formula, and the historical version encoding mapping rules in the log fragment compression storage database are updated to save the new version encoding mapping rules and the new encoding version. This includes loading the new version's encoding mapping rules and the new encoding version into the cache and marking them as the latest active version; The new version encoding mapping rules and new encoding version corresponding to the latest active version are used to encode each vertical log fragment and each horizontal log fragment received at the next moment; for each vertical encoded compressed log fragment and each horizontal encoded compressed log fragment at the current moment, they are decoded according to the corresponding encoding version to generate the original log.

8. A log fragmented compressed storage device, characterized in that, include: The current task incremental log file acquisition module is used to acquire the current task incremental log file corresponding to the current pipeline task in real time; wherein, the current task incremental log file includes at least one current log to be fragmented and compressed; The current log block generation module is used to generate current standard logs to be compressed by replacing each current log with a pre-set regular expression matching unified placeholder replacement method, and to divide the logs into blocks in combination with a pre-set dynamic time window strategy, thereby generating at least one current log block to be compressed corresponding to each current standard log. The vertical log sharding and horizontal log sharding determination module is used to extract vertical shards from each current log block to be sharded and compressed sequentially using a sliding window algorithm to obtain each vertical log shard and the remaining log that has not been vertically sharded. The module also combines a pre-set horizontal log sharding processing method to extract horizontal shards from the remaining log that has not been vertically sharded to obtain each horizontal log shard. The storage module is used to encode each vertical log shard and each horizontal log shard using a pre-set adaptive encoding method, and store the encoded data in the log shard compression storage database. The log shard compression storage database uses different encoding methods based on the frequency of occurrence of different log shards. When the frequency of occurrence of log shards changes, the encoding method of the log shard compression storage database needs to be updated periodically and a new encoding version needs to be generated.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a log fragmentation and compression storage method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute a log fragmented compressed storage method as described in any one of claims 1-7.

Images