Log data storage method, log data query method and related devices
By segmenting, sorting, and compressing log data, and building a data index, the problems of low log data query efficiency and low compression ratio in existing technologies are solved, achieving efficient log data storage and fast querying.
Patent Information
- Application Number
- CN202511058145.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-11-11
AI Technical Summary
Existing log data storage methods cannot be queried quickly, have low query efficiency, and low compression ratio.
The log data to be stored is divided into at least two data segments, sorted according to a preset sorting rule, and then divided into multiple log data blocks. Each block is compressed and marked, and a data index is built to indicate the sorting list and storage location.
It improves the query efficiency and compression ratio of log data during storage, optimizes the structure of log data, and achieves fast querying and efficient storage.
Smart Images
Figure CN120929436A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of information technology operation and maintenance technology, and in particular to a log data storage method, a log data query method, and related devices. Background Technology
[0002] The rapid development of the internet has led to a dramatic increase in log data. In particular, some financial service platforms generate more than 20TB of log data every day in payment scenarios alone. This places extremely high demands on improving the efficiency of log data querying and on achieving high compression ratio storage of log data (to reduce storage space requirements).
[0003] Existing log data storage methods result in log data that cannot be queried quickly, leading to low log query efficiency. Furthermore, the compression ratio of log data during storage is low. Summary of the Invention
[0004] This invention provides a log data storage method, a log data query method, and related apparatus to solve the problems of low log query efficiency and low compression ratio of log data when storing log data using existing log data storage methods.
[0005] To solve the above-mentioned technical problems, the present invention is implemented as follows:
[0006] In a first aspect, embodiments of the present invention provide a log data storage method, including:
[0007] Each log data in the log data set to be stored is divided into at least two data segments, and all the log data in the log data set is sorted according to a preset sorting rule based on the data segments.
[0008] The sorted log data is divided into multiple log data blocks;
[0009] Each of the log data blocks is compressed separately, and the compressed log data blocks are stored in a pre-allocated storage space. Furthermore, a tag is created for each compressed log data block, and the tag is used to indicate the sorting list of the log data in each log data block and the storage location of the log data block with preset key fields. All the tags are combined to obtain a data index.
[0010] Optionally, each log data in the collection to be stored is divided into at least two data segments, including:
[0011] According to the segmentation rule of one-to-one correspondence between log fields and data segments, the log data is divided into at least two data segments;
[0012] The log fields include at least two of the following fields:
[0013] Log message application name, log time, log level, log thread number, log sequence number, and other log information.
[0014] Optionally, each log data block includes a preset first number of log data.
[0015] Optionally, the Zstd compression algorithm is used to compress each of the log data blocks separately.
[0016] Secondly, embodiments of the present invention provide a log data query method, including:
[0017] Determine the target data that the user needs to query;
[0018] According to the data index, query the storage space that stores the compressed log data blocks, and determine the log data block containing the target data as the target log data block; wherein, the data index is obtained by set labeling, and the labeling is used to indicate the sorted list of the log data in each log data block and the storage location of the log data block with preset key fields;
[0019] Obtain the target data from the target log data block.
[0020] Thirdly, embodiments of the present invention provide a log data storage device, comprising:
[0021] The splitting module is used to divide each log data in the log data set to be stored into at least two data segments, and sort all the log data in the log data set according to a preset sorting rule based on the data segments;
[0022] The splitting module is also used to divide the sorted log data into multiple log data blocks;
[0023] An execution module is used to compress each of the log data blocks separately, store the compressed log data blocks in a pre-allocated storage space, and establish a mark for the compressed log data blocks. The mark is used to indicate the sorting list of the log data in each log data block and the storage location of the log data block with preset key fields. All the marks are combined to obtain a data index.
[0024] Fourthly, embodiments of the present invention provide a log data query device, comprising:
[0025] The determination module is used to determine the target data that the user needs to query;
[0026] The first execution module is used to query the storage space containing compressed log data blocks according to the data index, and determine the log data block containing the target data as the target log data block; wherein, the data index is obtained by a set of tags, and the tags are used to indicate the sorted list of the log data in each log data block and the storage location of the log data block with preset key fields;
[0027] The first execution module is further configured to obtain the target data from the target log data block.
[0028] Fifthly, embodiments of the present invention provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps in the log data storage method as described in any one of the first aspects, or implement the steps in the log data query method as described in the second aspect.
[0029] In a sixth aspect, embodiments of the present invention provide a readable storage medium on which a program or instructions are stored. When the program or instructions are executed by a processor, they implement the steps in the log data storage method as described in any one of the first aspects, or implement the steps in the log data query method as described in the second aspect.
[0030] In a seventh aspect, embodiments of the present invention provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the log data storage method as described in any one of the first aspects, or implement the steps in the log data query method as described in the second aspect.
[0031] In this embodiment of the invention, each log data in the log data set to be stored is divided into at least two data segments, and all log data in the log data set is sorted according to a preset data segment-based sorting rule. The sorted log data is then divided into multiple log data blocks. Each log data block is compressed, and the compressed log data blocks are stored in a pre-allocated storage space. Furthermore, a marker is established for each compressed log data block, indicating the sorting list of log data in each block and the storage location of log data blocks with preset key fields. All markers are combined to obtain a data index. Log data stored according to this embodiment can be quickly queried, which improves log query efficiency. In addition, by sorting the log data and dividing the sorted log data into multiple log data blocks, the structure of the log data is optimized, and the compression ratio of the log data during storage is improved. Attached Figure Description
[0032] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0033] Figure 1 This is a flowchart illustrating the log data storage method according to an embodiment of the present invention;
[0034] Figure 2 A diagram illustrating data segmentation;
[0035] Figure 3 A diagram illustrating the sorting results of log data;
[0036] Figure 4 A diagram illustrating the results of log data segmentation;
[0037] Figure 5 This is a schematic diagram of the compression result of log data blocks;
[0038] Figure 6 This is a schematic diagram of a compressed storage format;
[0039] Figure 7 A schematic diagram illustrating the internal structure of the labeled data;
[0040] Figure 8 This is a diagram illustrating the proposed metadata file;
[0041] Figure 9 This is a schematic diagram illustrating the internal structure of a metadata file.
[0042] Figure 10 This is a flowchart illustrating the log data query method according to an embodiment of the present invention;
[0043] Figure 11 A diagram illustrating the location of a log data block;
[0044] Figure 12 This is a schematic block diagram of the log data storage device according to an embodiment of the present invention;
[0045] Figure 13 This is a schematic diagram of the log data query device according to an embodiment of the present invention;
[0046] Figure 14 This is a schematic block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0048] The terms "first," "second," etc., used in this embodiment of the invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, in this embodiment of the invention, "or" indicates at least one of the connected objects. For example, "A or B" covers three scenarios: Scenario 1: includes A but does not include B; Scenario 2: includes B but does not include A; Scenario 3: includes both A and B. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0049] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0050] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of personal information involved in the technical solutions of this invention comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to personal information data and to maintain personal information security and network security.
[0051] This invention provides a log data storage method, see [link to relevant documentation]. Figure 1 As shown, Figure 1 This is a flowchart illustrating the log data storage method according to an embodiment of the present invention. The log data storage method includes:
[0052] Step 11: Divide each log data in the log data set to be stored into at least two data segments, and sort all the log data in the log data set according to the preset data segment-based sorting rules;
[0053] Step 12: Divide the sorted log data into multiple log data blocks;
[0054] Step 13: Compress each log data block separately, store the compressed log data blocks in the pre-allocated storage space, and create a tag for the compressed log data blocks. The tag is used to indicate the sorting list of log data in each log data block and the storage location of log data blocks with preset key fields. Collect all tags to obtain the data index.
[0055] In this embodiment of the invention, data segmentation is achieved by dividing each log data in the set of log data to be stored into at least two data segments in step 11. Data segmentation refers to the process of decomposing a single log record into multiple parts. Data segmentation allows for better data organization and enables queries to read only the relevant parts, thereby improving query efficiency.
[0056] In some embodiments, segmentation can be based on specific log formats, business types, or other logic. For example, segmenting access request logs of a payment application can be based on fields such as log application name, log time, log level, log thread number, and log serial number. The segmented message is shown in [reference]. Figure 2 As shown, each log entry is divided into six data segments: segments A, segments B, segments C, segments D, segments E, and segments F. Segments A is the application name of the log message, segments B is the log time, segments C is the log level, segments D is the log thread number, segments E is the log sequence number, and segments F is other log information.
[0057] In some embodiments, the preset sorting rule based on data segments can be based on one or more fields, which serve as filtering or sorting conditions in the query. After sorting, similar data can be stored contiguously, and unnecessary data blocks can be skipped during the query, reducing disk seek time and thus improving query speed. For example, see... Figure 3 As shown, Figure 3 The system displays the sorted log data, allowing users to select from the segmented data and sort by Segments A (log message application name), Segments E (log serial number), and Segments B (log time).
[0058] In some embodiments, step 12 may specifically involve presetting a first quantity, dividing the log data into blocks according to the first quantity, so that each log data block has the first quantity of log data. For example, see... Figure 4As shown, the log data is divided into blocks of 8912 entries. If the received log data does not exceed 8912, it is accumulated until 8912 entries are generated into the next block. In this example, 8912 is the preset first number.
[0059] Regarding the design rationale for setting the first quantity to 8912 in the above example: 1. 8912 is 2 to the power of 13, allowing for pre-allocation of memory based on a size of 8192, reducing memory fragmentation, and handling I / O operations more efficiently. 2. 8912 can divide large-scale log data into relatively uniform blocks. 3. Accumulating received log data up to 8912 records helps reduce unnecessary block splitting operations and improves data processing efficiency. Once 8912 records are reached, a new block is generated, giving the data organization a certain regularity. 4. The last data block may contain fewer than 8912 records. In this case, the data block can be kept to a smaller size.
[0060] In some embodiments, log data is compressed and stored by selecting the ZSTD algorithm to compress the segmented data in each data block into a [segments].[mark].zst compressed block. For example, for the above... Figure 4 The image shows a compressed block of the access request log for a payment application, with Segments A (log message application name) and Segments E (log sequence number) as examples. (See also...) Figure 5 As shown, Figure 5 The compression results of the Segments A (log message application name) and Segments E (log serial number) fields are displayed after compression.
[0061] In some embodiments, to achieve a balance between query performance and compression ratio, and to reduce the number of small files, the step of compressing each log data block separately may include compressing 128 log data blocks together into a single file. It should be noted that for ease of subsequent querying, see [link to documentation]. Figure 6 As shown, this invention designs a compressed storage format, and the compressed file is [Segments].log, which contains 128 compressed log data blocks.
[0062] It's important to note that tagged data and metadata are crucial for quickly locating data. Tagged data includes metadata for each data block or segment (e.g., starting offset, size, number of records contained), while metadata contains a sorted list of key fields and their corresponding data block locations. This allows queries to quickly locate relevant data blocks directly through the index without scanning the entire dataset.
[0063] In practical applications, tagged data is created for the actual compressed and stored data. See also... Figure 7 As shown, a marker data file [segments].mrk2 records the marker data for 128 compressed blocks. Each line of marker data records the offset information of one compressed data block, meaning that each [segments].[mark].zst compressed block file has a corresponding line of marker data.
[0064] Furthermore, create labeled data blocks, for example, the above. Figure 4 The compressed block of the access request log of the payment application shown is included. Figure 8 As shown, Figure 8 This demonstrates the creation of multiple sorting segments: Segments A (log message application name), Segments E (log sequence number), and Segments B (log time), with a suggested metadata idx file. See also Figure 9 As shown, Figure 9 This shows the result of creating metadata idx files for Segments A (log message application name), Segments E (log serial number), and Segments B (log time), i.e., the final idx file.
[0065] In some embodiments, after step 13, the process includes: establishing a query model for compressed log data blocks stored in pre-allocated storage space. For example, for the above... Figure 4 The compressed access request log of the payment application shown here requires querying data that includes a time-consuming field. Since the time-consuming field is located in other log segments, a query model needs to be built for these other log segments. The query model can model data in common formats such as JSON, CSV, XML, and KV. It can also model plain text using fuzzy matching.
[0066] As shown in Table 1, configure the model for access request logs of the payment application, including time consumption, return code, log action type, and request path parsing model.
[0067] Table 1 shows the results of establishing the query model.
[0068]
[0069] Once the model is configured, subsequent queries only require the model name to apply complex query syntax and aggregation analysis to specific fields.
[0070] It should be noted that this invention sorts log data according to a preset data segment-based sorting rule, dividing the sorted log data into multiple log data blocks. The data segments have high similarity, optimizing the log data structure and improving the compression ratio. Furthermore, compared to full-text compression of all log data, compressing each log data block separately achieves a higher compression ratio.
[0071] In this embodiment of the invention, each log data in the log data set to be stored is divided into at least two data segments, and all log data in the log data set is sorted according to a preset data segment-based sorting rule. The sorted log data is then divided into multiple log data blocks. Each log data block is compressed, and the compressed log data blocks are stored in a pre-allocated storage space. Furthermore, a marker is established for each compressed log data block, indicating the sorting list of log data in each block and the storage location of log data blocks with preset key fields. All markers are combined to obtain a data index. Log data stored according to this embodiment can be quickly queried, which improves log query efficiency. In addition, by sorting the log data and dividing the sorted log data into multiple log data blocks, the structure of the log data is optimized, and the compression ratio of the log data during storage is improved.
[0072] In some embodiments, each log data in the set of log data to be stored is divided into at least two data segments, including:
[0073] According to the one-to-one segmentation rule of log fields and data segments, the log data is divided into at least two data segments;
[0074] The log fields include at least two of the following fields:
[0075] Log message application name, log time, log level, log thread number, log sequence number, and other log information.
[0076] In this embodiment of the invention, each log data in the set of log data to be stored is divided into at least two data segments. This includes dividing the log data into at least two data segments according to a one-to-one segmentation rule between log fields and data segments. The log fields include at least two of the following: log message application name, log time, log level, log thread number, log sequence number, and other log information. This invention enables the decomposition of a single log record into multiple parts. Data segmentation allows for better data organization and enables queries to read only the relevant parts, thereby improving query efficiency.
[0077] In some embodiments, each log data block includes a preset first number of log data.
[0078] In some embodiments, the first quantity may be 8912. See also Figure 4 As shown, log data is divided into blocks of 8912 records. If the received log data does not exceed 8912, it is accumulated until 8912 records are generated before the next block is created. In this example, 8912 is the preset first number. The design rationale for setting the first number to 8912 in the above example is as follows: 1. 8912 is 2 to the power of 13, allowing for pre-allocation of memory based on a size of 8192, reducing memory fragmentation and handling I / O operations more efficiently. 2. 8912 can divide large-scale log data into relatively uniform blocks. 3. Accumulating log data when it does not exceed 8912 records helps reduce unnecessary block division operations and improves data processing efficiency. Once 8912 records are reached, a new block is generated, giving the data organization a certain regularity. 4. The last data block may contain fewer than 8912 records. In this case, the data block can be kept to a smaller size.
[0079] In some embodiments, the Zstd compression algorithm is used to compress each log data block separately.
[0080] Zstandard (usually abbreviated as Zstd) is a compression algorithm developed by Facebook, designed to provide high compression ratios and speeds. Zstd is designed to balance compression performance and speed, making it perform well in many application scenarios.
[0081] This invention provides a log data query method, see [link to relevant documentation]. Figure 10 As shown, Figure 10 This is a flowchart illustrating the log data query method according to an embodiment of the present invention. The log data query method includes:
[0082] Step 21: Determine the target data that the user needs to query;
[0083] Step 22: According to the data index, query the storage space that stores the compressed log data blocks, and determine the log data block with the target data as the target log data block; wherein, the data index is obtained by set labeling, and the label is used to indicate the sorted list of log data in each log data block and the storage location of the log data block with preset key fields;
[0084] Step 23: Obtain the target data from the target log data block.
[0085] In some embodiments, determining the target data that the user needs to query may specifically include: parsing the query request input by the user to determine the range of data to be retrieved, conditions, etc., and performing conditional word segmentation parsing in the actual scenario.
[0086] In this embodiment of the invention, log data is stored according to the following method: each log data in the log data set to be stored is divided into at least two data segments; all log data in the log data set is sorted according to a preset sorting rule based on the data segments; the sorted log data is divided into multiple log data blocks; each log data block is compressed; the compressed log data blocks are stored in a pre-allocated storage space; and a mark is established for each compressed log data block, the mark indicating the sorting list of the log data in each log data block and the storage location of the log data block with preset key fields. All the marks are combined to obtain a data index. This storage method uses segmented storage. During data query, data of the same segment are stored in one log data block. Data segments that do not participate in the query calculation do not participate in the query calculation, reducing I / O, reducing the data reading pressure during the query, and accelerating the query.
[0087] To illustrate with specific examples:
[0088] The user's query request is parsed, and the result indicates a need to query log data with application name K (i.e., target data), meaning a query for the SegmentsA section is required. Then, based on the data index and the indication of the storage location of log data blocks with preset key fields, the storage location of the target log data block containing log data with application name K (i.e., target data) can be determined. Further, based on the indication of the sorted list of log data in each log data block, the log data with application name K can be identified from the target log data block. Finally, the log data with application name K (target data) can be retrieved.
[0089] In some embodiments with established query models, as shown in Table 1, if a time-consuming field needs to be queried, the response_duration query model is called; if a return code field needs to be queried, the log_msgId query model is called.
[0090] For example, query request to locate block data: query the average time taken over 20 points for the application named cmosp-cgw-out.
[0091] Based on the query parsing, it was determined that the query needed to retrieve SegmentsA='cmosp-cgw-out', the SegmentsB time field, and the response_duration time model. Based on the parsed conditions, the log data block can be quickly located. See [link / reference]. Figure 11As shown, the yellow markers indicate the located log data blocks. Within 0.idx, the SegmentsA section, after sorting, consists of activities, cmosp-cgw-out, and fc-capital. To query the cmosp-cgw-out application, at offset 40961 in the 0.idx metadata, the starting line is activities_agw110292105587123578997_2021-10-29 20:30:47,219, but the ending line might start with either activities_ or cmosp-cgw-out, so this database needs to be located. Similarly, at offset 0 in the 1.idx metadata, the starting line is cmosp-cgw-out_CTSNPUB1REGREG3124081421261176343_2021-10-29 21:26:11,687, so this database also needs to be located. At offset 8182 in the 1.idx metadata, the starting behavior is fc-capital_eg-enterprise408142130300000369217_2021-10-29 21:30:30,457. Therefore, this database does not need to be located because cmosp-cgw-out ends at most in the 0th data block of the 1.idx metadata.
[0092] The database 0.idx metadata 40961 data block and 1.idx metadata 0 offset data block were located. The specific data was queried. At the same time, according to the filtering conditions, 8 data points of Segments B of these two blocks were queried. Finally, it was found that 245 rows of 0.idx metadata 40961 data block met the conditions and 40 rows of 1.idx metadata 0 offset data block met the conditions.
[0093] Based on the offset of the row found in the block, we can quickly locate the data that meets the conditions in the SegmentsF segment. We only need to query 285 rows of data to reduce the amount of data to be queried.
[0094] Query models can be built for data segments, enabling various query methods such as structured search and aggregation analysis for segmented data. Log formats are highly complex. Even when the exact number of formats and schemas of the incoming logs is unknown, the segmented query model allows for easy access to the logs. Once the logs are connected, a new field mapping model can be defined to respond to various data format query requirements.
[0095] In this embodiment of the invention, the target data that the user needs to query is determined; according to the data index, the storage space storing the fully compressed log data blocks is queried, and the log data block containing the target data is determined as the target log data block; wherein, the data index is obtained by a set of tags, and the tags are used to indicate the sorted list of log data in each log data block and the storage location of the log data block with preset key fields; the target data is obtained from the target log data block. This invention can realize fast querying of log data and improve query efficiency.
[0096] This invention provides a log data storage device, see [link to documentation]. Figure 12 As shown, Figure 12 This is a schematic block diagram of a log data storage device according to an embodiment of the present invention. The log data storage device 120 includes:
[0097] The splitting module 121 is used to divide each log data in the log data set to be stored into at least two data segments, and sort all the log data in the log data set according to a preset sorting rule based on the data segments;
[0098] The splitting module 121 is also used to divide the sorted log data into multiple log data blocks;
[0099] The execution module 122 is used to compress each of the log data blocks respectively, store the compressed log data blocks in a pre-allocated storage space, and establish a mark for the compressed log data blocks. The mark is used to indicate the sorting list of the log data in each log data block and the storage location of the log data block with preset key fields. All the marks are combined to obtain a data index.
[0100] In some embodiments, the splitting module 121 is further configured to divide the log data into at least two data segments according to a one-to-one segmentation rule between log fields and data segments;
[0101] The log fields include at least two of the following fields:
[0102] Log message application name, log time, log level, log thread number, log sequence number, and other log information.
[0103] In some embodiments, each log data block includes a preset first number of log data.
[0104] In some embodiments, the Zstd compression algorithm is used to compress each of the log data blocks separately.
[0105] The log data storage device provided in this embodiment of the invention can implement the various processes implemented in the log data storage method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0106] This invention provides a log data query device, see [link to documentation]. Figure 13 As shown, Figure 13 This is a schematic diagram of the log data query device according to an embodiment of the present invention. The log data query device 130 includes:
[0107] Module 131 is used to determine the target data that the user needs to query.
[0108] The first execution module 132 is used to query the storage space containing compressed log data blocks according to the data index, and determine the log data block containing the target data as the target log data block; wherein, the data index is obtained by a set of tags, and the tags are used to indicate the sorted list of the log data in each log data block and the storage location of the log data block with preset key fields;
[0109] The first execution module 132 is further configured to obtain the target data from the target log data block.
[0110] The log data query device provided in this embodiment of the invention can implement all the processes implemented in the log data query method embodiment and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0111] This invention provides an electronic device 140, see [link to relevant documentation]. Figure 14 As shown, Figure 14 This is a schematic block diagram of an electronic device 140 according to an embodiment of the present invention, including a processor 141, a memory 142, and a program or instructions stored in the memory 142 and executable on the processor 141. When the program or instructions are executed by the processor, they implement the steps in any log data storage method of the present invention, or implement the steps in any log data query method of the present invention.
[0112] This invention provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements the various processes of an embodiment of the log data storage method as described above, or implements the various processes of an embodiment of the log data query method as described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0113] The readable storage medium may include, for example, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.
[0114] This invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement various processes of an embodiment of the log data storage method as described above, or implement various processes of an embodiment of the log data query method as described above, and can achieve the same technical effect. To avoid repetition, these will not be described again here.
[0115] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0117] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.
Claims
1. A log data storage method, characterized in that, include: Each log data in the log data set to be stored is divided into at least two data segments, and all the log data in the log data set is sorted according to a preset sorting rule based on the data segments. The sorted log data is divided into multiple log data blocks; Each of the log data blocks is compressed separately, and the compressed log data blocks are stored in a pre-allocated storage space. Furthermore, a tag is created for each compressed log data block, and the tag is used to indicate the sorting list of the log data in each log data block and the storage location of the log data block with preset key fields. All the tags are combined to obtain a data index.
2. The data storage method according to claim 1, characterized in that, Each log data in the collection to be stored is divided into at least two data segments, including: According to the segmentation rule of one-to-one correspondence between log fields and data segments, the log data is divided into at least two data segments; The log fields include at least two of the following fields: Log message application name, log time, log level, log thread number, log sequence number, and other log information.
3. The data storage method according to claim 1, characterized in that, Each log data block includes a preset first number of log data.
4. The data storage method according to claim 1, characterized in that, Each of the log data blocks was compressed using the Zstd compression algorithm.
5. A log data query method, characterized in that, include: Determine the target data that the user needs to query; According to the data index, query the storage space that stores the compressed log data blocks, and determine the log data block containing the target data as the target log data block; wherein, the data index is obtained by set labeling, and the labeling is used to indicate the sorted list of the log data in each log data block and the storage location of the log data block with preset key fields; Obtain the target data from the target log data block.
6. A log data storage device, characterized in that, include: The splitting module is used to divide each log data in the log data set to be stored into at least two data segments, and sort all the log data in the log data set according to a preset sorting rule based on the data segments; The splitting module is also used to divide the sorted log data into multiple log data blocks; An execution module is used to compress each of the log data blocks separately, store the compressed log data blocks in a pre-allocated storage space, and establish a mark for the compressed log data blocks. The mark is used to indicate the sorting list of the log data in each log data block and the storage location of the log data block with preset key fields. All the marks are combined to obtain a data index.
7. A log data query device, characterized in that, include: The determination module is used to determine the target data that the user needs to query; The first execution module is used to query the storage space containing compressed log data blocks according to the data index, and determine the log data block containing the target data as the target log data block; wherein, the data index is obtained by a set of tags, and the tags are used to indicate the sorted list of the log data in each log data block and the storage location of the log data block with preset key fields; The first execution module is further configured to obtain the target data from the target log data block.
8. An electronic device, characterized in that: It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps in the log data storage method as described in any one of claims 1 to 4, or implement the steps in the log data query method as described in claim 5.
9. A readable storage medium, characterized in that: The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps in the log data storage method as described in any one of claims 1 to 4, or the steps in the log data query method as described in claim 5.
10. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the steps in the log data storage method as described in any one of claims 1 to 4, or implement the steps in the log data query method as described in claim 5.