Method and system for efficiently retrieving discrete log information in mass data

Abstract

The invention discloses a method and system for efficiently retrieving discrete log information in mass data, and the method comprises the steps: writing a corresponding log file metadata record into a database in a data backup process; monitoring a newly added log file in the database at the current time point in a deamon process mode, and persisting metadata fragments of the log file to local storage in a sequential appending and writing mode; creating a check point snapshot before the current time point, and constructing a temporary snapshot table of the log file metadata record set by using the check point snapshot; writing latest log file metadata record information into an aggregation table by concurrently comparing the log file backup time points recorded in the log file metadata record set temporary snapshot table and the log file metadata fragments; and aggregating and writing the log file data into the storage data block to finish aggregation until the current time point. According to the method, the storage space can be effectively saved, the cost is reduced, and the retrieval performance of the log information is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data backup, and in particular to a method and system for efficiently retrieving discrete log information in massive data. Background Art

[0002] In the case of massive unstructured data, in order to ensure the validity and consistency of data in the event of abnormal downtime, the checkpoint log corresponding to the business will be recorded in combination with the corresponding time point during the data backup process. In actual application scenarios, the backup task cycle is usually a very long chain logical structure, and a large number of discrete checkpoint log files will be generated under this long chain. In addition, the size of these log files is affected by the specific data characteristics and is random, which is reflected in the decrease of random read performance of physical data. This greatly affects the efficiency of subsequent log information retrieval. Summary of the invention

[0003] In order to solve the above problems, the present invention proposes a method and system for efficiently retrieving discrete log information in massive data, which can effectively improve log retrieval performance.

[0004] In order to achieve the above object, the present invention is implemented by the following technical solutions: A method for efficiently retrieving discrete log information in massive data of the present invention comprises the following operations: During the data backup process, the corresponding log file metadata records are written into the database; As a deamon process, it monitors the newly added log files in the database at the current time point, and persists the log file metadata shards to local storage in a fixed format by sequentially appending writes; Create a checkpoint snapshot based on the current time point, and use the checkpoint snapshot to build a temporary snapshot table of log file metadata record set; By concurrently comparing the log file metadata record set temporary snapshot table with the log file metadata shard recorded in the log file backup time point, the latest log file metadata record information is written into the aggregation table; Aggregate and write log file data to storage data blocks to complete aggregation up to the current point in time.

[0005] A further improvement of the present invention is that a multi-channel logical architecture solution is used for read and write operations, each channel processes a sequential log file metadata slice, and each log file metadata slice corresponds to a directory subtree organization structure at the file system level, that is, the entire directory tree is divided into independent directory subtrees by dividing the breadth and depth. A further improvement of the present invention is that the log file metadata record mainly includes the log file name, the backup time point, and the data offset of the log file in the physical storage.

[0006] A further improvement of the present invention is that the log files are stored in the log file metadata record set temporary snapshot table in the order of backup time.

[0007] A further improvement of the present invention is that the aggregation and sharding operations of the log file data adopt a producer-consumer model to improve concurrency performance.

[0008] A system for efficiently retrieving discrete log information in massive data of the present invention comprises: The index module is used to save the metadata information of the log file and write the corresponding log file metadata records into the database during the data backup process; The monitoring module is used to monitor the newly added log files in the database at the current time point in the form of a deamon process, and persist the log file metadata shards to local storage in a fixed format in a sequential append-write manner; The snapshot module is used to create a checkpoint snapshot based on the current time point and use the checkpoint snapshot to build a temporary snapshot table of the log file metadata record set; The sharding module is used to write the latest log file metadata record information into the aggregate table log file metadata record set temporary snapshot table by concurrently comparing the log file metadata record set temporary snapshot table with the log file metadata shard recorded in the log file backup time point; The aggregation module is used to aggregate the log file data and write it into the storage data block to complete the aggregation up to the current time point.

[0009] A further improvement of the present invention is that a multi-channel logical architecture solution is used for read and write operations, each channel processes a sequential log file metadata slice, and each log file metadata slice corresponds to a directory subtree organization structure at the file system level, that is, the entire directory tree is divided into independent directory subtrees by dividing the breadth and depth.

[0010] A further improvement of the present invention is that the log file metadata mainly includes the log file name, the backup time point, and the data offset of the log file in the physical storage.

[0011] A further improvement of the present invention is that the log files are stored in the log file metadata record set temporary snapshot table in the order of backup time.

[0012] A further improvement of the present invention is that the aggregation and sharding operations of the log file data adopt a producer-consumer model to improve concurrency performance.

[0013] The beneficial effects of the present invention are as follows: the present invention converts discrete random IO (read and write) into local sequential IO by introducing the aggregation of log files, thereby significantly improving the retrieval performance of log information. At the same time, by introducing a sharding mechanism combined with application characteristics, the horizontal expansion capability of the system is better met. The efficient concurrency capability of the present invention effectively copes with the needs of future massive data scenarios, while saving storage space and reducing costs overall. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of a method in an embodiment of the present invention; Figure 2 It is an architectural diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0016] like Figure 1 and Figure 2 As shown, this embodiment provides a method for efficiently retrieving discrete log information in massive data. Through a customized log aggregation architecture, discrete log file records are efficiently aggregated, and discrete random IO (read and write) is converted into sequential IO (read and write), thereby effectively improving log retrieval performance. Specifically, the following operations are included: Step 1: During the data backup process, the corresponding log file metadata records are written into the database.

[0017] Step 2, introduce the sharding mechanism: monitor the newly added log files in the database at the current time point in the form of a deamon process, and persist the log file metadata shards to local storage in a fixed format and sequentially append writes. The specific fixed format includes: In order to improve the concurrent performance, a multi-channel logical architecture solution is adopted for read and write operations. Each channel processes a sequential log file metadata shard. Each log file metadata shard corresponds to the file system level directory subtree organization structure. The entire directory tree is divided into independent directory subtrees by dividing the breadth and depth, so as to achieve balanced distribution to the log file metadata shards and improve the overall concurrent capability.

[0018] Step 3: Create a checkpoint snapshot based on the current time point, and use the checkpoint snapshot to build a temporary snapshot table of the log file metadata record set. Building a temporary snapshot table of the log file metadata record set can reduce the concurrent lock contention of the log file metadata record set table, and is also conducive to the decoupling of application business logic and providing data consistency guarantee for the aggregation process.

[0019] Step 4, by concurrently comparing the log file backup time points of the log file metadata records in the temporary snapshot table of the log file metadata record set and the log file metadata records in the log file metadata shards, the latest log file metadata record information is written into the aggregate table. Since the log file records in the temporary snapshot table of the log file metadata record set are stored according to the backup time points, the historical versions of each time point are retained, and the log file metadata records finally written into the aggregate table correspond to the latest version of each log file. By making full use of the concurrent support capabilities brought by the directory subtree sharding, the original whole tree granularity is divided into finer-grained subtrees, and the synchronization lock contention for the whole tree can be reduced through local concurrent comparison, thereby further improving the overall efficiency of aggregate comparison.

[0020] Step 5: write the latest log file metadata record information into the aggregation table, and write the log file data aggregation into the storage data block to complete the aggregation up to the current time point.

[0021] The above method is implemented by a system for efficiently retrieving discrete log information in massive data in this embodiment, and the system includes: The index module is used to save the metadata information of the log file and write the corresponding log file metadata records into the database during the data backup process; The monitoring module is used to monitor the newly added log files in the database at the current time point in the form of a deamon process, and persist the log file metadata shards to local storage in a fixed format in a sequential append-write manner; The snapshot module is used to create a checkpoint snapshot based on the current time point, and use the checkpoint snapshot to build a temporary snapshot table of the log file metadata record set; the sharding module, each log file metadata shard corresponds to the directory organization structure at the file system level, and the directory tree is divided into independent directory subtrees by dividing the breadth and depth of the directory tree, so as to achieve a balanced distribution of the temporary snapshot table of the log file metadata record set in the log file metadata shard; The aggregation module is used to aggregate the log file data and write it into the storage data block to complete the aggregation up to the current time point.

[0022] The sharding module divides the massive change data set obtained by the monitoring module into width and depth locally according to the directory organization structure at the corresponding file system level, thereby dividing the entire change directory tree into independent directory subtrees and assigning them to corresponding independent channels. In this way, efficient horizontal expansion capabilities can be achieved in massive data scenarios. Figure 2 As shown, if the monitoring module detects that the data of file 5 has changed, the slicing module can be notified to perform slicing scheduling.

[0023] The function of the aggregation module is to write the unchanged log file metadata records in the snapshot table and the changed metadata records in the log file metadata shards into the aggregation table; at the same time, the log file data corresponding to the metadata records are aggregated into the storage data block. Figure 2 As shown, first, an aggregation table is created, and the log file metadata records corresponding to the time points 1 and 2 (T1 and T2) that have not changed in the snapshot table are written into the aggregation table; secondly, the metadata records of the files corresponding to the time point 3 (T3) that have changed in the log file metadata shard are written into the aggregation table; while the metadata records are written into the aggregation table, the log file data is written into the storage data block.

[0024] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have the same meaning as in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless defined similarly here.

[0025] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for efficiently retrieving discrete log information in massive data, characterized by: The following operations are included: During the data backup process, the corresponding log file metadata records are written into the database; As a deamon process, it monitors the newly added log files in the database at the current time point, and persists the log file metadata shards to local storage in a fixed format by sequentially appending writes; Create a checkpoint snapshot based on the current time point, and use the checkpoint snapshot to build a temporary snapshot table of log file metadata record set; By concurrently comparing the log file metadata record set temporary snapshot table with the log file metadata shard recorded in the log file backup time point, the latest log file metadata record information is written into the aggregation table; Aggregate and write log file data to storage data blocks to complete aggregation up to the current point in time.

2. The method for efficiently retrieving discrete log information in massive data according to claim 1, characterized in that: A multi-channel logical architecture solution is used for read and write operations. Each channel processes a sequential log file metadata shard. Each log file metadata shard corresponds to the directory subtree organization structure at the file system level, that is, the entire directory tree is divided into independent directory subtrees by dividing the breadth and depth.

3. The method for efficiently retrieving discrete log information in massive data according to claim 1, characterized in that: The log file metadata record mainly includes the log file name, backup time point, and data offset of the log file in physical storage.

4. The method for efficiently retrieving discrete log information in massive data according to claim 1, characterized in that: The log files are stored in the log file metadata record set temporary snapshot table in the order of backup time.

5. The method for efficiently retrieving discrete log information in massive data according to claim 1, characterized in that: The aggregation and sharding operations of the log file data adopt a producer-consumer model to improve concurrency performance.

6. A system for efficiently retrieving discrete log information in massive data, characterized by: The system comprises: The index module is used to save the metadata information of the log file and write the corresponding log file metadata records into the database during the data backup process; The monitoring module is used to monitor the newly added log files in the database at the current time point in the form of a deamon process, and persist the log file metadata shards to local storage in a fixed format in a sequential append-write manner; The snapshot module is used to create a checkpoint snapshot based on the current time point and use the checkpoint snapshot to build a temporary snapshot table of the log file metadata record set; The sharding module is used to write the latest log file metadata record information into the aggregate table log file metadata record set temporary snapshot table by concurrently comparing the log file metadata record set temporary snapshot table with the log file metadata shard recorded in the log file backup time point; The aggregation module is used to aggregate the log file data and write it into the storage data block to complete the aggregation up to the current time point.

7. The system for efficiently retrieving discrete log information in massive data according to claim 6, characterized in that: A multi-channel logical architecture solution is used for read and write operations. Each channel processes a sequential log file metadata shard. Each log file metadata shard corresponds to the directory subtree organization structure at the file system level, that is, the entire directory tree is divided into independent directory subtrees by dividing the breadth and depth.

8. The system for efficiently retrieving discrete log information in massive data according to claim 6, characterized in that: The log file metadata mainly includes the log file name, backup time point, and data offset of the log file in physical storage.

9. The system for efficiently retrieving discrete log information in massive data according to claim 6, characterized in that: The log files are stored in the log file metadata record set temporary snapshot table in the order of backup time.

10. The system for efficiently retrieving discrete log information in massive data according to claim 6, characterized in that: The aggregation and sharding operations of the log file data adopt a producer-consumer model to improve concurrency performance.

Images