A method and system for efficiently retrieving discrete log information in massive data

Abstract

The present invention discloses a method and system for efficiently retrieving discrete log information from massive amounts of data. The method comprises: writing corresponding log file metadata records into a database during data backup; monitoring newly added log files in the database at the current time point using a deamon process, and persisting log file metadata fragments to local storage by sequentially appending writes; creating a checkpoint snapshot based on a snapshot before the current time point, and using the checkpoint snapshot to construct a temporary snapshot table of a log file metadata record set; concurrently comparing the temporary snapshot table of the log file metadata record set with the log file backup time points recorded in the log file metadata fragments, and writing the latest log file metadata record information into an aggregation table; and aggregating the log file data and writing it into a storage data block to complete the aggregation up to the current time point. The present invention can effectively save storage space, reduce costs, and improve the retrieval performance of log information.

Description

Technical Field

[0001] The present invention relates to the technical field of data backup, and in particular to a method and system for efficiently retrieving discrete log information in massive data. Background Art

[0002] In scenarios with massive amounts of unstructured data, to ensure data validity and consistency in the event of unexpected downtime, checkpoint logs corresponding to the business are recorded at corresponding time points during the data backup process. In real-world applications, the backup task cycle is typically a long chain of logical structures. This long chain generates a large number of physically discrete checkpoint log files. Furthermore, the size of these log files is random, influenced by the specific characteristics of the data. This results in a decrease in random read performance of the physical data, significantly impacting the efficiency of subsequent log information retrieval. Summary of the Invention

[0003] In order to solve the above problems, the present invention proposes a method and system for efficiently retrieving discrete log information in massive data, which can effectively improve log retrieval performance.

[0004] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0005] A method for efficiently retrieving discrete log information from massive data of the present invention includes the following operations:

[0006] During the data backup process, the corresponding log file metadata records are written into the database;

[0007] As a deamon process, it monitors new log files added to the database at the current time point and persists the log file metadata shards to local storage in a fixed format by sequentially appending writes.

[0008] Create a checkpoint snapshot based on the current time point, and use the checkpoint snapshot to build a temporary snapshot table of log file metadata record set;

[0009] By concurrently comparing the log file metadata record set temporary snapshot table with the log file backup time points recorded in the log file metadata shards, the latest log file metadata record information is written to the aggregation table;

[0010] Aggregate and write log file data to storage data blocks to complete aggregation up to the current point in time.

[0011] A further improvement of the present invention is that a multi-channel logical architecture solution is adopted for read and write operations, each channel processes a sequential log file metadata shard, and each log file metadata shard corresponds to the file system level directory subtree organization structure, that is, the entire directory tree is divided into independent directory subtrees by breadth and depth.

[0012] A further improvement of the present invention is that the log file metadata record mainly includes the log file name, the backup time point, and the data offset of the log file in the physical storage.

[0013] A further improvement of the present invention is that the log files are stored in the log file metadata record set temporary snapshot table in the order of backup time.

[0014] A further improvement of the present invention is that the aggregation and sharding operations of the log file data adopt a producer-consumer model to improve concurrency performance.

[0015] The present invention provides a system for efficiently searching discrete log information in massive data, the system comprising:

[0016] The index module is used to save the metadata information of the log file and write the corresponding log file metadata records into the database during the data backup process;

[0017] The monitoring module is used to monitor the newly added log files in the database at the current time point in the form of a deamon process, and persist the log file metadata fragments to local storage in a fixed format by sequentially appending writes;

[0018] The snapshot module is used to create a checkpoint snapshot based on the current time point and use the checkpoint snapshot to build a temporary snapshot table of the log file metadata record set;

[0019] The sharding module is used to write the latest log file metadata record information into the aggregate table log file metadata record set temporary snapshot table by concurrently comparing the log file backup time points recorded in the log file metadata shards;

[0020] The aggregation module is used to aggregate log file data and write it into storage data blocks to complete the aggregation up to the current time point.

[0021] A further improvement of the present invention is that a multi-channel logical architecture solution is adopted for read and write operations, each channel processes a sequential log file metadata shard, and each log file metadata shard corresponds to the file system level directory subtree organization structure, that is, the entire directory tree is divided into independent directory subtrees by breadth and depth.

[0022] A further improvement of the present invention is that the log file metadata mainly includes the log file name, the backup time point, and the data offset of the log file in the physical storage.

[0023] A further improvement of the present invention is that the log files are stored in the log file metadata record set temporary snapshot table in the order of backup time.

[0024] A further improvement of the present invention is that the aggregation and sharding operations of the log file data adopt a producer-consumer model to improve concurrency performance.

[0025] The present invention offers the following advantages: by integrating log files, it transforms discrete random I / O (read and write) operations into localized sequential I / O, significantly improving log information retrieval performance. Furthermore, by introducing a sharding mechanism tailored to application characteristics, it better meets the system's horizontal scalability. Its efficient concurrency effectively addresses the demands of future massive data scenarios, while also saving storage space and reducing costs overall. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of a method in an embodiment of the present invention;

[0027] Figure 2 It is an architectural diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0029] like Figure 1 and Figure 2 As shown, this embodiment provides a method for efficiently retrieving discrete log information from massive data. Through a customized log aggregation architecture, it efficiently aggregates discrete log file records and converts discrete random I / O (read and write) into sequential I / O (read and write), thereby effectively improving log retrieval performance. Specifically, it includes the following operations:

[0030] Step 1: During the data backup process, the corresponding log file metadata records are written into the database.

[0031] Step 2: Introduce a sharding mechanism: Using a deamon process, monitor new log files added to the database at the current time point and persist the log file metadata shards to local storage in a fixed format, appending them sequentially. The specific fixed format includes: To improve concurrent performance, a multi-channel logical architecture is used for read and write operations. Each channel processes a sequential log file metadata shard. Each log file metadata shard corresponds to a directory subtree organization structure at the file system level. The entire directory tree is divided into independent directory subtrees by breadth and depth, achieving balanced distribution among the log file metadata shards and improving overall concurrent capabilities.

[0032] Step 3: Create a checkpoint snapshot before the current time point and use it to construct a temporary snapshot table for the log file metadata record set. This reduces concurrent lock contention on the log file metadata record set table, decouples application business logic, and ensures data consistency during the aggregation process.

[0033] Step 4, by concurrently comparing the log file metadata record set temporary snapshot table with the log file metadata record in the log file metadata shard at the log file backup time point, the latest log file metadata record information is written into the aggregate table. Since the log file records in the log file metadata record set temporary snapshot table are stored according to the backup time point, the historical version of each time point is retained, and the log file metadata record finally written into the aggregate table corresponds to the latest version of each log file. By making full use of the concurrent support capabilities brought by the directory subtree sharding, the original whole tree granularity is divided into finer-grained subtrees, and the synchronization lock contention for the whole tree can be reduced through local concurrent comparison, thereby further improving the overall efficiency of the aggregate comparison.

[0034] Step 5: While writing the latest log file metadata record information into the aggregation table, the log file data is aggregated and written into the storage data block to complete the aggregation up to the current time point.

[0035] The above method is implemented by a system for efficiently retrieving discrete log information from massive data in this embodiment, which includes:

[0036] The index module is used to save the metadata information of the log file and write the corresponding log file metadata records into the database during the data backup process;

[0037] The monitoring module is used to monitor the newly added log files in the database at the current time point in the form of a deamon process, and persist the log file metadata fragments to local storage in a fixed format by sequentially appending writes;

[0038] The snapshot module is used to create a checkpoint snapshot based on the current time point and use the checkpoint snapshot to build a temporary snapshot table of the log file metadata record set. The sharding module is used to divide the directory tree into independent directory subtrees by dividing the directory tree into breadth and depth, so as to achieve a balanced distribution of the temporary snapshot table of the log file metadata record set in the log file metadata shard.

[0039] The aggregation module is used to aggregate log file data and write it into storage data blocks to complete the aggregation up to the current time point.

[0040] The sharding module organizes the massive change data sets obtained by the monitoring module according to the corresponding file system level directory structure, and locally divides the change directory tree into breadth and depth, thereby cutting the entire change directory tree into independent directory subtrees and assigning them to corresponding independent channels. This achieves efficient horizontal expansion capabilities in massive data scenarios. Figure 2 As shown, if the monitoring module detects that the data of file 5 has changed, it can notify the sharding module to perform sharding scheduling.

[0041] The role of the aggregation module is to write the unchanged log file metadata records in the snapshot table and the changed metadata records in the log file metadata shards into the aggregation table; at the same time, it aggregates the log file data corresponding to the metadata records into the storage data block. Figure 2 As shown, first, an aggregation table is created, and the log file metadata records corresponding to time points 1 and 2 (T1 and T2) that have not changed in the snapshot table are written into the aggregation table; secondly, the metadata records of the files corresponding to time point 3 (T3) that have changed in the log file metadata shard are written into the aggregation table; at the same time as the metadata records are written into the aggregation table, the log file data is written into the storage data block.

[0042] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined similarly as herein, will not be interpreted in an idealized or overly formal sense.

[0043] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for efficiently retrieving discrete log information from massive data, characterized by: The following operations are included: During the data backup process, the corresponding log file metadata records are written into the database; As a deamon process, it monitors new log files added to the database at the current time point and persists the log file metadata shards to local storage in a fixed format by sequentially appending writes. Create a checkpoint snapshot based on the current time point, and use the checkpoint snapshot to build a temporary snapshot table of log file metadata record set; By concurrently comparing the log file metadata record set temporary snapshot table with the log file backup time points recorded in the log file metadata shards, the latest log file metadata record information is written to the aggregation table; Aggregate and write log file data to storage data blocks to complete aggregation up to the current point in time.

2. The method for efficiently retrieving discrete log information from massive data according to claim 1, characterized in that: A multi-channel logical architecture solution is used for read and write operations. Each channel processes a sequential log file metadata shard. Each log file metadata shard corresponds to the directory subtree organization structure at the file system level, that is, the entire directory tree is divided into independent directory subtrees by breadth and depth.

3. The method for efficiently retrieving discrete log information from massive data according to claim 1, characterized in that: The log file metadata record includes the log file name, backup time point, and the data offset of the log file in the physical storage.

4. The method for efficiently retrieving discrete log information from massive data according to claim 1, characterized in that: The log files are stored in the log file metadata record set temporary snapshot table in the order of backup time.

5. The method for efficiently retrieving discrete log information from massive data according to claim 1, characterized in that: The aggregation and sharding operations of the log file data adopt a producer-consumer model to improve concurrency performance.

6. A system for efficiently retrieving discrete log information from massive data, characterized by: The system comprises: The index module is used to save the metadata information of the log file and write the corresponding log file metadata records into the database during the data backup process; The monitoring module is used to monitor the newly added log files in the database at the current time point in the form of a deamon process, and persist the log file metadata fragments to local storage in a fixed format by sequentially appending writes; The snapshot module is used to create a checkpoint snapshot based on the current time point and use the checkpoint snapshot to build a temporary snapshot table of the log file metadata record set; The sharding module is used to write the latest log file metadata record information into the aggregate table log file metadata record set temporary snapshot table by concurrently comparing the log file backup time points recorded in the log file metadata shards; The aggregation module is used to aggregate log file data and write it into storage data blocks to complete the aggregation up to the current time point.

7. The system for efficiently retrieving discrete log information from massive data according to claim 6, characterized in that: A multi-channel logical architecture solution is used for read and write operations. Each channel processes a sequential log file metadata shard. Each log file metadata shard corresponds to the directory subtree organization structure at the file system level, that is, the entire directory tree is divided into independent directory subtrees by breadth and depth.

8. The system for efficiently retrieving discrete log information from massive data according to claim 6, characterized in that: The log file metadata includes the log file name, backup time point, and the data offset of the log file in the physical storage.

9. The system for efficiently retrieving discrete log information from massive data according to claim 6, characterized in that: The log files are stored in the log file metadata record set temporary snapshot table in the order of backup time.

10. The system for efficiently retrieving discrete log information from massive data according to claim 6, characterized in that: The aggregation and sharding operations of the log file data adopt a producer-consumer model to improve concurrency performance.

Images