A method and system for high-performance automatic uniform splitting of mass unstructured data

Abstract

The application discloses a kind of high-performance automatic uniform split mass unstructured data method and system, including split process and multiple backup processes, the task splitter of split process monitors the execution of all split threads in split thread pool, and realizes the automatic uniform split of mass unstructured data by stealing algorithm and weighted round robin load balancing algorithm.The present application makes full use of the concurrent processing capability of modern computer multi-core CPU, and greatly improves the traversal performance of mass unstructured data through automatic uniform split algorithm, further improves the backup and recovery performance of mass unstructured data as a whole.

Description

Technical Field

[0001] This invention relates to the field of data backup technology, specifically a high-performance method and system for automatically and uniformly splitting massive amounts of unstructured data. Background Technology

[0002] As various industries enter the era of information explosion, various big data platforms have emerged, bringing with them massive amounts of data, especially massive amounts of unstructured data. This type of massive unstructured data often uses distributed storage architectures such as NAS, S3, and HDFS as the underlying storage. How to quickly back up and restore such a huge amount of unstructured data, especially massive amounts of small unstructured files, has become a global challenge in the field of data backup technology.

[0003] Currently, when backing up massive amounts of unstructured data, the industry mainly adopts multi-node or multi-channel backup. The core idea is to utilize the concurrent processing capabilities of modern computer multi-core CPUs to perform multi-threaded or multi-process operations, thereby improving the overall performance of backup and recovery. Regardless of whether it is multi-threaded or multi-process backup, it involves how the backup source is split and how the workload of each thread or process is evenly distributed. The degree of balance in the distribution of backup source and workload determines whether the final performance of multi-channel backup can achieve optimal results.

[0004] Regarding backup source splitting schemes, there are currently two main approaches in the industry. Scheme 1 involves using data source analysis tools to scan the backup sources before performing multi-channel backups of massive amounts of unstructured data. Based on the scanned backup source directory topology, a specific splitting strategy is formulated. While this scheme achieves good splitting balance, the introduction of data source analysis tools increases the complexity of using backup products and requires significant additional time to wait for scan results. Scheme 2 involves setting a splitting batch threshold based on empirical values, such as sending data to backup channels for processing when splitting to 100,000 directories and files. While this scheme does not require prior knowledge of the backup source directory topology, it can lead to uneven distribution of workload across multiple channels. Due to differences in the total size of each batch and the processing capacity of each channel, it may result in situations where, for example, three backup channels are initially enabled, but only one channel is actually backing up, thus failing to achieve optimal performance for multi-channel backups. Summary of the Invention

[0005] To address the shortcomings of the two backup source splitting methods mentioned above, this invention proposes a high-performance method for automatically and uniformly splitting massive amounts of unstructured data.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: This invention is a high-performance method for automatically and uniformly splitting massive amounts of unstructured data, comprising the following operations: Step 1: Start the splitting process and backup process. The task splitter traverses the set of backup sources and assigns each backup source to an independent splitting thread for individual processing. Step 2: The learning model performs statistical analysis on the historical backup set information, calculates the number of directories and metadata load distribution threshold for each theft request in this splitting task through the feature directory, and sends it to the task splitter. Step 3: Each splitting thread's traversal executor performs a breadth-first traversal of the current traversal directory, adds the traversed subdirectories to the splitting task queue exclusively for the corresponding splitting thread, and adds the subdirectory metadata and subfile metadata to the metadata cache queue exclusively for the corresponding splitting thread. Step 4: The eavesdropper of each splitting thread monitors whether the queue of tasks to be split in this splitting thread is empty. If it is, a stealing request is triggered. If not, return to step 3. Step 5: Each splitting thread's eavesdropper sends a stealing request to all splitting threads except itself, and adds the stolen directories to be split to its own splitting thread's task queue. Step 6: Each split thread monitors whether the length of the metadata cache queue has reached the load distribution threshold. If so, a load distribution request is triggered; otherwise, return to step 3. Step 7: After receiving the load distribution request, the load balancer distributes the metadata set of the current batch to the backup process using a weighted round-robin load balancing algorithm. Step 8: After receiving the metadata set, the backup process adds it to the metadata queue from the tail of the backup process's metadata queue. Step 9: The read thread of the backup core module retrieves metadata from the head of the metadata queue, reads the file data of the metadata, and adds it to the data cache queue; Step 10: The write thread of the backup core module reads metadata and file data from the data cache queue and writes them to the storage medium; Step 11: The task splitter monitors whether all splitting threads in the splitting thread pool have finished splitting. If so, it notifies the backup process to start the stealing mechanism. If not, it returns to step 7. Step 12: After the backup process starts the theft mechanism, the eavesdropper of each backup process monitors whether the metadata queue of this process is empty. If it is, the theft request is triggered; otherwise, the process returns to step 9. Step 13: Each backup process's eavesdropper sends a stealing request to other backup processes besides itself, and adds the stolen metadata to the metadata queue of this process. Step 14: Once the backup core module of each backup process finishes running and the theft request processing of each backup process is completed, the current splitting and backup task ends.

[0007] A further improvement of the present invention is that: the task queue to be split and the metadata cache queue are both exclusive blocking queues for split threads, wherein the task queue to be split adopts a double-ended blocking queue and the metadata cache queue adopts a single-ended blocking queue.

[0008] A further improvement of the present invention is that: Step 3 specifically operates as follows: The traversal executor of the split thread obtains the directory to be split from the head of the task queue to be split, performs a breadth-first traversal, adds the traversed subdirectories to the task queue to be split from the tail of the task queue to be split, and adds the metadata of the subdirectories and subfiles to the metadata cache queue from the tail of the metadata cache queue.

[0009] A further improvement of the present invention is that, in step 5, the subdirectories traversed by the traversal executor in the stealing state are preferentially added to the task queue to be split, and the stolen directories to be split are added to the task queue to be split from the tail of the task queue to be split.

[0010] The present invention discloses a high-performance automatic uniform splitting system for massive unstructured data, the system comprising: a network interface, a memory, and a processor; The network interface is used for receiving and sending signals during the process of sending and receiving information with other external network elements; The memory is used to store computer programs that can run on the processor; The processor is configured to execute the steps of the above-described method for high-performance automatic uniform splitting of massive amounts of unstructured data when running the computer program.

[0011] The present invention provides a computer storage medium storing a program for automatically and uniformly splitting massive amounts of unstructured data in a high-performance manner. When the program for automatically and uniformly splitting massive amounts of unstructured data in a high-performance manner is executed by at least one processor, it implements the steps of the method for automatically and uniformly splitting massive amounts of unstructured data in a high-performance manner.

[0012] The beneficial effects of this invention are: (1) By splitting the multi-threaded concurrent jobs of the thread pool, the overall splitting performance of massive unstructured data is improved. (2) By using the thread-exclusive task queue and metadata cache queue, multi-threaded contention and waiting are avoided. By using the task queue to be split, the concurrent operation performance of the stealing thread and the traversal thread on the same queue is improved. (3) By continuously optimizing the stealing algorithm and the weighted round-robin load balancing algorithm through the learning module, the high-performance automatic uniform splitting of massive unstructured data is achieved. Attached Figure Description

[0013] Figure 1 This is an architecture diagram of an embodiment of the present invention; Figure 2 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0015] like Figure 1 As shown in the diagram, the core modules and components specifically include: Splitting process: An independent running process used to split massive amounts of unstructured data; The backup process is an independent running process used to back up massive amounts of unstructured data. In this embodiment, multiple backup processes are used concurrently. The task splitter is a core component of the splitting process, responsible for coordinating and scheduling the overall splitting work. Internally, it provides a splitting thread pool. The splitting thread is a core component of the task splitter, responsible for the specific splitting work. The split-thread eavesdropper is a core component of the split-thread mechanism, responsible for stealing tasks between split threads. The load balancer is a core component of the splitting process and is responsible for implementing the weighted round-robin load balancing algorithm. The task queue to be split is a core component of the splitting thread and is used to store directories to be split. In this embodiment, a double-ended blocking queue exclusively used by the splitting thread is adopted. The metadata cache queue is a core component of the split thread and is used to store the metadata set of directories and files to be backed up. In this embodiment, a single-ended blocking queue exclusively for the split thread is used. The backup core module is a core component of the backup process and is responsible for specific data backup tasks. The metadata queue is a core component of the backup process, used to store metadata information to be backed up. The backup process eavesdropper is a core component of the backup process, responsible for intercepting data between backup processes.

[0016] The above is about Figure 1 The architecture diagram of the embodiment introduces the functions of each core module and component. The following section will combine... Figure 2 The flowchart of the embodiments provides a further detailed description of the preferred embodiments of the present invention.

[0017] This embodiment provides a high-performance method for automatically and uniformly splitting massive amounts of unstructured data, the specific steps of which include: Step 1: Start a split process and multiple backup processes; Step 2: The task splitter in the splitting process traverses the set of backup sources and assigns one backup source to a splitting thread for individual processing; Step 3: The learning model performs statistical analysis on the historical backup set information, calculates the number of directories and metadata load distribution threshold for each theft request in this splitting task through the feature directory, and sends it to the task splitter. Step 4: The traversal executor of the split thread retrieves the directory to be split from the head of the task queue, performs a breadth-first traversal, adds the traversed subdirectories from the tail of the task queue, and adds the subdirectory metadata and subfile metadata to the metadata cache queue exclusive to the corresponding split thread. Step 5: The eavesdropper of the split thread monitors whether the queue of tasks to be split in this split thread is empty. If it is, a stealing request is triggered; otherwise, return to step 4. Step 6: The eavesdropper of the split thread sends a stealing request to other split threads except itself, and adds the stolen directories to be split from the tail of the task queue. In the stealing state, the subdirectories traversed by the executor are added to the task cache queue first. Step 7: Split the thread to monitor whether the length of the metadata cache queue has reached the load distribution threshold. If yes, trigger the load distribution request; otherwise, return to step 4. Step 8: The load balancer distributes the metadata set of the current batch to the backup process through a weighted round-robin load balancing algorithm to execute specific backup tasks. The metadata set includes subdirectory metadata and subfile metadata. Step 9: After receiving the metadata set, the backup process adds it to the metadata queue from the tail of the backup process's metadata queue. Step 10: The read thread of the backup core module retrieves metadata from the head of the metadata queue, reads the file data of the metadata, and adds it to the data cache queue; Step 11: The write thread of the backup core module reads metadata and file data from the data cache queue and writes them to the storage medium; Step 12: The task splitter monitors whether all splitting threads in the splitting thread pool have finished splitting. If so, it notifies the backup process to start the stealing mechanism; otherwise, it returns to step 8. Step 13: After the backup process starts the theft mechanism, the eavesdropper of each backup process monitors whether the metadata queue of this process is empty. If it is, the theft request is triggered; otherwise, the process returns to step 10. Step 14: Each backup process's eavesdropper sends a stealing request to other backup processes besides itself, and adds the stolen metadata to the end of its own metadata queue. Step 15: If the backup core module of each backup process finishes running and the theft request processing of each backup process finishes, then the current splitting and backup task ends.

[0018] In this embodiment, the task queue to be split adopts a double-ended blocking queue, where data can be accessed from both the head and tail. In step 3, the learning model calculates the optimal number of directories to be stolen and the load distribution threshold based on information from the historical backup set, thereby reducing the number of theft attempts and the network latency consumed by theft requests. This continuously optimizes the theft algorithm and the weighted round-robin load balancing algorithm. In step 6, the stolen directories to be split are added to the queue from the tail of the task queue. During the theft process, subdirectories traversed by the executor are preferentially added to the task queue to avoid competition with the theft thread for the right to use the tail of the task queue, maximizing the performance of concurrent thread operations.

[0019] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0020] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-performance method for automatically and uniformly splitting massive amounts of unstructured data, characterized in that: This includes the following operations: Step 1: Start the splitting process and backup process. The task splitter traverses the set of backup sources and assigns each backup source to an independent splitting thread for individual processing. Step 2: The learning model performs statistical analysis on the historical backup set information, calculates the number of directories and the load distribution threshold of metadata for each theft request in this splitting task through the feature directory, and sends it to the task splitter. Step 3: Each splitting thread's traversal executor performs a breadth-first traversal of the current traversal directory, adds the traversed subdirectories to the splitting task queue exclusively for the corresponding splitting thread, and adds the subdirectory metadata and subfile metadata to the metadata cache queue exclusively for the corresponding splitting thread. Step 4: The eavesdropper of each splitting thread monitors whether the queue of tasks to be split in this splitting thread is empty. If it is, a stealing request is triggered. If not, return to step 3. Step 5: Each splitting thread's eavesdropper sends a stealing request to all splitting threads except itself, and adds the stolen directories to be split to its own splitting thread's task queue. Step 6: Each split thread monitors whether the length of the metadata cache queue has reached the load distribution threshold. If so, a load distribution request is triggered; otherwise, return to step 3. Step 7: After receiving the load distribution request, the load balancer distributes the metadata set of the current batch to the backup process using a weighted round-robin load balancing algorithm. Step 8: After receiving the metadata set, the backup process adds it to the metadata queue from the tail of the backup process's metadata queue. Step 9: The read thread of the backup core module retrieves metadata from the head of the metadata queue, reads the file data of the metadata, and adds it to the data cache queue; Step 10: The write thread of the backup core module reads metadata and file data from the data cache queue and writes them to the storage medium; Step 11: The task splitter monitors whether all splitting threads in the splitting thread pool have finished splitting. If so, it notifies the backup process to start the stealing mechanism. If not, it returns to step 7. Step 12: After the backup process starts the theft mechanism, the eavesdropper of each backup process monitors whether the metadata queue of this process is empty. If it is, the theft request is triggered; otherwise, the process returns to step 9. Step 13: Each backup process's eavesdropper sends a stealing request to other backup processes besides itself, and adds the stolen metadata to the metadata queue of this process. Step 14: Once the backup core module of each backup process finishes running and the theft request processing of each backup process is completed, the current splitting and backup task ends.

2. The method for high-performance automatic uniform splitting of massive unstructured data according to claim 1, characterized in that: Both the task queue to be split and the metadata cache queue are exclusive blocking queues for the split thread. The task queue to be split is a double-ended blocking queue, and the metadata cache queue is a single-ended blocking queue.

3. The method for high-performance automatic uniform splitting of massive unstructured data according to claim 1, characterized in that: Step 3 specifically involves the following steps: The traversal executor of the split thread retrieves the directory to be split from the head of the task queue, performs a breadth-first traversal, adds the traversed subdirectories to the task queue from the tail of the task queue, and adds the metadata of the subdirectories and subfiles to the metadata cache queue from the tail of the metadata cache queue.

4. The method for high-performance automatic uniform splitting of massive unstructured data according to claim 1, characterized in that: In step 5, subdirectories traversed by the traversal executor in the stealing state are preferentially added to the task queue to be split, and the stolen directories to be split are added to the task queue to be split from the tail of the task queue to be split.

5. A high-performance system for automatically and uniformly splitting massive amounts of unstructured data, characterized in that: The system includes: a network interface, a memory, and a processor; The network interface is used for receiving and sending signals during the process of sending and receiving information with other external network elements; The memory is used to store computer programs that can run on the processor; The processor is configured to, when running the computer program, execute the steps of the method for high-performance automatic uniform splitting of massive unstructured data as described in any one of claims 1 to 4.

6. A computer storage medium, characterized in that: A computer storage medium stores a program for automatically and uniformly splitting massive amounts of unstructured data. When the program is executed by at least one processor, it implements the steps of the method for automatically and uniformly splitting massive amounts of unstructured data as described in any one of claims 1 to 4.

Images