Hybrid breadth-first and depth-first traversal backup method and apparatus

Abstract

The application relates to the technical field of data processing, and discloses a hybrid breadth-first and depth-first traversal backup method and device, which comprises the following steps: acquiring a maximum depth value and a concurrency control threshold value, determining a first backup list and a directory to be backed up according to the maximum depth value, and determining a second backup list according to the concurrency control threshold value; and calling a file processing callback function to back up each file in the backup list. Through the above method, the maximum depth value and the concurrency control threshold value are pre-configured, and the directory structure characteristics of different business scenarios are adjusted. According to the configuration parameters, the breadth-first traversal and the depth-first traversal are switched, the breadth-first traversal is preferentially used when the directory structure is detected to be relatively flat, and the depth-first traversal is automatically switched when a nested structure is encountered to perform in-depth traversal. The application can be applied to the business fields of financial technology, medical health and old-age care, and the efficiency of data backup of a data management system is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device and medium for hybrid traversal backup based on breadth-first and depth-first search. Background Technology

[0002] In today's world where data has become a core asset, the fintech and healthcare / elderly care industries have placed unprecedentedly high demands on the integrity, timeliness, and reliability of data backups. However, the single traversal strategies employed by existing backup systems, such as DFS (Depth-First Search) or BFS (Breadth-First Search), have revealed serious flaws when faced with the unique data environments of these industries, becoming a bottleneck for ensuring business continuity and data security.

[0003] In the fintech field, data systems typically exhibit a hybrid structure of "broad and deep". For example, in a core trading system, there is a large amount of recent transaction data that requires rapid access, as well as historical details and contract documents that are deeply archived by year, product, and customer. If DFS is used, when traversing the massive shallow transaction directory, prematurely delving into a certain branch may delay the backup of other critical data at the same level, potentially causing the recovery point target for critical business data to be unmet. If BFS is used, although it can quickly overwrite shallow files, it will generate massive queue memory overhead when processing deeply nested customer profiles or compliance documents, and the traversal efficiency is low.

[0004] In the healthcare and elderly care sectors, data is characterized by "heterogeneous structure and deep correlation." Electronic health records may contain imaging data (large single-layer files), continuous monitoring waveforms (time-series file sequences), structured medical records (database exported files), and various scanned documents, typically organized into a deep catalog based on visit time, department, and disease. The shortcomings of existing backup solutions are particularly prominent. Using Depth-First Search (DFS) to recursively traverse patient record catalogs spanning many years is highly susceptible to stack overflow due to excessively deep paths, causing the backup process to crash and threatening the security of critical medical data. While Breadth-First Search (BFS) can stably control memory usage, it cannot prioritize backing up recently updated intensive care data or emergency records, lacking sensitivity to urgent data. Therefore, in the fintech and healthcare / elderly care sectors, improving the efficiency of data management systems for data backup has become a pressing technical challenge. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for backup based on a hybrid breadth-first and depth-first traversal, in order to improve the efficiency of data backup in a data management system.

[0006] Firstly, this application provides a hybrid traversal backup method based on breadth-first and depth-first search, the method comprising: Get the pre-configured maximum depth value of breadth-first traversal and the concurrency control threshold of depth-first traversal; Based on the maximum depth value, the first backup list and the backup directory are determined from the system to be backed up by the breadth-first traversal. Based on the concurrency control threshold, a second backup list is determined from the directory to be backed up through the depth-first traversal; The preset file processing callback function is invoked to back up each file in the first backup list and the second backup list.

[0007] Secondly, this application also provides a hybrid traversal backup device based on breadth-first and depth-first search, the device comprising: The parameter acquisition module is used to obtain the pre-configured maximum depth value of breadth-first traversal and the concurrency control threshold of depth-first traversal. A breadth-first traversal module is used to determine a first backup list and a backup directory from the system to be backed up based on the maximum depth value through breadth-first traversal. A depth-first traversal module is used to determine a second backup list from the directory to be backed up based on the concurrency control threshold through the depth-first traversal. The backup module is used to call a preset file processing callback function to back up each file in the first backup list and the second backup list.

[0008] Thirdly, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the hybrid traversal backup method based on breadth-first and depth-first search as described above.

[0009] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the hybrid traversal backup method based on breadth-first and depth-first search as described above.

[0010] This application discloses a hybrid traversal backup method, apparatus, device, and medium based on breadth-first search and depth-first search. The hybrid traversal backup method includes obtaining a pre-configured maximum depth value for breadth-first search and a concurrency control threshold for depth-first search; determining a first backup list and a backup directory from the system to be backed up based on the maximum depth value using breadth-first search; determining a second backup list from the backup directory using depth-first search based on the concurrency control threshold; and calling a preset file processing callback function to back up each file in the first and second backup lists. Through this method, this application adaptively adjusts the maximum depth value for breadth-first search and the concurrency control threshold for depth-first search according to the directory structure characteristics of different business scenarios. It automatically switches between breadth-first search and depth-first search based on configuration parameters. When a relatively flat directory structure is detected, breadth-first search is used for rapid overwriting; when a nested structure is encountered, it automatically switches to depth-first search for in-depth traversal. This improves the efficiency of data backup in data management systems in business areas such as fintech and healthcare / elderly care. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a hybrid traversal backup method based on breadth-first and depth-first search provided in an embodiment of this application; Figure 2 A schematic block diagram of a hybrid traversal backup device based on breadth-first and depth-first search provided for embodiments of this application; Figure 3 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0015] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0016] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] This application provides a method, apparatus, device, and medium for backup based on a hybrid breadth-first and depth-first traversal. This hybrid breadth-first and depth-first traversal backup method can be applied to data management systems. By pre-configuring the maximum depth value of breadth-first traversal and the concurrency control threshold of depth-first traversal, it adaptively adjusts the traversal based on the directory structure characteristics of different business scenarios. It automatically switches between breadth-first and depth-first traversal according to the configured parameters. When a relatively flat directory structure is detected, breadth-first traversal is used for rapid overwriting; when nested structures are encountered, it automatically switches to depth-first traversal for in-depth traversal. In business areas such as fintech and healthcare / elderly care, this improves the efficiency of data backup in data management systems.

[0018] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0019] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a hybrid breadth-first and depth-first traversal backup method provided in an embodiment of this application. This hybrid breadth-first and depth-first traversal backup method can be applied in data management systems to improve the efficiency of data backup.

[0020] like Figure 1 As shown, the hybrid traversal backup method based on breadth-first and depth-first search specifically includes steps S10 to S40.

[0021] Step S10: Obtain the pre-configured maximum depth value of breadth-first traversal and the concurrency control threshold of depth-first traversal; Specifically, key pre-defined runtime parameters are obtained from configuration files, user input interfaces, or management consoles, including: The maximum depth value (BFS Depth) of BFS (Breadth-First Search) is used to control the number of directory levels traversed in the BFS phase. For example, setting it to 3 means that only the contents within 3 levels below the root directory are traversed. The concurrency control threshold (concurrency_ls) for DFS (Depth-First Search) limits the number of recursive traversal tasks that can be executed concurrently during the DFS phase; for example, it can be set to 10. Additionally, other optional configurations can be loaded, such as whether to skip hidden files, filtering rules for specific file types, and error handling strategies.

[0022] Step S20: Based on the maximum depth value, determine the first backup list and the backup directory from the system to be backed up through the breadth-first traversal; Specifically, starting from the root directory to be backed up, a breadth-first traversal is performed. A queue is created, and the root directory (depth 0) is added to the queue. The queue is processed iteratively, retrieving all directory entries at the current depth each time, and traversing the files and subdirectories within them. For files, their paths are added to the first backup list. For subdirectories, if their current depth is less than the preset maximum depth value, the subdirectory is added to the queue for the next level of traversal. If their depth has reached or exceeded the maximum depth value, the directory is marked as a directory to be backed up and added to a set of directories to be processed, without further processing. After this stage, the first backup list contains all files found within the specified depth, while the set of directories to be backed up contains all directories that exceed the depth limit and require further depth traversal.

[0023] Before entering the depth-first traversal (DFS) phase, the set of directories to be backed up can be optimized, for example, by sorting them according to directory size, number of files or business priority, or by grouping and batching them to improve subsequent concurrency efficiency.

[0024] Taking the fintech business as an example, the root directory of the data storage of the core financial business system is taken as the starting point, and a breadth-first traversal is initiated. During the traversal, the directory structure is scanned layer by layer according to the preset maximum depth value (e.g., BFSDepth=3).

[0025] Layer 1 (Depth 0): Quickly scan key business directories under the root directory, such as transaction data, risk control data, and user accounts. For real-time files in these directories (such as daily transaction summary files), save their paths to the first backup list to ensure that high-value business data is backed up first.

[0026] The second layer (depth 1): Continue traversing the subdirectories of each business directory. For example, identify subdirectories such as domestic transactions and international transactions. Important files within these subdirectories (such as real-time transaction log files) are also added to the first backup list.

[0027] Level 3 (Depth 2): When depth 2 is reached, more granular directories are processed, such as payment transaction details directories. Files discovered at this depth are added to the first backup list, while these deep directories themselves (depth equal to the maximum depth value of 3) are marked as directories to be backed up.

[0028] This layered processing prioritizes the rapid backup of core business data when the backup window is limited, while complex, deep historical data catalogs (such as multi-year transaction archives organized by date) are handled concurrently in subsequent stages. This satisfies the financial industry's requirement for high availability of critical data and ensures the overall efficiency of the backup task.

[0029] In one embodiment, the root directory of the system to be backed up is used as the starting point of BFS traversal. A linked queue is instantiated and the root path wrapper structure is initialized with its depth field set to 0. A breadth-first scan loop is started. In response to the "tens of millions of small files" characteristic in the financial scenario, a batch loading strategy is adopted in each level of traversal. For example, a memory refresh is triggered after reading 500 transaction file information to prevent memory overflow caused by hundreds of millions of transactions per day. Financial technology compliance preprocessing is performed synchronously for each file path.

[0030] The file processing callback function is called to calculate and append the processed file path and its metadata (size, modification time, file type tag) to the first backup list. When scanning the structured product directory, the current depth value is recorded for each subdirectory identified. If the depth value reaches the preset maximum depth value (e.g., configured to 3 levels to balance traversal efficiency and memory usage), the BFS main loop is stopped immediately. At this time, the untraversed deep subdirectories remaining in the queue (such as product nested directories at level 5 and below, historical archive directories) are batch transferred to the queue of directories to be backed up.

[0031] Taking the medical, health and elderly care business as an example, the backup system stores extremely sensitive and complex data such as electronic health records, medical images, real-time vital signs monitoring data, long-term care plans and patient management information. It is usually organized into a deeply nested directory structure according to dimensions such as patient ID, timestamp, department, and data type.

[0032] The maximum depth value preset for medical backup scenarios is read (e.g., BFSDepth = 4). This setting is based on a typical medical data archiving mode: the 0th level is the root directory, the 1st level may represent the year (e.g., 2025), the 2nd level represents the department, the 3rd level represents the patient ID, and the 4th level is the patient's specific medical records or image data.

[0033] Starting with the root directory of the medical data center to be backed up, add it to the queue and mark its depth as 0. Process the root directory at depth 0, and when subdirectories are found (such as 2024, 2023), add them to the queue and mark their depth as 1. Traverse the directories at depth 1 (such as 2024), which may contain subdirectories divided by department (such as radiology, intensive care unit, etc.). When processing this level, add the department subdirectories to the queue and mark their depth as 2.

[0034] Traverse the department directory at depth 2. Taking the radiology department directory as an example, its subdirectories are named after patient IDs. Add these patient ID directories to the queue, and record the depth as 3.

[0035] When traversing to a directory with a depth of 3 (i.e., the directory containing a specific patient ID), a threshold check is performed. Since the current depth (3) is less than the maximum depth value (4), the contents of that patient's directory will be read.

[0036] Within this patient directory, further subdirectories are organized by date or data type, and the depth of these subdirectories is determined to be 4.

[0037] Since the depth (4) is equal to the maximum depth value (4), it is not appropriate to add these deep clinical data directories to the BFS queue. Instead, add them all to the list of directories to be backed up.

[0038] Meanwhile, when BFS traverses the patient directory at depth 3, if any file directly located at that level (such as the patient's basic information file) is found, its path will be added to the first backup list.

[0039] Step S30: Based on the concurrency control threshold, determine the second backup list from the directory to be backed up by the depth-first traversal; Specifically, based on the concurrency control threshold, a depth-first traversal is initiated concurrently for each directory in the set of directories to be backed up. A semaphore or coroutine pool with a capacity equal to the concurrency control threshold is created; an independent DFS task is started for each directory to be backed up, executing after acquiring permission from the semaphore. Each DFS task recursively traverses all subdirectories and files under its assigned directory: when a file is encountered, its path is added to the local list corresponding to that task; after the recursion ends, the local lists are merged into the global second backup list. Through the semaphore mechanism, the system always keeps the number of concurrent tasks within the threshold, avoiding excessive consumption of system I / O or memory resources.

[0040] During concurrent execution of DFS, CPU utilization, memory usage, and disk I / O load can be monitored in real time, and concurrency control thresholds can be dynamically fine-tuned. For example, when the system load is too high, the concurrency can be temporarily reduced to maintain the stability of the production environment.

[0041] Step S40: Call the preset file processing callback function to back up each file in the first backup list and the second backup list.

[0042] Specifically, after completing the entire traversal and generating the complete first and second backup lists, the system sequentially or concurrently calls preset file processing callback functions to perform actual backup operations on all files in the lists. These callback functions are predefined by the user and typically include logic for file reading, compression, encryption, checksum calculation, and transfer to backup storage media (such as tape libraries, object storage, or off-site data centers). The system can provide a progress callback interface to report backup progress and status in real time.

[0043] After all files are processed, the system generates a backup report, including information such as the number of successfully backed-up files, failed items, time taken, and data volume statistics; performs cleanup operations to release temporary memory and data structures; and, depending on the configuration, can trigger a backup verification process or an incremental backup chain update.

[0044] This embodiment discloses a hybrid traversal backup method, apparatus, device, and medium based on breadth-first and depth-first search. The hybrid traversal backup method includes obtaining a pre-configured maximum depth value for breadth-first search and a concurrency control threshold for depth-first search; determining a first backup list and a directory to be backed up from the system to be backed up based on the maximum depth value using breadth-first search; determining a second backup list from the directory to be backed up based on the concurrency control threshold using depth-first search; and calling a preset file processing callback function to back up each file in the first backup list and the second backup list. Through this method, this application adaptively adjusts the maximum depth value for breadth-first search and the concurrency control threshold for depth-first search according to the directory structure characteristics of different business scenarios by pre-configuring the maximum depth value for breadth-first search and the concurrency control threshold for depth-first search. It automatically switches between breadth-first search and depth-first search based on the configured parameters. When a relatively flat directory structure is detected, breadth-first search is used for rapid overwriting; when a nested structure is encountered, it automatically switches to depth-first search for in-depth traversal. In business areas such as fintech and healthcare / elderly care, this improves the efficiency of data backup in data management systems.

[0045] based on Figure 1 In the illustrated embodiment, step S20 includes: Starting from the root directory of the system to be backed up, perform the breadth-first traversal. The directory structure is scanned layer by layer through the breadth-first traversal until the maximum depth value of the system to be backed up is reached, and the file paths scanned during the traversal are stored in the first backup list. The backup directory is generated based on the file paths in the system to be backed up that exceed the maximum depth value.

[0046] Specifically, the root directory of the system to be backed up, as specified by the user, is used as the starting point for breadth-first traversal and added to a first-in-first-out queue data structure. At the same time, its depth is marked as 0. Directories are continuously taken out from the queue for processing until the queue is empty or the depth of the currently processed directory has reached the preset maximum depth value.

[0047] When traversing each directory level, all entries in the current directory are read. For each entry, if it is a file, its full path is immediately recorded in the "first backup list"; if it is a subdirectory, it is checked whether the depth of the subdirectory (i.e., the current directory depth + 1) is less than the maximum depth value. If it is less, the subdirectory is added to the end of the queue, waiting for subsequent traversal using the breadth-first strategy; if the subdirectory depth is equal to the maximum depth value, the system does not continue traversing the subdirectory, but instead adds the path of the subdirectory itself to a special "backup directories" set, marking this as the starting point of the depth-first traversal.

[0048] After the breadth-first traversal phase is completed, the "first backup list" contains all the specific file paths found within the maximum depth value limit, while the "directories to be backed up" set contains all directory paths that are exactly at the maximum depth value boundary and require further exploration. These directories to be backed up will serve as input for subsequent depth-first traversal phases to ensure the integrity of the backups.

[0049] In one embodiment, the root directory to be backed up is added as the initial node to a first-in-first-out queue, and its depth is marked as 0, thus initiating the BFS loop process. At the beginning of each loop, all directory nodes at the same depth in the current queue are checked. For each directory at that level, all entries (including files and subdirectories) are read. For each entry, if it is a file, its full path is immediately added to the first backup list; if it is a subdirectory, its depth (parent directory depth + 1) is calculated and compared with a preset maximum depth value.

[0050] If the depth of a subdirectory is less than the maximum depth, the directory is still within the traversal range of BFS. Its path and current depth are added as new nodes to the end of the queue, waiting for scanning of subsequent levels. If the depth of a subdirectory is equal to or greater than the maximum depth value, it means that the directory has exceeded the traversal boundary of BFS. Its path is added to a special "backup directory" list, which marks the starting point for subsequent concurrent DFS traversals.

[0051] This loop continues, processing directory by directory, until the queue is empty or all directories not yet at the maximum depth have been processed. Ultimately, the first backup list contains all scanned file paths from the root directory to the maximum depth, while the list of directories to be backed up contains all discovered deep directory paths that require further traversal.

[0052] based on Figure 1 In the illustrated embodiment, the steps preceding step S30 include: Based on the concurrency control threshold, a concurrency control signal is generated; The concurrency control signal is assigned to each file path in the directory to be backed up through the depth-first traversal.

[0053] Specifically, a "concurrency control signal" is generated based on a preset "concurrency control threshold". In practice, the concurrency control signal is usually a counting semaphore, whose initial number of allowed operations is equal to the set concurrency control threshold. For example, if the threshold is set to 10, a semaphore with an initial value of 10 is created.

[0054] The process iterates through each directory path in the "Directories to be Backup" list, launching a separate DFS traversal task (usually encapsulated as a coroutine or thread) for each directory. Before any task can start execution, it must first request a permit from the semaphore. This request is synchronous: if there are available permits in the semaphore (i.e., the number of currently active DFS tasks has not reached the threshold), the task immediately acquires a permit and begins its DFS traversal; if all permits are occupied, the task is blocked and enters a waiting queue until a previous task finishes execution and releases a permit, at which point it can be awakened and continue execution.

[0055] Within each DFS task, all files and subdirectories under the allocated directory are recursively traversed, and the discovered file paths are added to a second backup list. When the task completes all its traversal work, whether successful or not, it calls the release interface before exiting, returning the permits it held to the semaphore. This release operation increases the number of available permits on the semaphore, automatically waking up a waiting subsequent task (if any), forming a smooth pipeline operation.

[0056] This method ensures that the number of concurrently executed DFS tasks will not exceed the set threshold at any time, effectively avoiding system resource contention and I / O bottlenecks caused by excessive concurrency.

[0057] In one embodiment, a concurrency control semaphore is generated based on a preset DFS concurrency control threshold (such as the maximum number of concurrent threads, coroutines, or processes). For example, if the threshold is 10, a semaphore with 10 permits is created, each permit representing a DFS traversal task unit that can be executed concurrently.

[0058] The list of directories to be backed up is converted into a thread-safe task queue. Each queue element contains the absolute path of the directory, the directory's depth information in the file system, and optional traversal configuration parameters (such as whether to follow symbolic links, file filtering rules, etc.). Simultaneously, the following components are initialized: Worker thread pool or coroutine pool (if no automatic management mechanism is used); A thread-safe set used to collect file paths discovered during the DFS phase—that is, a second backup list; Main thread wait groups or barriers used for synchronization.

[0059] Loop through the task queue to retrieve directories to be backed up, and assign concurrent control signals to each directory: Before executing a specific DFS traversal, a worker thread must first request a permit through a semaphore acquisition operation. If no permit is currently available, the thread will be blocked until another task completes and releases the permit. After successfully acquiring the permit, a separate DFS traversal task (thread, coroutine, or asynchronous task) is created for the directory, and the directory path, depth information, and semaphore reference are passed to that task.

[0060] During execution, continuously monitor CPU utilization, memory usage, I / O load; semaphore wait queue length; and the execution time of each DFS task. Dynamically adjust the actual concurrency based on monitoring data. If the system load is low and there are directories to be processed, the number of concurrent licenses can be temporarily increased (not exceeding the hardware limit). If memory pressure or I / O contention is detected, the number of active concurrent tasks should be reduced accordingly.

[0061] All DFS tasks are ensured to complete correctly through the following mechanism: The main thread uses a wait group to wait for all concurrent tasks to finish; when a task encounters an exception, a try-catch mechanism is used to ensure that the semaphore permission is released, and all entries in the second backup list collected by all threads are merged, deduplicated, and sorted.

[0062] For directories that encounter errors during traversal (such as insufficient permissions or non-existent paths): the error information is recorded in a dedicated log queue, the semaphore permission occupied by the task is immediately released, and the failure directory is re-added to the end of the queue for retry, depending on the configuration.

[0063] In a specific embodiment, step S30 includes: In response to the concurrency control signal, the number of concurrently executed tasks is determined according to the concurrency control threshold; The number of concurrent tasks is used to dynamically schedule each subdirectory in the directory to be backed up, and the depth-first traversal is performed concurrently based on the number of concurrent tasks. The file paths scanned during the depth-first traversal are determined as the second backup list.

[0064] Specifically, in response to generated concurrency control signals (usually semaphores), the maximum number of DFS tasks that can be executed simultaneously is determined based on a preset concurrency control threshold, such as 10. A task scheduler is created, the core of which is a queue containing all directories to be backed up and a pool of worker coroutines (or threads). The scheduler dynamically retrieves directory paths from the queue and assigns them to idle work units. Each work unit must successfully acquire permission from the semaphore before starting to process a directory, ensuring that the number of active tasks never exceeds the threshold. Once permission is acquired, the work unit independently and concurrently executes the standard DFS algorithm: recursively traversing all subdirectories of the assigned directory, adding its full path to a shared second backup list whenever a file is scanned. When a directory and all its subdirectories have been traversed, the work unit releases the semaphore permission and becomes idle. The scheduler then assigns it a new directory to be backed up, and this process repeats until all directories in the queue have been processed, ultimately generating a complete second backup list.

[0065] In one embodiment, in response to the concurrency control signal (i.e., semaphore mechanism), the current available count of the semaphore is queried in real time to dynamically determine the number of concurrent tasks that can actually be executed this time. The upper limit of the number of concurrent tasks is a preset concurrency control threshold, but it is adaptively adjusted at runtime according to system load, I / O pressure and the completion status of active tasks.

[0066] Based on the actual concurrent number, a corresponding number of subdirectory paths are retrieved in batches from the queue of directories to be backed up. An independent DFS traversal task is assigned to each path and submitted to the coroutine pool for execution. Each task must obtain an execution permission before starting to ensure that the number of concurrent tasks never exceeds the dynamically determined concurrency limit. After the task is completed, the semaphore is returned to trigger the scheduling of the next batch of tasks.

[0067] During the concurrent execution of DFS, each task recursively scans all deep files and subdirectories under the assigned subdirectory, and appends the scanned file paths, metadata, and hierarchical relationship information to the thread's local second backup list cache in real time. At the same time, it records the directory depth and scanning progress for the monitoring module to collect. After all directories to be backed up have been processed, the second backup lists cached by each thread are summarized and merged. After deduplication, sorting, and integrity verification, the final second backup list is generated.

[0068] based on Figure 1 In the illustrated embodiment, step S40 includes: The first backup list and the second backup list are merged to generate a third backup list; The third backup list is deduplicated to generate a target backup list, and the file paths of each file in the target backup list are traversed to read the metadata of each file; The preset file processing callback function backs up each file according to its file path and metadata.

[0069] Specifically, the first backup list (the shallow file path list generated by the BFS phase) and the second backup list (the deep file path list generated by the concurrent DFS phase) are merged to generate a temporary third backup list. To ensure the accuracy of the backup targets, this third backup list is deduplicated, that is, duplicate file paths that may be caused by symbolic links or traversal boundary conditions are removed, ultimately generating a complete and unique target backup list.

[0070] Iterate through each file path in the target backup list and read the metadata of the corresponding file one by one. This metadata usually includes key attributes such as file size, last modification time, permission settings, owner information, and file checksum.

[0071] The system invokes a pre-defined file processing callback function, passing the current file path and the read metadata as parameters. The callback function encapsulates the specific backup business logic. Executing this function completes the actual backup operation for each file, opening the file stream based on the file path, determining whether incremental backup is needed based on the metadata, encrypting and compressing the file data, and securely transmitting or writing the final data blocks to the specified backup storage medium. This process of list integration, deduplication, metadata reading, and callback ensures both the integrity and accuracy of the backup, while decoupling the core traversal logic from the variable backup strategies, thus ensuring the system's flexibility and reliability.

[0072] based on Figure 1 In the illustrated embodiment, the steps preceding step S20 include: If the maximum depth value is less than or equal to 0, the root directory of the system to be backed up is taken as the starting point, and the depth-first traversal is performed to generate the fourth backup list. The preset file processing callback function is used to back up each file in the fourth backup list.

[0073] Specifically, if the maximum depth value (BFSDepth) is less than or equal to 0, it will be determined that the user wants to use a pure depth-first traversal strategy, skipping the entire BFS stage, including the initialization of the BFS queue, level traversal, and the generation of the queue of directories to be backed up, and directly using the root directory to be backed up specified by the user as the sole starting point of the depth-first traversal.

[0074] A recursive DFS process is initiated, first reading all entries in the root directory. For each file encountered, its path is directly recorded in the final target backup list. For each subdirectory encountered, the process immediately recursively calls itself, deeply traversing that subdirectory and all its descendant directories, following the principle of "deepening first and then backtracking," until the entire directory tree has been traversed. In this pure DFS mode, since there are no concurrency control thresholds (unless DFS itself is configured with concurrency), the entire traversal process is typically executed sequentially in a single thread to ensure the simplest execution model and the lowest memory management complexity.

[0075] Once the entire DFS traversal process is complete, a complete list of target backups is obtained, which contains all accessible file paths starting from the root directory. This list is traversed, and preset file processing callback functions are called to perform specific backup operations one by one.

[0076] Based on any of the above embodiments, in this embodiment, step S10 includes: Read the initial depth value and initial concurrency threshold from the preset configuration file; The initial depth value and the initial concurrency threshold are validated for validity, respectively. If the initial depth value passes the validity check, the initial depth value is determined as the maximum depth value; If the initial concurrency threshold passes the validity check, the initial concurrency threshold is determined as the concurrency control threshold.

[0077] Specifically, two key configuration items are read from the preset configuration file: the initial depth value and the initial concurrency threshold. After reading the raw strings or numbers, these two parameters are subjected to strict validity checks. The validation logic typically includes: Type checking ensures that the value read is a valid integer type; Range check: For the initial depth value, check whether it is greater than or equal to a certain minimum threshold (e.g., not less than -1, where negative values ​​have special meaning). For the initial concurrency threshold, check if it is greater than 0 and less than the maximum limit that a system can withstand (e.g., less than 1000) to prevent resource exhaustion.

[0078] If the initial depth value passes all validity checks, this validated value from the configuration file is officially designated as the maximum depth value (BFSDepth) used in subsequent processes. If the validation fails, the invalid value is not used, and a fallback strategy is adopted, such as logging an error, throwing an exception to terminate the process, or more commonly, falling back to a predefined, safe default depth value (for example) to ensure that the system can continue to operate in a known state.

[0079] Similarly, if the initial concurrency threshold passes its validity check, it is determined as the concurrency control threshold used by the concurrency control module. If the check fails, a degradation strategy will be triggered, such as falling back to a conservative default concurrency number (e.g., 5).

[0080] Please see Figure 2 , Figure 2 This application provides a schematic block diagram of a hybrid breadth-first and depth-first traversal backup device, which is used to execute the aforementioned hybrid breadth-first and depth-first traversal backup method. This hybrid breadth-first and depth-first traversal backup device can be configured on a server.

[0081] like Figure 2 As shown, the hybrid traversal backup device 400 based on breadth-first and depth-first search includes: The parameter acquisition module 410 is used to acquire the pre-configured maximum depth value of breadth-first traversal and the concurrency control threshold of depth-first traversal. Breadth-first traversal module 420 is used to determine a first backup list and a backup directory from the system to be backed up based on the maximum depth value through breadth-first traversal; The depth-first traversal module 430 is used to determine a second backup list from the directory to be backed up based on the concurrency control threshold through the depth-first traversal. The backup module 440 is used to call a preset file processing callback function to back up each file in the first backup list and the second backup list.

[0082] Furthermore, the breadth-first traversal module 420 includes: The BFS execution unit is used to take the root directory of the system to be backed up as the starting point and execute the breadth-first traversal. The first backup list generation unit is used to scan the directory structure layer by layer through the breadth-first traversal until the maximum depth value of the system to be backed up is reached, and to store the file paths scanned during the traversal into the first backup list. The backup directory generation unit is used to generate the backup directory based on the file paths in the backup system that exceed the maximum depth value.

[0083] Furthermore, the hybrid traversal backup device 400 based on breadth-first and depth-first search includes: A concurrent control signal generation module is used to generate concurrent control signals according to the concurrent control threshold. The concurrent control signal allocation module is used to allocate the concurrent control signal to each file path in the directory to be backed up through the depth-first traversal.

[0084] Furthermore, the depth-first traversal module 430 includes: A concurrent task execution number determination unit is used to determine the number of concurrent tasks in response to the concurrent control signal, based on the concurrent control threshold. The DFS execution unit is used to dynamically schedule each subdirectory in the directory to be backed up according to the number of concurrent execution tasks, and concurrently execute the depth-first traversal based on the number of concurrent execution tasks; The second backup list generation unit is used to determine the file paths scanned during the depth-first traversal as the second backup list.

[0085] Furthermore, the backup module 440 includes: The third backup list generation unit is used to merge the first backup list and the second backup list to generate a third backup list; Meta-information reading unit is used to perform deduplication processing on the third backup list to generate a target backup list, and to traverse the file paths of each file in the target backup list to read the meta-information of each file; The backup unit is used to back up each file according to the file path and the metadata through the preset file processing callback function.

[0086] Furthermore, the hybrid traversal backup device 400 based on breadth-first and depth-first search includes: The fourth backup list generation module is used to generate the fourth backup list by taking the root directory of the system to be backed up as the starting point and performing the depth-first traversal when the maximum depth value is less than or equal to 0. The fourth backup list backup module is used to back up each file in the fourth backup list through the preset file processing callback function.

[0087] Furthermore, the parameter acquisition module 410 includes: The initial parameter reading unit is used to read the initial depth value and the initial concurrency threshold from the preset configuration file; A validity verification unit is used to verify the validity of the initial depth value and the initial concurrency threshold, respectively. The maximum depth value determination unit is used to determine the initial depth value as the maximum depth value if the initial depth value passes the validity check. A concurrency control threshold determination unit is used to determine the initial concurrency threshold as the concurrency control threshold if the initial concurrency threshold passes the validity check.

[0088] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0089] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 3 It runs on the computer device shown.

[0090] Please see Figure 3 , Figure 3 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0091] See Figure 3 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0092] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any hybrid breadth-first and depth-first traversal backup method.

[0093] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0094] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to perform any hybrid traversal backup method based on breadth-first and depth-first search.

[0095] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0096] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0097] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: Get the pre-configured maximum depth value of breadth-first traversal and the concurrency control threshold of depth-first traversal; Based on the maximum depth value, the first backup list and the backup directory are determined from the system to be backed up by the breadth-first traversal. Based on the concurrency control threshold, a second backup list is determined from the directory to be backed up through the depth-first traversal; The preset file processing callback function is invoked to back up each file in the first backup list and the second backup list.

[0098] In one embodiment, based on the maximum depth value, a first backup list and a backup directory are determined from the system to be backed up through breadth-first traversal, to achieve the following: Starting from the root directory of the system to be backed up, perform the breadth-first traversal. The directory structure is scanned layer by layer through the breadth-first traversal until the maximum depth value of the system to be backed up is reached, and the file paths scanned during the traversal are stored in the first backup list. The backup directory is generated based on the file paths in the system to be backed up that exceed the maximum depth value.

[0099] In one embodiment, before determining the second backup list from the directory to be backed up based on the concurrency control threshold using the depth-first traversal, the following is implemented: Based on the concurrency control threshold, a concurrency control signal is generated; The concurrency control signal is assigned to each file path in the directory to be backed up through the depth-first traversal.

[0100] In one embodiment, based on the concurrency control threshold, a second backup list is determined from the directory to be backed up through a depth-first traversal, for the following purpose: In response to the concurrency control signal, the number of concurrently executed tasks is determined according to the concurrency control threshold; The number of concurrent tasks is used to dynamically schedule each subdirectory in the directory to be backed up, and the depth-first traversal is performed concurrently based on the number of concurrent tasks. The file paths scanned during the depth-first traversal are determined as the second backup list.

[0101] In one embodiment, a preset file processing callback function is invoked to back up each file in the first backup list and the second backup list, for the purpose of: The first backup list and the second backup list are merged to generate a third backup list; The third backup list is deduplicated to generate a target backup list, and the file paths of each file in the target backup list are traversed to read the metadata of each file; The preset file processing callback function backs up each file according to its file path and metadata.

[0102] In one embodiment, when the maximum depth value is greater than 0, before determining the first backup list and the backup directory from the system to be backed up based on the maximum depth value through the breadth-first traversal, the following is implemented: If the maximum depth value is less than or equal to 0, the root directory of the system to be backed up is taken as the starting point, and the depth-first traversal is performed to generate the fourth backup list. The preset file processing callback function is used to back up each file in the fourth backup list.

[0103] In one embodiment, the pre-configured maximum depth value of breadth-first traversal and the concurrency control threshold of depth-first traversal are obtained to implement: Read the initial depth value and initial concurrency threshold from the preset configuration file; The initial depth value and the initial concurrency threshold are validated for validity, respectively. If the initial depth value passes the validity check, the initial depth value is determined as the maximum depth value; If the initial concurrency threshold passes the validity check, the initial concurrency threshold is determined as the concurrency control threshold.

[0104] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the hybrid traversal backup methods based on breadth-first and depth-first search provided in the embodiments of this application.

[0105] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0106] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hybrid traversal backup method based on breadth-first and depth-first search, characterized in that, include: Get the pre-configured maximum depth value of breadth-first traversal and the concurrency control threshold of depth-first traversal; If the maximum depth value is greater than 0, the first backup list and the backup directory are determined from the system to be backed up based on the maximum depth value through the breadth-first traversal. Based on the concurrency control threshold, a second backup list is determined from the directory to be backed up through the depth-first traversal; The preset file processing callback function is invoked to back up each file in the first backup list and the second backup list.

2. The hybrid traversal backup method based on breadth-first and depth-first search as described in claim 1, characterized in that, The process of determining the first backup list and the backup directory from the system to be backed up based on the maximum depth value through breadth-first traversal includes: Starting from the root directory of the system to be backed up, perform the breadth-first traversal. The directory structure is scanned layer by layer through the breadth-first traversal until the maximum depth value of the system to be backed up is reached, and the file paths scanned during the traversal are stored in the first backup list. The backup directory is generated based on the file paths in the system to be backed up that exceed the maximum depth value.

3. The hybrid traversal backup method based on breadth-first and depth-first search as described in claim 1, characterized in that, Before determining the second backup list from the directory to be backed up based on the concurrency control threshold using the depth-first traversal, the process includes: Based on the concurrency control threshold, a concurrency control signal is generated; The concurrency control signal is assigned to each file path in the directory to be backed up through the depth-first traversal.

4. The hybrid traversal backup method based on breadth-first and depth-first search as described in claim 3, characterized in that, The step of determining the second backup list from the directory to be backed up based on the concurrency control threshold through the depth-first traversal includes: In response to the concurrency control signal, the number of concurrently executed tasks is determined according to the concurrency control threshold; The number of concurrent tasks is used to dynamically schedule each subdirectory in the directory to be backed up, and the depth-first traversal is performed concurrently based on the number of concurrent tasks. The file paths scanned during the depth-first traversal are determined as the second backup list.

5. The hybrid traversal backup method based on breadth-first and depth-first search as described in claim 1, characterized in that, The step of calling the preset file processing callback function to back up each file in the first backup list and the second backup list includes: The first backup list and the second backup list are merged to generate a third backup list; The third backup list is deduplicated to generate a target backup list, and the file paths of each file in the target backup list are traversed to read the metadata of each file; The preset file processing callback function backs up each file according to its file path and metadata.

6. The hybrid traversal backup method based on breadth-first and depth-first search as described in claim 1, characterized in that, Before determining the first backup list and the backup directory from the system to be backed up based on the maximum depth value when the maximum depth value is greater than 0, the process includes: If the maximum depth value is less than or equal to 0, the root directory of the system to be backed up is taken as the starting point, and the depth-first traversal is performed to generate the fourth backup list. The preset file processing callback function is used to back up each file in the fourth backup list.

7. The hybrid traversal backup method based on breadth-first and depth-first search according to any one of claims 1 to 6, characterized in that, The process of obtaining the pre-configured maximum depth value for breadth-first traversal and the concurrency control threshold for depth-first traversal includes: Read the initial depth value and initial concurrency threshold from the preset configuration file; The initial depth value and the initial concurrency threshold are validated for validity, respectively. If the initial depth value passes the validity check, the initial depth value is determined as the maximum depth value; If the initial concurrency threshold passes the validity check, the initial concurrency threshold is determined as the concurrency control threshold.

8. A hybrid traversal backup device based on breadth-first and depth-first search, characterized in that, include: The parameter acquisition module is used to obtain the pre-configured maximum depth value of breadth-first traversal and the concurrency control threshold of depth-first traversal. A breadth-first traversal module is used to determine a first backup list and a backup directory from the system to be backed up based on the maximum depth value through breadth-first traversal. A depth-first traversal module is used to determine a second backup list from the directory to be backed up based on the concurrency control threshold through the depth-first traversal. The backup module is used to call a preset file processing callback function to back up each file in the first backup list and the second backup list.

9. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the hybrid traversal backup method based on breadth-first and depth-first as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the hybrid traversal backup method based on breadth-first and depth-first search as described in any one of claims 1 to 7.

Images