File differential backup method, apparatus and computer-readable storage medium

Abstract

This application provides a file differential backup method, apparatus, and computer-readable storage medium. After obtaining the directory to be backed up and the target directory on the target storage device, the method dynamically allocates a first thread from a first thread pool to each subtask based on the file parameters of each subtask in the directory to be backed up. Based on this first thread, the directory to be backed up and the target directory are compared to obtain a difference comparison result. Then, based on the difference comparison result, each differential subtask is determined, and a second thread from a second thread pool is dynamically allocated to each differential subtask. Finally, each differential subtask is transmitted to the target storage device based on the second thread. This method, by dynamically allocating threads according to file parameters during the comparison and transmission phases, and by executing the comparison and transmission asynchronously, ensures the rationality of resource allocation and improves resource utilization and backup efficiency.

Description

Technical Field

[0001] This application relates to the field of computer data storage technology, and in particular to a file differential backup method, apparatus and computer-readable storage medium. Background Technology

[0002] Existing file backup solutions are mainly divided into full backup, incremental backup, and differential backup. Among them, differential backup, by backing up only the files that have changed since the last full backup, strikes a balance between storage space and recovery efficiency, and has become the mainstream solution.

[0003] However, current differential backup solutions have the following drawbacks: 1) Traditional solutions use a single thread to sequentially perform file comparison and transfer, which leads to the blocking of the transfer process during the comparison stage, making it impossible to fully utilize multi-core CPUs and network bandwidth, resulting in low backup efficiency; 2) Although multi-threaded solutions can improve efficiency, the number of threads is fixed or manually preset, which can easily lead to thread idleness or resource contention when dealing with mixed file types (such as a large number of small files mixed with a small number of large files); 3) Existing technologies require waiting for a complete difference list to be generated before the transfer can be started, which cannot achieve asynchronous parallel processing of comparison and transfer, resulting in wasted time.

[0004] Therefore, existing file differential backup methods suffer from technical problems such as low backup efficiency and low resource utilization, and need to be improved. Summary of the Invention

[0005] This application provides a file differential backup method, apparatus, and computer-readable storage medium to improve backup efficiency and resource utilization.

[0006] To address the aforementioned technical problems, this application provides the following technical solution:

[0007] This application provides a file differential backup method, including:

[0008] Obtain the directory to be backed up, the target directory, and the file parameters of each subtask in the directory to be backed up;

[0009] Based on the file parameters of each subtask, dynamically allocate the first thread in the first thread pool to each subtask;

[0010] Based on the comparison between the directory to be backed up and the target directory by the first thread, the difference comparison results are obtained;

[0011] Based on the difference comparison results, each difference subtask is determined, and a second thread of the second thread pool is dynamically allocated to each difference subtask.

[0012] Based on the second thread, each differentiated subtask is transmitted to the target storage.

[0013] Accordingly, this application also provides a file differential backup device, comprising:

[0014] The first acquisition module is used to acquire the directory to be backed up, the target directory, and the file parameters of each subtask in the directory to be backed up.

[0015] The first thread allocation module is used to dynamically allocate the first thread in the first thread pool to each subtask based on the file parameters of each subtask.

[0016] The difference comparison module is used to compare the directory to be backed up and the target directory based on the first thread to obtain the difference comparison result;

[0017] The second thread allocation module is used to determine each difference subtask based on the difference comparison result, and dynamically allocate the second thread of the second thread pool to each difference subtask.

[0018] The file transfer module is used to transfer the different subtasks to the target storage end based on the second thread.

[0019] Meanwhile, this application provides an electronic device including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to run the computer program in the memory to perform the steps in the above-described file differential backup method.

[0020] In addition, this application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform the steps in the above-described file differential backup method.

[0021] Beneficial Effects: This application provides a file differential backup method, apparatus, and computer-readable storage medium. Specifically, after obtaining the directory to be backed up, the target directory, and the file parameters of each subtask in the directory to be backed up, the method dynamically allocates a first thread from a first thread pool to each subtask based on the file parameters of each subtask. Based on this first thread, the directory to be backed up and the target directory are compared to obtain a difference comparison result. Then, based on the difference comparison result, each differential subtask is determined, and a second thread from a second thread pool is dynamically allocated to each differential subtask. Finally, each differential subtask is transmitted to the target storage terminal based on the second thread. This method ensures the rationality of resource allocation and improves resource utilization and backup efficiency by calculating the comparison weight and transmission weight based on the file parameters of the subtasks and differential subtasks respectively during the comparison and transmission stages, and dynamically allocating corresponding threads from the first and second thread pools accordingly. Furthermore, the asynchronous execution of comparison and transmission solves the file transmission blocking problem, saves a significant amount of backup time, and improves backup efficiency. Attached Figure Description

[0022] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0023] Figure 1 This is a schematic diagram illustrating the application scenario of the file differential backup method provided in the embodiments of this application.

[0024] Figure 2 This is a design diagram of a multi-threaded asynchronous file backup system based on Robocopy, provided in an embodiment of this application.

[0025] Figure 3 This is a flowchart illustrating the file differential backup method provided in the embodiments of this application.

[0026] Figures 4-10 This is the file backup operation interface provided in the embodiments of this application.

[0027] Figures 11-12 This is the file recovery operation interface provided in the embodiments of this application.

[0028] Figure 13 This is a schematic diagram of the file differential backup device provided in the embodiments of this application.

[0029] Figure 14 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0031] The terms "comprising" and "having" and any variations thereof in the specification and claims of this application are intended to cover non-exclusive inclusion; the division of modules appearing in this application is merely a logical division, and in actual application, there may be other division methods, such as multiple modules being combined into or integrated into another system, or some features being ignored or not executed.

[0032] This application provides a file differential backup method, apparatus, electronic device, and computer-readable storage medium, wherein the file differential backup apparatus can be integrated into an electronic device, which can be a server or a terminal or other similar device.

[0033] Please see Figure 1 , Figure 1This is a schematic diagram illustrating a scenario in which the file differential backup method provided in this application is applied, such as... Figure 1 As shown, this scenario can include terminals and devices, with terminals communicating with each other, devices communicating with each other, and terminals communicating with devices via the Internet composed of various gateways. This application scenario includes at least a source storage terminal 101 and a target storage terminal 102.

[0034] The source storage device 101 can be a physical host or virtual machine running a Windows operating system, and its storage media includes local disks, direct-attached storage arrays, or network storage mounted on SMB / NFS.

[0035] The target storage device 102 can be a file sharing service that supports SMBv2 and above protocols. Its physical carrier includes, but is not limited to, enterprise-grade NAS devices, distributed file system clusters, and SMB shared directories mapped by cloud storage gateways. The target storage device 102 can provide file sharing services through the SMB protocol and is configured to support persistent handles and multi-channel transmission.

[0036] The source storage terminal 101 and the target storage terminal 102 are based on different networks but are connected by a communication link to enable information exchange. The type of communication link may include wired, wireless communication links, or fiber optic cables, etc., and this application does not impose any restrictions on this.

[0037] like Figure 2 As shown, after obtaining the backup task, the source storage terminal 101 determines the directory to be backed up and the target directory, and obtains the file parameters of each subtask in the directory to be backed up. Then, based on the file parameters of each subtask, it dynamically allocates the first thread in the first thread pool to each subtask, and compares the directory to be backed up and the target directory based on the first thread to obtain the difference comparison result. Next, based on the difference comparison result, it determines each difference subtask and dynamically allocates the second thread in the second thread pool to each difference subtask. Finally, based on the second thread, it transmits each difference subtask to the target storage terminal 102, and the target storage terminal 102 obtains the backup set.

[0038] In the above file differential backup process, the comparison and transmission are executed asynchronously, which solves the file transmission blocking problem and saves a lot of backup time compared with traditional backup schemes, thus improving backup efficiency. In addition, in the comparison and transmission phases, the comparison weight and transmission weight are calculated according to the file parameters of the subtasks, and the corresponding threads are dynamically allocated to the subtasks from the first thread pool and the second thread pool based on the comparison weight and transmission weight, respectively, which ensures the rationality of resource allocation and improves resource utilization and backup efficiency.

[0039] It should be noted that, Figure 1The illustrated scenario is merely an example. The terminals, devices, and scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems. Detailed descriptions are provided below. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0040] In the embodiments of this application, please refer to Figure 3 As shown, Figure 3 This is a flowchart illustrating a file differential backup method provided in an embodiment of this application. The file differential backup method is applied to the source storage end. The method includes at least the following steps:

[0041] S201: Obtain the directory to be backed up, the target directory of the target storage, and the file parameters of each subtask in the directory to be backed up.

[0042] The directory to be backed up refers to the original data storage location that needs to be backed up, which contains a tree structure of files and subdirectories.

[0043] The target directory refers to the final storage location of the difference files.

[0044] Each subtask in the directory to be backed up refers to the smallest unit of execution within the directory; the file parameters of each subtask may include the number of files, file size, and file type.

[0045] It should be noted that the backup directory and the target directory of the target storage can be entered by the user from the front end. After the source storage is determined, the source storage will obtain the backup directory and the target directory of the target storage. Figures 4-9 This is the file backup operation interface provided in the embodiments of this application.

[0046] like Figure 4 As shown, users can select the Windows agent (such as...) through the operation interface displayed on the front end. Figure 3 The production equipment shown has an IP address of 145.170.45.208, from which the source storage can be determined; then, as... Figure 5 As shown, a backup plan is created through the user interface, thereby determining the target storage device (such as...). Figure 5 The device shown has an IP address of 145.170.45.203, and it is determined that differential backup should be performed on the source storage end; such as Figure 6As shown, by adding the protected file through the operation interface, the backup directory (D:\whl\data\data-mini) and the target directory of the target storage end (D:\whl\data\data-mini-copy) are determined.

[0047] In one embodiment, step S201 includes: obtaining a first preset task partitioning condition, a directory to be backed up, and a target directory of the target storage end; splitting the directory to be backed up into independent subtasks according to the first preset task partitioning condition; and reading the attributes of each subtask to obtain the file parameters of each subtask. The first preset task partitioning condition refers to the conditions used to split the directory to be backed up into subtasks. This condition can be configured by the user through a front-end interface or it can be a default configuration. For example, the first preset task partitioning condition can be a file quantity threshold, a file size threshold, or a subdirectory level threshold, etc. The file parameters can include the file size, number of files, and file type of the subtask.

[0048] Specifically, the steps for dividing the directory into subtasks can be as follows: recursively scan the directory to be backed up, and split the directory into independent subtasks according to the first preset task division conditions. For example, recursively scan the entire directory according to the file tree structure of the backup directory. If the number of files in the directory is greater than the file number threshold, continue scanning the next level of subdirectories; if the already scanned directory level is less than the subdirectory level threshold, continue scanning the next level; repeat the scanning of subdirectories at each level until the corresponding subtasks are obtained. If the number of files in the directory is less than the file number threshold, aggregate the files in the directory into one subtask. If the scanned directory level is greater than the subdirectory level threshold, aggregate the files in the directory into one subtask. It should be noted that when there are a large number of small files in the directory, the division can be based on the file number threshold (e.g., merging every 500 files into one subtask); when there are large files, the division can be based on the file size threshold (e.g., a single file > 100MB is considered an independent subtask); when the directory level is shallow, a single file can also be considered a subtask.

[0049] In another embodiment, the directory to be backed up is divided into independent subtasks according to a first preset task division condition. For example, the subfolders of the file tree in the directory to be backed up are taken as the smallest unit, and each subfolder is initially regarded as a candidate subtask; if the number of files in the candidate subtask is greater than or equal to the number of files (e.g., 1000), it is clustered and split into multiple new subtasks according to file type (e.g., the disclosure document splits the source code of 1200 files into two subtasks of 600 files each); if the size of a single file contained in a candidate subtask is greater than or equal to the file size threshold (e.g., 500MB), then the file is directly upgraded to an independent subtask.

[0050] In extremely large-scale data scenarios, the first preset task partitioning conditions for dividing the backup directory can be dynamically adjusted to achieve a triple breakthrough of load balancing, precise progress, and fault isolation, providing an ideal task granularity basis for subsequent differential backups.

[0051] S202: Based on the file parameters of each subtask, dynamically allocate the first thread in the first thread pool to each subtask.

[0052] In one embodiment, step S202 includes: determining the comparison weight of each subtask based on the file parameters of each subtask; and dynamically allocating a first thread from the first thread pool to each subtask based on the comparison weight.

[0053] Specifically, after determining the comparison weight of each subtask, a corresponding number of first threads are allocated from the first thread pool to the corresponding subtask based on the magnitude of the comparison weight, for performing the difference comparison of that subtask. For example, if the comparison weight of a subtask accounts for 60% of the total comparison weight of all subtasks, then 60% of the first threads in the first thread pool are allocated to that subtask to perform the difference comparison. During the thread allocation process, different numbers of threads are allocated based on the magnitude of the weight value; more threads are allocated to tasks with high weights, thereby avoiding comparison blocking. In addition, the number of threads is dynamically adjusted according to file parameters, which reduces comparison time and improves comparison efficiency compared to a fixed-thread scheme.

[0054] In one embodiment, the step of determining the comparison weight of each subtask based on the file parameters of each subtask includes: obtaining preset coefficients; determining the base weight based on the number of files, file size, and preset coefficients; determining the weight adjustment factor based on the file type distribution; and determining the comparison weight of the subtask based on the weight adjustment factor and the base weight. Wherein, the base weight = a * file size + b * number of files; the preset coefficients refer to the pre-coefficients for file size and number of files in the base weight formula, i.e., a and b, which are defaulted to a = 0.6 and b = 0.4, but can also be configured by the user according to actual conditions; the file type distribution refers to the distribution of file types in the subtask, including fragmented small files (such as logs, code), large files (such as videos, databases), encrypted / compressed files, and regular files; the comparison weight = base weight * c; and the weight adjustment factor refers to the factor that adjusts the base weight, i.e., c. The weight adjustment factor is dynamically selected based on the mainstream file types in the subtask, not a simple weighted average.

[0055] Specifically, based on the formula "base weight = a * file size + b * number of files", the base weight of each subtask is calculated using its file size and number. This helps identify the file types within the subtask, confirm the distribution of file types, and thus determine the weight adjustment factor. For example, if there are many fragmented small files, a weight adjustment factor of 1.2 to 1.5 is suitable because more threads are needed to offset the overhead of metadata operations. If there are many large files, a weight adjustment factor of 0.7 to 0.9 is suitable because single-threaded sequential transmission is more efficient and avoids multi-threaded bandwidth contention. If there are many encrypted / compressed files, a weight adjustment factor of 0.8 is suitable to prevent CPU overload due to excessive threads. If there are many regular files, the base weight can be used, and the weight adjustment factor can be 1. Finally, the comparison weight of each subtask is determined using the formula "comparison weight = base weight * c". It should be noted that for subtasks involving small file clusters, a value greater than 1 can be used as a weight adjustment factor to increase the number of threads; for subtasks involving large files, a value less than 1 can be used as a weight adjustment factor to limit the number of threads. This approach enables precise scheduling of thread resources, ensuring reasonable resource allocation and improving resource utilization and comparison efficiency.

[0056] S203: Based on the comparison of the directory to be backed up and the target directory in the first thread, the difference comparison results are obtained.

[0057] In one embodiment, step S203 includes: obtaining preset difference comparison parameters; comparing the directory to be backed up and the target directory based on the difference comparison parameters and the first thread to obtain a difference file information stream; and parsing the difference file information stream to obtain a difference comparison result. Here, the difference comparison parameters refer to command-line comparison parameters, such as " / L (list only differences, no actual transmission)," and the copying tool in this application can be Robocopy; the difference file information stream refers to the real-time standard output stream generated by Robocopy when performing difference comparison, containing file difference information between the source directory and the target backup set (e.g., file operation type, file path, file size, modification time, progress indicator, etc.); the difference comparison result refers to the structured dataset (including file path, change type, file size, directory, etc.) parsed from the difference file information stream, used to guide the multi-threaded task allocation during the transmission phase.

[0058] like Figures 7-8As shown, users configure storage and backup methods through the user interface, thereby determining the storage location of the target directory on the target storage device, the number of backup copies, the command method, creation time, backup method, etc. The command method can be Robocopy. Specifically, a corresponding first thread is started for each subtask. The first thread, based on preset difference comparison parameters, calls Robocopy to compare the directory to be backed up and the target directory, thereby outputting a difference file information stream. Subsequently, the difference file information stream is asynchronously parsed to extract the newly added / modified file paths, obtaining structured difference records. These difference records are stored in memory or a temporary database to obtain the difference comparison results (e.g., a set of difference file paths) for use in subsequent transmission stages. This application's difference comparison method achieves asynchronous execution of comparison and transmission, solving the file transmission blocking problem and saving significant backup time compared to traditional comparison and transmission schemes.

[0059] S204: Determine each difference subtask based on the difference comparison results, and dynamically allocate a second thread from the second thread pool to each difference subtask.

[0060] In one embodiment, step S204 includes: determining each difference subtask and its transmission weight based on the difference comparison result; and allocating a second thread from the second thread pool to each difference subtask based on its transmission weight. The difference comparison result may be a set of difference file paths.

[0061] Specifically, the process iterates through each file path in the difference comparison results, calls the operating system API to read file attributes in batches, determines each difference subtask and its transmission weight, and then allocates a corresponding number of second threads from the second thread pool to the corresponding difference subtask based on the transmission weight. For example, if the transmission weight of a difference subtask accounts for 50% of the total transmission weight of all difference subtasks, then 50% of the second threads in the second thread pool are allocated to the corresponding difference subtask to execute the transmission of the difference files.

[0062] In this application, the transmission weights of each difference subtask can be sorted in descending order, and then the number of threads is allocated proportionally based on the total number of threads and the transmission weights. It should be noted that each difference subtask is allocated at least one thread. The number of threads is dynamically adjusted according to the file parameters, which reduces transmission time and improves transmission efficiency compared to a fixed-thread scheme.

[0063] In one embodiment, the step of determining each differential subtask and its transmission weight based on the difference comparison results includes: obtaining a second preset task partitioning condition; determining each differential subtask based on the difference comparison results and the second preset task partitioning condition; reading the attributes of each differential subtask to obtain its file parameters; and determining its transmission weight based on its file parameters. The second preset task partitioning condition refers to the conditions used to split the difference comparison results into differential subtasks. This condition can be configured by the user through a front-end interface or it can be a default configuration. For example, the second preset task partitioning condition can be a file quantity threshold, a file size threshold, or a subdirectory level threshold, etc. The file parameters of the differential subtasks can include the subtask's file size, file quantity, and file type, etc.

[0064] Specifically, the steps for dividing the difference comparison results into difference subtasks can be as follows: recursively scan the difference comparison results, and determine each difference subtask based on the difference comparison results and the second preset task division conditions. For example, recursively scan the entire directory according to the structure of the difference comparison result file tree. If the number of files in the directory is greater than the file number threshold, continue scanning the next level of subdirectories; if the already scanned directory level is less than the subdirectory level threshold, continue scanning the next level; repeat the scanning of subdirectories at each level until the corresponding difference subtasks are obtained. If the number of files in the directory is less than the file number threshold, aggregate the files in the directory into one difference subtask. If the scanned directory level is greater than the subdirectory level threshold, aggregate the files in the directory into one difference subtask. It should be noted that when there are a large number of small files in the directory, they can be divided according to the file number threshold (e.g., merging every 200 files into one subtask); when there are large files, they can be split according to the file size threshold (e.g., a single file > 50MB is considered an independent subtask); when the directory level is shallow, a single file can also be regarded as a difference subtask.

[0065] In another embodiment, the step of determining each difference subtask based on the difference comparison results and the second preset task division conditions can also include: analyzing the directory structure of the difference comparison results and dividing the difference subtasks according to a preset directory hierarchy; traversing the size of each file in the difference comparison results and dividing the difference subtasks according to a preset file size; traversing the type of each file in the difference comparison results and merging files of the same type into one difference subtask. Furthermore, files under the same subdirectory can also be merged into subtasks, thereby avoiding file fragmentation.

[0066] In one embodiment, the attributes of each differential subtask are read to obtain the file parameters of each differential subtask; based on the file parameters of each differential subtask, the transmission weight of each differential subtask is determined. Specifically, the basic weight of the differential subtask is calculated according to the formula "basic weight = x * file size + y * number of files", the file size and number of files of the differential subtask, where x and y are preset transmission coefficients. Since the transmission stage focuses more on file size, x can take a value of 0.7 and y can take a value of 0.3. Further, a transmission correction factor z can be determined according to the network quality during transmission, and the specific value of the transmission correction factor can be determined according to the formula "z = baseline bandwidth / current bandwidth". In addition, a type factor w can be determined according to the file type of each differential subtask; finally, the transmission weight of each differential subtask is determined according to the formula "transmission weight = (basic weight * w) / z". It should be noted that the type factor w can be determined according to the proportion of file type distribution.

[0067] In the above process, the transmission efficiency was maximized through three-layer dynamic control (i.e., basic weight, transmission correction factor and type factor), which solved the resource allocation problem in mixed file type scenarios.

[0068] S205: Based on the second thread, transmit each different subtask to the target storage end.

[0069] In one embodiment, after step S205, the method further includes: receiving file reception status data fed back by the target storage terminal; and generating a backup report based on the file reception status data. The file reception status data refers to the acknowledgment signal data from the target storage terminal for each transmission sub-task, used to ensure data integrity and transmission reliability. The file reception status data includes a sub-task representation, a transmission result flag, and a data checksum.

[0070] Specifically, when backing up files, the target storage end can generate file reception status data containing an MD5 checksum and the number of bytes received, and send it back to the source storage end via UDP protocol. The source storage end decodes the received file reception status data, extracts key fields, stores them according to the subtask ID index, and generates a backup report. For example... Figure 9 and Figure 10 As shown, users can construct differential data and then query differential backup task details through the front-end interface. The front-end interface provides a clear view of file backup status, such as the number of files backed up and backup progress.

[0071] Furthermore, if multiple backups fail consecutively, the corresponding difference subtask can be marked as a permanent error, and manual intervention should be requested. For example, ... Figure 11 and Figure 12As shown, users can perform recovery operations and view recovery details through the front-end interface. Recovery refers to the process of restoring data from a backup set to the source storage or a specified location. Its purpose is to restore the backup data on the target storage to a usable state, ensuring business continuity. The data foundation consists of difference files and the base backup set, and the complete version of the files can be reconstructed through inverse differential operations. Specifically, users determine the path to the base backup set and the sequence of difference files through the front-end interface, call Robocopy to asynchronously compare the differences between the current source storage and the target storage backup sets, obtain the list of files to be recovered, and dynamically allocate transmission threads for each subtask in the file recovery list to transfer the difference files from the target storage to the source storage. It should be noted that the recovery process uses the same methods as the backup process; the difference is that the recovery process is the reverse of the backup process.

[0072] As can be seen from the above embodiments, the file differential backup method of this application implements a complete file-level differential backup and recovery process based on Robocopy. It also achieves real-time output of backup and recovery progress based on file parameters during the backup and recovery process, and returns the backup and recovery result based on the subtask execution status at the end of the task. Furthermore, it has passed massive file testing and exhibits high stability. In addition, by executing the comparison and transmission steps separately with the first and second threads, the file transmission blocking problem is solved, saving a significant amount of backup time compared to traditional comparison and transmission schemes. During the comparison and transmission stages, task weights are calculated based on file parameters, and a specified number of threads are dynamically allocated to ensure reasonable resource allocation and improve resource utilization and comparison and transmission efficiency.

[0073] Based on the above embodiments, this application provides a file differential backup device. Specifically, please refer to... Figure 13 The device includes:

[0074] The first acquisition module 301 is used to acquire the directory to be backed up, the target directory of the target storage end, and the file parameters of each subtask in the directory to be backed up.

[0075] The first thread allocation module 302 is used to dynamically allocate the first thread in the first thread pool to each subtask according to the file parameters of each subtask.

[0076] The difference comparison module 303 is used to compare the directory to be backed up and the target directory based on the first thread to obtain the difference comparison results.

[0077] The second thread allocation module 304 is used to determine each difference subtask based on the difference comparison results and dynamically allocate the second thread of the second thread pool to each difference subtask.

[0078] The file transfer module 305 is used to transfer the different subtasks to the target storage end based on the second thread.

[0079] In one embodiment, the first acquisition module 301 includes:

[0080] The first acquisition submodule is used to acquire the first preset task division conditions, the directory to be backed up, and the target directory of the target storage end;

[0081] The task splitting module is used to split the directory to be backed up into independent subtasks according to the first preset task division conditions.

[0082] The parameter determination module is used to read the attributes of each subtask and obtain the file parameters of each subtask.

[0083] In one embodiment, the first thread allocation module 302 includes:

[0084] The comparison weight determination module is used to determine the comparison weight of each subtask based on the file parameters of each subtask.

[0085] The first allocation module is used to dynamically allocate the first thread in the first thread pool to each subtask based on the comparison weight.

[0086] In one embodiment, the comparison weight determination module includes:

[0087] The coefficient acquisition module is used to acquire preset coefficients;

[0088] The basic weight determination module is used to determine the basic weight based on the number of files, file size, and preset coefficients;

[0089] The correction factor determination module is used to determine the weight correction factor based on the file type distribution.

[0090] The weight correction module is used to determine the comparison weights of subtasks based on the weight correction factor and the base weights.

[0091] In one embodiment, the difference comparison module 303 includes:

[0092] The second acquisition module is used to acquire preset difference comparison parameters;

[0093] The directory comparison module is used to compare the directory to be backed up and the target directory based on the difference comparison parameters and the first thread to obtain the difference file information stream.

[0094] The information parsing module is used to parse the information stream of the difference file and obtain the difference comparison results.

[0095] In one embodiment, the second thread allocation module 304 includes:

[0096] The transmission weight determination module is used to determine the transmission weight of each difference subtask and each difference subtask based on the difference comparison results.

[0097] The second allocation module is used to allocate a second thread from the second thread pool to each differential subtask according to the transmission weight of each differential subtask.

[0098] In one embodiment, the transmission weight determination module includes:

[0099] The third acquisition module is used to acquire the second preset task division conditions;

[0100] The difference subtask determination module is used to determine each difference subtask based on the difference comparison results and the second preset task division conditions;

[0101] The difference file parameter determination module is used to read the attributes of each difference subtask and obtain the file parameters of each difference subtask.

[0102] The transmission weight determination submodule is used to determine the transmission weight of each differential subtask based on the file parameters of each differential subtask.

[0103] In one embodiment, the file differential backup device further includes:

[0104] The status data receiving module is used to receive file reception status data fed back by the target storage terminal;

[0105] The report generation module is used to generate backup reports based on the file reception status data.

[0106] Unlike current technologies, the file differential backup device provided in this application is equipped with a first thread allocation module, a difference comparison module, and a second thread allocation module. File comparison and transmission are performed asynchronously through these three modules. During the comparison and transmission phases, task weights are calculated based on file parameters, and a specified number of threads are dynamically allocated to each task. This ensures the rationality of resource allocation, improves resource utilization and comparison and transmission efficiency, and, moreover, the asynchronous execution of comparison and transmission solves the file transmission blocking problem, saving a significant amount of backup time compared to traditional comparison and transmission schemes.

[0107] Accordingly, embodiments of this application also provide an electronic device, such as... Figure 14As shown, the electronic device may include a processor 401 with one or more processing cores, a wireless (WiFi, Wireless Fidelity) module 402, a memory 403 with one or more computer-readable storage media, an audio circuit 404, a display unit 405, an input unit 406, a sensor 407, a power supply 408, and a radio frequency (RF) circuit 409, etc. Those skilled in the art will understand that... Figure 14 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0108] The processor 401 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 403, and by calling data stored in the memory 403, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. In one embodiment, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0109] WiFi is a short-range wireless transmission technology. Electronic devices, through the wireless module 402, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 14 The wireless module 402 is shown, but it is understood that it is not an essential component of the terminal and can be omitted as needed without changing the nature of the invention.

[0110] The memory 403 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 403. The memory 403 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal (such as audio data, phone book, etc.). In addition, the memory 403 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 403 may also include a memory controller to provide access to the memory 403 for the processor 401 and the input unit 406.

[0111] The audio circuit 404 includes a speaker, which provides an audio interface between the user and the electronic device. The audio circuit 404 can transmit the electrical signal converted from received audio data to the speaker, where the speaker converts it into a sound signal for output.

[0112] Display unit 405 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the terminal. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 405 may include a display panel, which in one embodiment can be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Furthermore, a touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to processor 401 to determine the type of touch event. Subsequently, processor 401 provides corresponding visual output on the display panel according to the type of touch event. Although in Figure 14 In this context, the touch-sensitive surface and the display panel are two separate components for implementing input and output functions. However, in some embodiments, the touch-sensitive surface and the display panel can be integrated to achieve input and output functions.

[0113] Input unit 406 can be used to receive input numerical or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in one embodiment, input unit 406 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, can collect user touch operations on or near it (e.g., user operations using fingers, styluses, or any suitable object or accessory on or near the touch-sensitive surface) and drive corresponding connection devices according to a pre-set program. In one embodiment, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the user's touch orientation and the signal generated by the touch operation, transmitting the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 401, and can receive and execute commands from the processor 401. Furthermore, various types of touch-sensitive surfaces, such as resistive, capacitive, infrared, and surface acoustic wave, can be used. In addition to the touch-sensitive surface, input unit 406 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0114] The electronic device may also include at least one sensor 407, such as a light sensor. Specifically, the light sensor may include an ambient light sensor and a distance sensor. The ambient light sensor can adjust the brightness of the display panel according to the brightness of the ambient light. Other sensors that may be configured in the electronic device, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0115] The electronic device also includes a power supply 408 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 408 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0116] The radio frequency (RF) circuit 409 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 401 for processing; additionally, it transmits uplink data to the base station. Typically, the RF circuit 409 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 409 can also communicate wirelessly with networks and other devices. Wireless communication can use any communication standard or protocol, including but not limited to GSM, GPRS, CDMA, WCDMA, LTE, email, and SMS.

[0117] Although not shown, the electronic device may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 403 according to the following instructions, and the processor 401 runs the applications stored in the memory 403 to achieve the following functions:

[0118] Get the directory to be backed up, the target directory, and the file parameters of each subtask in the directory to be backed up;

[0119] Based on the file parameters of each subtask, dynamically allocate the first thread in the first thread pool to each subtask;

[0120] Based on the comparison between the directory to be backed up and the target directory in the first thread, the difference comparison results are obtained;

[0121] Based on the difference comparison results, determine each difference subtask and dynamically allocate the second thread of the second thread pool to each difference subtask;

[0122] The second thread transmits the different subtasks to the target storage.

[0123] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0124] Therefore, embodiments of this application provide a computer-readable storage medium storing multiple instructions that can be loaded by a processor to implement the function of the above-described file differential backup method.

[0125] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0126] The file differential backup method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of differential backup of files, characterized by, The method comprises the following steps: obtaining a to-be-backed-up directory, a target directory of a target storage end and file parameters of each subtask in the to-be-backed-up directory; dynamically allocating a first thread in a first thread pool to each subtask according to the file parameters of each subtask, wherein the file parameters comprise the number of files, the size of files and the distribution of file types of the subtask, obtaining a preset coefficient, determining a basic weight according to the number of files, the size of files and the preset coefficient, determining a weight correction factor according to the distribution of file types, determining a comparison weight of the subtask according to the weight correction factor and the basic weight, and dynamically allocating a first thread in the first thread pool to each subtask according to the comparison weight; comparing the to-be-backed-up directory and the target directory based on the first thread to obtain a difference comparison result; determining each difference subtask according to the difference comparison result and dynamically allocating a second thread in a second thread pool to each difference subtask; transmitting each difference subtask to the target storage end based on the second thread, wherein the comparison and the transmission are executed asynchronously.

2. The file differential backup method according to claim 1, characterized by, The step of obtaining the to-be-backed-up directory, the target directory of the target storage end and the file parameters of each subtask in the to-be-backed-up directory comprises the following steps: obtaining a first preset task division condition, a to-be-backed-up directory and a target directory of a target storage end; splitting the to-be-backed-up directory into independent subtasks according to the first preset task division condition; reading the attributes of each subtask to obtain the file parameters of each subtask.

3. The method of claim 1, wherein, The step of comparing the to-be-backed-up directory and the target directory based on the first thread to obtain a difference comparison result comprises the following steps: obtaining a preset difference comparison parameter; comparing the to-be-backed-up directory and the target directory based on the difference comparison parameter and the first thread to obtain a difference file information stream; parsing the difference file information stream to obtain a difference comparison result.

4. The method of claim 1, wherein, The step of determining each difference subtask according to the difference comparison result and dynamically allocating a second thread in a second thread pool to each difference subtask comprises the following steps: determining each difference subtask and a transmission weight of each difference subtask according to the difference comparison result; allocating a second thread in the second thread pool to each difference subtask according to the transmission weight of each difference subtask.

5. The method of claim 4, wherein, The step of determining each difference subtask according to the difference comparison result and determining a transmission weight of each difference subtask comprises the following steps: obtaining a second preset task division condition; determining each difference subtask according to the difference comparison result and the second preset task division condition; reading the attributes of each difference subtask to obtain the file parameters of each difference subtask; determining the transmission weight of each difference subtask according to the file parameters of each difference subtask.

6. The method of claim 1, wherein, After the step of transmitting each difference subtask to the target storage end based on the second thread, the method further comprises the following steps: receiving file receiving state data fed back by the target storage end; generating a backup report according to the file receiving state data.

7. A file differential backup apparatus characterized by comprising: The method comprises the following steps: a first obtaining module is configured to obtain a to-be-backed-up directory, a target directory of a target storage end and file parameters of each subtask in the to-be-backed-up directory; The first thread allocation module is configured to dynamically allocate a first thread in a first thread pool to each sub-task according to file parameters of the sub-tasks, wherein the file parameters include a file quantity, a file size, and a file type distribution of the sub-tasks, obtain a preset coefficient, determine a basic weight according to the file quantity, the file size, and the preset coefficient, determine a weight correction factor according to the file type distribution, determine a comparison weight of the sub-task according to the weight correction factor and the basic weight, and dynamically allocate the first thread in the first thread pool to each sub-task according to the comparison weight. The difference comparison module is configured to compare the to-be-backed-up directory and the target directory based on the first thread to obtain a difference comparison result. The second thread allocation module is configured to determine each difference sub-task according to the difference comparison result and dynamically allocate a second thread in a second thread pool to each difference sub-task. The file transmission module is configured to transmit each difference sub-task to the target storage end based on the second thread, wherein the comparison and the transmission are executed asynchronously.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, which are adapted to be loaded by the processor to execute the steps in the file difference backup method of any one of claims 1 to 6.

Images