Data backup method, system and device and medium

Abstract

The invention discloses a data backup method, system and device and a medium, relates to the field of data processing, and is used for solving the problems of low data backup speed and insufficient resource utilization in a traditional data backup method. A source data disk is divided into a plurality of sub-data disks according to the total number of concurrent threads, and one thread is distributed for each sub-data disk for parallel processing, so that the computing power of a multi-core processor can be fully utilized, and the concurrency and the processing speed of backup operation are remarkably improved; by calculating the number of offset blocks of each sub-data disk, the offset positions of the data blocks on the source data disk and the target data disk are accurately controlled, the bandwidth bottleneck problem possibly occurring when a single thread is executed is avoided, the utilization rate of the disk bandwidth is effectively improved, and delay in the backup process is reduced; according to the method, the total backup time is shortened by distributing the multiple threads for parallel operation, and therefore the data migration task can be completed more efficiently and rapidly when the large-capacity data backup requirement is met.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data backup method, system, device and medium. Background Art

[0002] With the rapid development of science and technology and the advancement of the Internet of Everything, the generation of massive data requires a large amount of hard disk storage for backup and recovery. Conventional data backup operations such as copy commands in Windows systems (such as the cop command) and copy commands in Linux systems (such as the cp command and the scp command) are simple, but they have problems such as slow speed and scenario limitations. In particular, backup and recovery at the file system level cannot cover hard disk partition information and file system types.

[0003] To solve the above problems, the Linux system provides another data copy command (referred to as the dd command). The dd command can perform sequential read and write backup of hard disk data according to the specified block size and supports control of backup parameters. However, its single-thread execution speed is slow and fails to fully utilize the disk bandwidth. Summary of the invention

[0004] The purpose of the present invention is to provide a data backup method, system, device and medium, which avoids the bandwidth bottleneck problem that may occur when a single thread is executed, effectively improves the utilization rate of the disk bandwidth, and reduces the delay in the backup process; by allocating multiple threads to operate in parallel, the total backup time is reduced, so that when facing the demand for large-capacity data backup, the data migration task can be completed more efficiently and quickly.

[0005] In a first aspect, the present application provides a data backup method, comprising:

[0006] Obtain a first data capacity and a total number of concurrent threads of a source data disk, and divide the source data disk into N parts according to the total number of concurrent threads and the first data capacity to obtain N sub-data disks, where N is the total number of concurrent threads and is an integer greater than 1;

[0007] Calculating a second data capacity of each sub-data disk, and calculating a first number of first data blocks included in each sub-data disk according to the second data capacity and a first unit capacity, wherein the first unit capacity represents a size of the first data block;

[0008] Calculate the number of first offset blocks of each sub-data disk and the corresponding number of second offset blocks in the destination data disk according to the first number corresponding to all the sub-data disks;

[0009] Start N threads and allocate one sub-data disk to each thread, and each thread backs up the data in the source data disk to the destination data disk according to the first number, the first offset block number and the second offset block number corresponding to the sub-data disk allocated to itself.

[0010] Optionally, also include:

[0011] Acquire feedback information of each of the threads after backing up the data corresponding to the sub-data disk from the source data disk to the destination data disk;

[0012] It is determined whether there is erroneous data in each of the sub-data disks according to the feedback information.

[0013] Optionally, after determining whether there is erroneous data in each of the sub-data disks according to the feedback information, the method further includes:

[0014] When there is erroneous data in the sub-data disk, log information of the data backup process is obtained, and a target location of the erroneous data is determined according to the log information, where the target location is the location of the erroneous data in the source data disk or the destination data disk;

[0015] Clear the error data at the target location corresponding to the destination data disk.

[0016] Optionally, each of the first data blocks includes M second data blocks of second unit capacity, where M is an integer greater than 1;

[0017] Determining a target location of the error data according to the log information includes:

[0018] Determine the offset address of the second data block where the erroneous data is located according to the log information;

[0019] After clearing the error data at the target location corresponding to the destination data disk, the method further includes:

[0020] Calculate the remaining number of the second data blocks between the second data block where the erroneous data is located and the last second data block in the first data block where the erroneous data is located;

[0021] Determine whether the data backup of the first data block where the erroneous data is located is completed according to the remaining number.

[0022] Optionally, determining whether the data backup of the first data block where the erroneous data is located is completed according to the remaining number includes:

[0023] When the remaining number is 0, it is determined that the data backup of the first data block where the erroneous data is located is completed;

[0024] When the remaining number is not zero, it is determined that the data backup of the first data block where the erroneous data is located is not completed, and the remaining data in the first data block where the erroneous data is located is backed up from the source data disk to the destination data disk.

[0025] Optionally, backing up the remaining data in the first data block where the erroneous data is located from the source data disk to the destination data disk includes:

[0026] Determine the offset address of the next second data block after the offset address of the second data block where the erroneous data is located as the starting address of the backup;

[0027] Taking the second unit capacity as the unit, starting from the starting address, back up one second data block from the first data block where the erroneous data is located to the destination data disk at a time in offset order until the last second data block in the first data block where the erroneous data is located is backed up.

[0028] Optionally, after calculating the number of data blocks included in each of the sub-data disks according to the second data capacity and the first unit capacity, the method further includes:

[0029] Calculating a second number of second data blocks included in each of the first data blocks according to a first unit capacity and a second unit capacity, wherein the second unit capacity represents a size of the second data block;

[0030] Calculate the starting offset address of the first second data block and the ending offset address of the last second data block in each of the sub-data disks according to the total second quantity;

[0031] Determining a target location of the error data according to the log information includes:

[0032] Filtering out all the error data from the log information by searching keywords;

[0033] Determine the offset address of the second data block where each of the erroneous data is located;

[0034] According to the offset address of the second data block where the error data is located and the address range corresponding to each of the sub-data disks, all error data corresponding to each of the threads are determined; the address range corresponding to each of the sub-data disks is a range between the starting offset address of the first second data block in the sub-data disk and the ending offset address of the last second data block;

[0035] For all error data corresponding to each of the threads, the latest error data is screened out according to the timestamp, and the target location of the latest error data is determined.

[0036] In a second aspect, the present application provides a data backup system, comprising:

[0037] an acquisition unit, configured to acquire a first data capacity and a total number of concurrent threads of a source data disk, and divide the source data disk into N parts according to the total number of concurrent threads and the first data capacity to obtain N sub-data disks, where N is the total number of concurrent threads and is an integer greater than 1;

[0038] A first calculation unit is used to calculate a second data capacity of each sub-data disk, and calculate a first number of first data blocks included in each sub-data disk according to the second data capacity and a first unit capacity, wherein the first unit capacity represents a size of the first data block;

[0039] A second calculation unit, calculating the number of first offset blocks of each of the sub-data disks and the number of corresponding second offset blocks in the destination data disk according to the first number corresponding to all the sub-data disks;

[0040] The data backup unit starts N threads and allocates one sub-data disk to each thread. Each thread backs up the data in the source data disk to the destination data disk according to the first number, the first offset block number and the second offset block number corresponding to the sub-data disk allocated to it.

[0041] In a third aspect, the present application provides a data backup device, comprising:

[0042] Memory for storing computer programs;

[0043] The processor is used to implement the steps of the data backup method as described above when executing a computer program.

[0044] In a fourth aspect, the present application provides a non-volatile storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the steps of the data backup method as described above.

[0045] The present invention provides a data backup method, system, device and medium, which relates to the field of data processing and is used to solve the problems of slow data backup speed and insufficient resource utilization in traditional data backup methods. By dividing the source data disk into multiple sub-data disks according to the total number of concurrent threads, and assigning a thread to each sub-data disk for parallel processing, the computing power of the multi-core processor can be fully utilized, and the concurrency and processing speed of the backup operation can be significantly improved; by calculating the number of offset blocks of each sub-data disk, the offset position of the data block on the source data disk and the destination data disk is accurately controlled, avoiding the bandwidth bottleneck problem that may occur when a single thread is executed, effectively improving the utilization rate of the disk bandwidth, and reducing the delay in the backup process; by assigning multiple threads to operate in parallel, the total backup time is reduced, so that when facing the demand for large-capacity data backup, the data migration task can be completed more efficiently and quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A flow chart of a data backup method provided by the present invention;

[0048] Figure 2 A flowchart of a preparation process before data backup provided by the present invention;

[0049] Figure 3 A flowchart corresponding to a specific embodiment of a data backup method provided by the present invention;

[0050] Figure 4 A schematic diagram of a data backup system provided by the present invention. DETAILED DESCRIPTION

[0051] The core of the present invention is to provide a data backup method, system, device and medium, which avoids the bandwidth bottleneck problem that may occur when a single thread is executed, effectively improves the utilization rate of the disk bandwidth, and reduces the delay in the backup process; by allocating multiple threads to operate in parallel, the total backup time is reduced, so that when facing the demand for large-capacity data backup, the data migration task can be completed more efficiently and quickly.

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In a first aspect, the present application provides a data backup method, comprising:

[0054] S11: Obtain a first data capacity and a total number of concurrent threads of a source data disk, and divide the source data disk into N parts according to the total number of concurrent threads and the first data capacity to obtain N sub-data disks, where N is the total number of concurrent threads and is an integer greater than 1;

[0055] In this embodiment, during the data backup process, the total capacity of the source data disk needs to be divided according to the total number of concurrent threads. The capacity of the source data disk is considered as a whole. If it is allocated to multiple threads for concurrent backup, it is first necessary to clarify the number of data blocks that each thread is responsible for. Specifically, the first data capacity of the source data disk, that is, the total capacity of the source data disk (usually in bytes), is obtained. Then, the total number of concurrent threads (the number of backup threads to be started) is used to determine how to divide the total capacity of the source data disk into multiple sub-data disks. This division is based on the parallel execution capability of each thread. Therefore, the source data disk is divided into N sub-data disks, where N is the total number of concurrent threads, and N must be greater than 1, which means that at least two threads are working concurrently. The size of each sub-data disk is usually obtained by dividing the first data capacity by the total number of concurrent threads (N), thereby ensuring that the size of each sub-data disk is substantially equal, so that each thread can process a sub-data disk in parallel.

[0056] It should be understood that the division mentioned in this embodiment not only takes into account the physical capacity of the source data disk, but also makes reasonable allocation according to the processing power of each thread and the limitation of hardware resources, so as to improve the efficiency and concurrency of the backup process.

[0057] S12: calculating the second data capacity of each sub-data disk, and calculating the first number of first data blocks included in each sub-data disk according to the second data capacity and the first unit capacity, wherein the first unit capacity represents the size of the first data block;

[0058] In this step, it is necessary to calculate the number of data blocks of each sub-data disk based on the second data capacity of each sub-data disk. The second data capacity refers to the actual data capacity allocated to each sub-data disk, that is, the amount of data each sub-data disk is responsible for. According to the second data capacity of each sub-data disk divided in the above steps, the first unit capacity is introduced, and the content of each sub-data disk can be further subdivided into several smaller units (first data blocks). Specifically, the first unit capacity is usually a preset fixed value (such as 4M), which defines the actual size of each first data block. By dividing the second data capacity (that is, the capacity of each sub-data disk) by the first unit capacity, the first number of first data blocks contained in each sub-data disk can be obtained. The core of this calculation process is to divide the total capacity of each sub-data disk into several data blocks of equal size, so as to ensure that the amount of data of each data block is the same and facilitate subsequent parallel processing.

[0059] Calculate the number of first data blocks in each sub-data disk. This number reflects how many complete first data blocks can be accommodated in each sub-data disk, and thus provides a basic unit structure for subsequent data migration. In actual operation, this calculation depends not only on the actual capacity of the sub-data disk and the size of the data block, but also on the performance limitations of the hardware and system to ensure that the number of data blocks in each sub-data disk meets the concurrent processing capability of the system.

[0060] S13: Calculate the number of first offset blocks of each sub-data disk and the corresponding number of second offset blocks in the destination data disk according to the first number corresponding to all sub-data disks;

[0061] In this step, based on the number of data blocks of each sub-data disk calculated above, the offset of each sub-data disk in the source data disk and the destination data disk is further determined, that is, the relative position of each first data block in the source data disk and the destination data disk.

[0062] Specifically, the goal of this step is to calculate the number of offset blocks of the first data block in each sub-data disk and the corresponding position in the destination data disk, so as to achieve accurate data backup positioning. First, according to the first number of first data blocks contained in each sub-data disk obtained in the above steps, the number of offset blocks of each sub-data disk can be deduced. The number of offset blocks represents the offset from the starting position of the source data disk to the current first data block. For the source data disk, the number of offset blocks of each sub-data disk is calculated based on the data area allocated to it in the source data disk. Usually, the total capacity of the source data disk is divided into N sub-data disks according to the number of concurrent threads, so as to determine the starting offset block number (i.e., the first offset block number) and the ending offset block number of each sub-data disk. For the destination data disk, the number of offset blocks is mapped and calculated accordingly according to the allocation of the source data disk to ensure that the data of the source data disk can be accurately written to the corresponding position of the destination data disk.

[0063] By calculating the number of offset blocks of each sub-data disk, a clear starting and ending position is provided for the operation of each thread during the data migration process, thereby ensuring the accuracy and completeness of the data backup process and laying the foundation for subsequent parallel data migration.

[0064] S14: Start N threads and allocate a sub-data disk to each thread, and each thread backs up the data in the source data disk to the destination data disk according to the first number, first offset block number and second offset block number corresponding to the sub-data disk allocated to itself.

[0065] In this step, multiple threads are started in parallel, and each thread is responsible for backing up a portion of the data in the source data disk to the destination data disk (specifically, each thread corresponds to a sub-data disk, and each data disk backs up the data in the sub-data disk in sequence in units of the first data block).

[0066] Specifically, the source data disk is divided into N sub-data disks, and a sub-data disk is assigned to each thread to ensure that the amount of data processed by each thread is roughly the same. Each thread reads the first data block of the sub-data disk assigned to it, starts from the corresponding position of the source data disk, and migrates the data to the corresponding position of the destination data disk according to the offset information (i.e., the first offset block number and the second offset block number). The concurrent execution of threads enables the data of multiple sub-data disks to be backed up at the same time, greatly improving the backup efficiency and making full use of the computing power of multi-core processors. Specifically, the threads read and write data in sequence according to their assigned tasks, and accurately locate the source and destination positions of the data according to the first / second offset block number and the first number of the first data block calculated above, ensuring that the data of each sub-data disk is correctly backed up to the corresponding area in the destination data disk. Through this parallel backup method, the efficiency of large-scale data backup is achieved, and the speed of data migration is significantly improved.

[0067] like Figure 2 As shown, as an optional embodiment, the first data capacity of the source data disk and the total number of concurrent threads are obtained, and before the source data disk is divided into N parts according to the total number of concurrent threads and the first data capacity, it also includes: judging whether the server can correctly load the source data disk and the destination data disk, and if so, obtaining the drive letter of the source data disk and the drive letter of the destination data disk, and inputting the drive letter of the source data disk, the drive letter of the destination data disk, and the first data capacity of the source data disk and the total number of concurrent threads into the data backup and recovery tool according to the total number of concurrent threads and the first data capacity to trigger the execution of the data backup process until the data backup is completed.

[0068] In this embodiment, it is first necessary to verify whether the server can correctly load the source data disk and the destination data disk. This is to ensure that during the data backup process, both hard disk devices are in an available state and can read and write data normally. The loading process includes operating system detection and mounting the hard disk. If the hard disk works normally and is identified as a valid device, the drive letter information of the source data disk and the destination data disk will be obtained. Next, the system will obtain the first data capacity of the source data disk and the total number of concurrent threads entered by the user. These parameters will be used to calculate the data block size allocated to each thread so that the data blocks can be reasonably divided and processed in parallel during the backup process. By dividing the source data disk into N sub-data disks according to the total number of concurrent threads and data capacity, the efficiency of the data backup operation can be ensured. Finally, this information is input into the data backup and recovery tool, the data backup process is triggered, and the data backup task is started until the data of the source data disk is completely backed up to the destination data disk. In this process, the entire backup task is accelerated by concurrent threads, thereby maximizing the use of the server's computing and storage resources.

[0069] In summary, the data backup method provided by the present application can fully utilize the computing power of multi-core processors and significantly improve the concurrency and processing speed of backup operations by dividing the source data disk into multiple sub-data disks according to the total number of concurrent threads and assigning a thread to each sub-data disk for parallel processing; by calculating the number of offset blocks of each sub-data disk, the offset position of the data block on the source data disk and the destination data disk is accurately controlled, thereby avoiding the bandwidth bottleneck problem that may occur when a single thread is executed, effectively improving the utilization rate of the disk bandwidth, and reducing the delay in the backup process; by assigning multiple threads to operate in parallel, the total backup time is reduced, so that when facing large-capacity data backup needs, the data migration task can be completed more efficiently and quickly.

[0070] Based on the above embodiments:

[0071] As an optional embodiment, it also includes:

[0072] Obtain feedback information from each thread after backing up the data of the corresponding sub-data disk from the source data disk to the destination data disk;

[0073] Determine whether there is erroneous data in each sub-data disk based on the feedback information.

[0074] In this embodiment, the purpose is to check whether an error occurs in the data backup process by obtaining feedback information of the thread after each thread completes the data backup operation.

[0075] Specifically, after each thread successfully backs up the data of its corresponding sub-data disk from the source data disk to the destination data disk, it receives feedback information from the thread, which usually includes whether the backup is successful and whether an error has occurred. By analyzing this feedback information, it is possible to promptly find out which sub-data disks have data problems. If an error is reported during the backup process of a sub-data disk, the specific error data area can be located based on the feedback information, so that corresponding error handling measures (such as re-backup, data verification, etc.) can be taken.

[0076] This process effectively guarantees the integrity and accuracy of data backup and ensures that potential errors can be discovered and corrected during large-scale concurrent backup, thereby improving the reliability and fault tolerance of the backup system.

[0077] As an optional embodiment, after determining whether there is erroneous data in each sub-data disk according to the feedback information, the method further includes:

[0078] When there is erroneous data in the sub-data disk, the log information of the data backup process is obtained, and the target location of the erroneous data is determined according to the log information. The target location is the location of the erroneous data in the source data disk or the destination data disk;

[0079] Clear the error data at the corresponding target location in the destination data disk.

[0080] In this embodiment, when erroneous data is detected in a sub-data disk, the log information of the data backup process is first obtained. The log information usually records each step of the entire backup process, including each data read and write operation, possible abnormal conditions, and the specific location where the error occurred. By analyzing these logs, the "target location" of the erroneous data can be determined, that is, the specific location of the erroneous data in the source data disk or the destination data disk. Next, in order to ensure the data integrity in the destination data disk, the erroneous data at the target location is cleared, that is, the content of the corresponding storage block or data area is set to zero, indicating that the data has been marked as invalid or damaged.

[0081] Among them, the reasons for data errors may include but are not limited to the following: storage devices such as hard disks and SSDs may fail after long-term use, resulting in some data being unable to be correctly read or written. For example, bad sectors on the hard disk, damage to flash memory particles, aging of storage media, etc. may cause data read and write errors; during the backup process, data transmission between the storage device and the computer system may cause I / O (input and output) errors due to interface problems, insufficient bandwidth, network problems, etc., which may cause some data to fail to be successfully transferred from the source device to the destination device; the file system of the source data disk or the destination data disk may be damaged or inconsistent, resulting in some files or data blocks being unable to be correctly read or written. For example, sudden power failure of the disk, system crash, etc. may cause metadata of the file system to be damaged, thereby causing errors in the backup process; when using the network for backup (such as through SCP commands, etc.), network instability or interruption may cause data packet loss or transmission errors, especially in large-scale data migration, packet loss, delays and other problems may cause data inconsistency or damage.

[0082] This operation not only prevents the erroneous data from being misused or further copied, but also provides a clear starting point for subsequent repair or re-backup. In this way, erroneous data in the backup process can be effectively managed and repaired, ensuring the reliability and integrity of data backup.

[0083] As an optional embodiment, each first data block includes M second data blocks of second unit capacity, where M is an integer greater than 1;

[0084] Determine the target location of the error data based on the log information, including:

[0085] Determine the offset address of the second data block where the error data is located according to the log information;

[0086] After clearing the error data at the corresponding target location in the target data disk, the method further includes:

[0087] Calculate the remaining number of second data blocks between the second data block where the erroneous data is located and the last second data block in the first data block where the erroneous data is located;

[0088] It is determined whether the data backup of the first data block where the erroneous data is located is completed according to the remaining number.

[0089] In this embodiment, it is first assumed that each first data block contains multiple smaller second data blocks (i.e., unit data blocks). Each first data block contains M second data blocks, where M is an integer greater than 1. The second data block is a basic unit of data storage, and each second data block has a specific capacity (i.e., a second unit capacity, such as 4k). During the backup process, data is migrated from the source data disk to the destination data disk, and data integrity verification is performed. When an error occurs in a data block, the location of the specific erroneous data is located according to the log information.

[0090] The target location of the error data refers to the location of the error data in the source data disk or the destination data disk. The log information provides detailed information about the backup process, especially the specific location related to the error data (usually stored in the form of the offset address of the data block). By analyzing the log information, the offset address of the second data block where the error data is located, that is, the storage location of the second data block, is determined.

[0091] Once the location of the erroneous data is determined, the erroneous data at the target location is cleared to prevent the erroneous data from interfering with subsequent operations. Then, the number of remaining second data blocks between the second data block where the erroneous data is located and the last second data block in the first data block where the erroneous data is located is calculated. Here, the remaining number represents the number of all data blocks between the second data block where the erroneous data is located and the last second data block of the first data block where the data block is located.

[0092] By calculating the number of remaining second data blocks, it is possible to determine whether the data backup of the first data block where the erroneous data is located is completed.

[0093] As an optional embodiment, determining whether the data backup of the first data block where the erroneous data is located is completed according to the remaining number includes:

[0094] When the remaining number is 0, it is determined that the data backup of the first data block where the erroneous data is located is completed;

[0095] When the remaining number is not zero, it is determined that the data backup of the first data block where the erroneous data is located is not completed, and the remaining data in the first data block where the erroneous data is located is backed up from the source data disk to the destination data disk.

[0096] Specifically, if the number of remaining second data blocks is 0, it means that the data backup of the first data block where the erroneous data is located has been completed, because all the data in the data block has been correctly migrated and backed up to the destination data disk. If the number of remaining second data blocks is not 0, it means that the backup of the first data block where the erroneous data is located has not been completed, and the remaining data needs to be backed up from the source data disk to the destination data disk in sequence until all the data in the data block is successfully backed up.

[0097] It can be seen that this embodiment ensures the integrity and reliability of data backup by tracking the progress of data backup and locating erroneous data block by block, thereby preventing erroneous data from having a long-term impact on the backup result.

[0098] As an optional embodiment, backing up the remaining data in the first data block where the erroneous data is located from the source data disk to the destination data disk includes:

[0099] Determine the offset address of the next second data block after the offset address of the second data block where the erroneous data is located as the starting address of the backup;

[0100] Taking the second unit capacity as a unit, starting from the starting address, back up one second data block from the first data block where the erroneous data is located to the destination data disk in offset order each time until the last second data block in the first data block where the erroneous data is located is backed up.

[0101] In this embodiment, when erroneous data is detected and the first data block where the erroneous data is located is located, in order to ensure the integrity of the data backup, it is necessary to continue to back up the remaining data in the data block from the source data disk to the destination data disk step by step starting from the location of the erroneous data.

[0102] First, determine the offset address of the second data block where the erroneous data is located. Based on the offset address, calculate the offset address of the next second data block, that is, the next position of the second data block where the erroneous data is located. This offset address will be used as the starting address of the data backup. In this way, the backup process starts from the next position of the data block where the erroneous data is located, ensuring that the correct backup starts from the data after the erroneous data.

[0103] The specific process of backing up the remaining data is as follows: taking the second unit capacity as the unit, back up the data blocks one by one in the offset order according to the starting address. Each time the backup is performed, the second data block corresponding to the current offset address in the source data disk is read and written to the corresponding position of the destination data disk. This process continues until all the second data blocks in the first data block where the erroneous data is located are backed up.

[0104] This block-by-block, sequential backup method can ensure the integrity of data backup, especially when erroneous data appears, it can start from the next location of the erroneous data to ensure smooth data backup and repair any missing or erroneous parts. This method effectively avoids the problem of partial data loss, while ensuring the continuity and accuracy of data backup.

[0105] As an optional embodiment, after calculating the first number of first data blocks included in each sub-data disk according to the second data capacity and the first unit capacity, it also includes: calculating the second number of second data blocks included in each first data block according to the first unit capacity and the second unit capacity, the second unit capacity representing the size of the second data block; calculating the starting offset address of the first second data block and the ending offset address of the last second data block in each sub-data disk according to the total second number; determining the target position of the erroneous data according to the log information, including: filtering out all erroneous data from the log information by searching keywords; determining the offset address of the second data block where each erroneous data is located; determining all erroneous data corresponding to each thread according to the offset address of the second data block where the erroneous data is located and the address range corresponding to each sub-data disk; the address range corresponding to each sub-data disk is the range between the starting offset address of the first second data block in the sub-data disk and the ending offset address of the last second data block; for all erroneous data corresponding to each thread, filtering out the latest erroneous data according to the timestamp, and determining the target position of the latest erroneous data.

[0106] The core purpose of this embodiment is to ensure the integrity and consistency of data by locating, screening and repairing erroneous data in the data backup process. Specifically, the number of second data blocks contained in each first data block is calculated based on the first unit capacity and the second unit capacity. The second unit capacity is used to define the size of the second data block (such as 4k). According to the relationship between the first unit capacity and the second unit capacity, it is determined how many second data blocks can be accommodated in each first data block, which is convenient for subsequent data processing and positioning.

[0107] Based on the number of second data blocks in all sub-data disks, the starting offset address of the first second data block and the ending offset address of the last second data block in each sub-data disk are further calculated. These offset addresses define the storage range of data in each sub-data disk and provide an accurate reference for subsequent erroneous data location.

[0108] When a data error occurs, the log information is retrieved first, and all possible error data records are filtered out from the log by using specific keywords. This allows you to quickly determine which data has errors during the backup process.

[0109] For each piece of erroneous data, its offset address in the second data block is determined, and the specific location of the erroneous data in the storage is further calculated based on the offset address. At this time, it is necessary to refer to the address range of each sub-data disk, that is, the starting offset address of the first second data block and the ending offset address of the last second data block in the sub-data disk. This range helps the system determine the data block that each thread is responsible for, so as to find the specific erroneous data.

[0110] Among all the error data, the latest error data will be filtered out according to the timestamp in the log. Through this step, the latest error can be processed all the time to avoid repeated operations on earlier errors. After determining the target location of the latest error data, the error data can be cleared or reset to zero.

[0111] In summary, this embodiment can efficiently and accurately identify and process erroneous data by accurately calculating and locating the offset address of each data block and combining the screening of log information and timestamps. This method can not only process single erroneous data, but also process a large number of errors distributed in different sub-data disks and data blocks, ensuring the integrity and efficiency of the data backup process.

[0112] like Figure 3 As shown, in a specific embodiment, the implementation process is as follows:

[0113] 1. Receive input parameters: receive the source data disk letter (if), the destination data disk letter (of), the total number of concurrent threads (N), and the current number of concurrent threads (T, which is the thread number).

[0114] 2. Obtain the first data capacity of the source data disk and calculate related parameters: Use the lsblk -b command to obtain the first data capacity of the source data disk.

[0115] The first number (tcount) of the first data blocks of each concurrent thread is calculated according to the total number of concurrent threads (N), that is, the number of 4M data blocks that each thread is responsible for.

[0116] Calculate the offsets of the current concurrent threads, which are:

[0117] skip: 4M data block offset of the source data disk, seek: 4M data block offset of the destination data disk, start_4k and end_4k: the starting offset and ending offset of the 4k data block in the source data disk, which are used for subsequent error positioning.

[0118] 3. Data backup and recovery: Set the dd command parameters: use if, of, skip, seek, bs (set to 4M), count (set to tcount) to perform data backup operations (where bs is the number of bytes read or written at a single time).

[0119] After executing the dd command, determine the return value:

[0120] The return value is 0: it means that the data backup of the current thread has been completed successfully and jump to the next step.

[0121] If the return value is not 0, it means that an error occurred during the data backup process and the error handling logic needs to be entered.

[0122] 4. Error handling: When dd returns non-zero, it indicates that an error occurred during the data backup process: Get the error position A (in 4K units) from the system dmesg log. The error position A is within the range of start_4k and end_4k.

[0123] Use the dd command to clear the data at the error location A on the destination data disk.

[0124] 5. Restore the data at the error location: Calculate the location of the 4M data block B where the error location A is located, and determine the number C of 4K data remaining in the block that has not been backed up.

[0125] If C is 0, it means that the backup of the 4M data block B has been completed:

[0126] Determine whether all data blocks have been backed up. If the current 4M block B is the last data block, it means that the current concurrent data backup is completed.

[0127] Otherwise, skip to step 3 to continue with the subsequent data backup.

[0128] If C is greater than 0, it means that there is still unbacked up data in the current 4M block B:

[0129] Use the dd command to continue backing up the remaining 4K data in the 4M block B. Set skip and seek to A+1. At this time, bs is set to 4K and count is set to C.

[0130] 6. Check the data recovery completion: Check the return value of the dd command:

[0131] The return value is 0: it means that the remaining data in the current 4M block B has been backed up and the next data block will be processed.

[0132] If the return value is not 0, it means that an error occurred during the recovery process and the new error location A1 needs to be obtained from the dmesg log for repair. At this time, set bs to 4K, count to 1, and recover again.

[0133] 7. Loop execution: Through the logic of steps 3-6, data backup and recovery are continuously performed until all data blocks are successfully backed up.

[0134] Each time the command is executed, the skip, seek, bs, and count parameters are updated to ensure that the data is backed up accurately until all the data in the Nth segment of the source data disk is successfully backed up to the corresponding position of the destination data disk.

[0135] 8. Complete all data backup: When all data backup is completed, confirm that the backup operation has been completed.

[0136] It can be seen that by executing concurrent data backup tasks and combining error detection and repair mechanisms, various abnormal situations can be handled during the data backup process. Each concurrent thread is responsible for a portion of the data on the source data disk. Through reasonable offset calculation and log error tracking mechanisms, the accuracy and integrity of the data can be ensured. In addition, using the return value of the dd command and system log information, errors that occur during the backup process can be discovered and resolved in a timely manner, ensuring the smooth completion of the data backup task to the greatest extent.

[0137] Second, as Figure 4 As shown, the present application provides a data backup system, including:

[0138] The acquisition unit 41 is used to acquire a first data capacity and a total number of concurrent threads of the source data disk, and divide the source data disk into N parts according to the total number of concurrent threads and the first data capacity to obtain N sub-data disks, where N is the total number of concurrent threads and is an integer greater than 1;

[0139] A first calculation unit 42 is used to calculate a second data capacity of each sub-data disk, and calculate a first number of first data blocks included in each sub-data disk according to the second data capacity and a first unit capacity, wherein the first unit capacity represents a size of the first data block;

[0140] The second calculation unit 43 calculates the number of first offset blocks of each sub-data disk and the corresponding number of second offset blocks in the destination data disk according to the first number corresponding to all sub-data disks;

[0141] The data backup unit 44 starts N threads and allocates a sub-data disk to each thread. Each thread backs up the data in the source data disk to the destination data disk according to the first number, first offset block number and second offset block number corresponding to the sub-data disk allocated to itself.

[0142] For further introduction to the data backup system, please refer to the above embodiments, which will not be repeated in this application.

[0143] In a third aspect, the present application provides a data backup device, comprising:

[0144] Memory for storing computer programs;

[0145] The processor is used to implement the steps of the data backup method as described above when executing a computer program.

[0146] For other introductions to the data backup device, please refer to the above embodiments, which will not be described in detail in this application.

[0147] In a fourth aspect, the present application provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data backup method described above are implemented. For other introductions to the non-volatile storage medium, please refer to the above embodiments, and the present application will not repeat them here.

[0148] On the other hand, the present application also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data backup method when the computer program / instruction is executed by a processor. For other introductions to the computer program product, please refer to the above-mentioned embodiments, and the present application will not repeat them here.

[0149] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0150] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data backup method, characterized in that: include: Obtain a first data capacity and a total number of concurrent threads of a source data disk, and divide the source data disk into N parts according to the total number of concurrent threads and the first data capacity to obtain N sub-data disks, where N is the total number of concurrent threads and is an integer greater than 1; Calculating a second data capacity of each sub-data disk, and calculating a first number of first data blocks included in each sub-data disk according to the second data capacity and a first unit capacity, wherein the first unit capacity represents a size of the first data block; Calculate the number of first offset blocks of each sub-data disk and the corresponding number of second offset blocks in the destination data disk according to the first number corresponding to all the sub-data disks; Start N threads and allocate one sub-data disk to each thread, and each thread backs up the data in the source data disk to the destination data disk according to the first number, the first offset block number and the second offset block number corresponding to the sub-data disk allocated to itself.

2. The data backup method according to claim 1, characterized in that: Also includes: Acquire feedback information of each of the threads after backing up the data corresponding to the sub-data disk from the source data disk to the destination data disk; It is determined whether there is erroneous data in each of the sub-data disks according to the feedback information.

3. The data backup method according to claim 2, characterized in that: After determining whether there is erroneous data in each of the sub-data disks according to the feedback information, the method further includes: When there is erroneous data in the sub-data disk, log information of the data backup process is obtained, and a target location of the erroneous data is determined according to the log information, where the target location is the location of the erroneous data in the source data disk or the destination data disk; Clear the error data at the target location corresponding to the destination data disk.

4. The data backup method according to claim 3, characterized in that: Each of the first data blocks includes M second data blocks of second unit capacity, where M is an integer greater than 1; Determining a target location of the error data according to the log information includes: Determine the offset address of the second data block where the erroneous data is located according to the log information; After clearing the error data at the target location corresponding to the destination data disk, the method further includes: Calculate the remaining number of the second data blocks between the second data block where the erroneous data is located and the last second data block in the first data block where the erroneous data is located; Determine whether the data backup of the first data block where the erroneous data is located is completed according to the remaining number.

5. The data backup method according to claim 4, characterized in that: Determining whether the data backup of the first data block where the erroneous data is located is completed according to the remaining number includes: When the remaining number is 0, it is determined that the data backup of the first data block where the erroneous data is located is completed; When the remaining number is not zero, it is determined that the data backup of the first data block where the erroneous data is located is not completed, and the remaining data in the first data block where the erroneous data is located is backed up from the source data disk to the destination data disk.

6. The data backup method according to claim 5, characterized in that: Backing up the remaining data in the first data block where the erroneous data is located from the source data disk to the destination data disk, comprising: Determine the offset address of the next second data block after the offset address of the second data block where the erroneous data is located as the starting address of the backup; Taking the second unit capacity as the unit, starting from the starting address, back up one second data block from the first data block where the erroneous data is located to the destination data disk at a time in offset order until the last second data block in the first data block where the erroneous data is located is backed up.

7. The data backup method according to any one of claims 3 to 6, characterized in that: After calculating the number of data blocks included in each of the sub-data disks according to the second data capacity and the first unit capacity, the method further includes: Calculating a second number of second data blocks included in each of the first data blocks according to a first unit capacity and a second unit capacity, wherein the second unit capacity represents a size of the second data block; Calculate the starting offset address of the first second data block and the ending offset address of the last second data block in each of the sub-data disks according to the total second quantity; Determining a target location of the error data according to the log information includes: Filtering out all the error data from the log information by searching keywords; Determine the offset address of the second data block where each of the erroneous data is located; According to the offset address of the second data block where the error data is located and the address range corresponding to each of the sub-data disks, all error data corresponding to each of the threads are determined; the address range corresponding to each of the sub-data disks is a range between the starting offset address of the first second data block in the sub-data disk and the ending offset address of the last second data block; For all error data corresponding to each of the threads, the latest error data is screened out according to the timestamp, and the target location of the latest error data is determined.

8. A data backup system, characterized in that: include: an acquisition unit, configured to acquire a first data capacity and a total number of concurrent threads of a source data disk, and divide the source data disk into N parts according to the total number of concurrent threads and the first data capacity to obtain N sub-data disks, where N is the total number of concurrent threads and is an integer greater than 1; A first calculation unit is used to calculate a second data capacity of each sub-data disk, and calculate a first number of first data blocks included in each sub-data disk according to the second data capacity and a first unit capacity, wherein the first unit capacity represents a size of the first data block; A second calculation unit, calculating the number of first offset blocks of each of the sub-data disks and the number of corresponding second offset blocks in the destination data disk according to the first number corresponding to all the sub-data disks; The data backup unit starts N threads and allocates one sub-data disk to each thread. Each thread backs up the data in the source data disk to the destination data disk according to the first number, the first offset block number and the second offset block number corresponding to the sub-data disk allocated to it.

9. A data backup device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the data backup method according to any one of claims 1 to 7 when executing a computer program.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data backup method according to any one of claims 1 to 7 are implemented.

Images