Cloud hard disk data backup and recovery method and device, electronic equipment and storage medium

Abstract

The invention discloses a cloud hard disk data backup and recovery method and device, electronic equipment and a storage medium, and relates to the technical field of cloud computing. A cloud hard disk is segmented into a plurality of segmentation blocks and a multi-thread parallel processing mode is adopted, a backup and recovery task of a large-capacity cloud hard disk is decomposed into a plurality of sub-tasks to be processed at the same time, and the data backup and recovery efficiency is improved. The operation time is greatly shortened, the number of threads in the thread pool is determined according to the number of the segmentation blocks and the current available computing resources, the utilization rate of the computing resources can be improved, cloud hard disk segmentation strategy parameters can be flexibly configured according to service requirements and cloud hard disk characteristics, data protection requirements in different scenes are met, and the user experience is improved. Consistency verification is carried out on the transmitted data, so that the integrity and consistency of the data in the backup and recovery process are ensured, the data loss risk is reduced, and the purposes of improving the backup and recovery efficiency of the cloud hard disk and enhancing the system reliability and the computing resource utilization rate are achieved.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a cloud hard drive data backup and recovery method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of cloud computing, more and more enterprises are placing their business operations and systems on cloud platforms. Cloud platforms can quickly build development environments and allocate computing resources according to the needs of different users, offering advantages such as elasticity, speed, and on-demand availability. Furthermore, with the widespread application of cloud computing technology, OpenStack, as a typical representative of open-source cloud computing platforms, is used by numerous enterprises and institutions to build private or hybrid cloud environments. In OpenStack environments, the backup and recovery of virtual machine disk data is a crucial step in ensuring business continuity and data security. Traditional cloud disk backup and recovery methods typically employ a single-threaded sequential processing approach. For large-capacity cloud disks, this approach suffers from excessively long backup and recovery times, severely impacting the availability of business systems. Simultaneously, a single backup strategy is insufficient to adapt to the differentiated data protection needs of various business scenarios. For example, some businesses require rapid backups, while others prioritize the integrity and consistency of backup data. Moreover, the lack of effective resource scheduling and error handling mechanisms during backup and recovery leads to low system resource utilization and makes it difficult to quickly locate and resolve problems when failures occur. Summary of the Invention

[0003] This application provides a cloud hard drive data backup and recovery method, apparatus, electronic device, and storage medium to at least solve the problem of low efficiency in cloud hard drive backup and recovery in related technologies.

[0004] This application provides a method for data backup and recovery on a cloud hard drive, including:

[0005] Configure cloud disk partitioning strategy parameters for backup or recovery;

[0006] In response to selecting to back up or restore cloud disk data, the partition on the cloud disk is divided into multiple partition blocks according to the cloud disk partitioning strategy parameters;

[0007] The number of threads in the thread pool is determined based on the number of the partitioned blocks and the currently available computing resources, and the data of each partitioned block is assigned to a thread for processing.

[0008] In response to the thread starting, a temporary virtual cloud disk is created, the temporary virtual cloud disk is mounted on the host machine, the asynchronous input / output library is used to perform multi-threaded data transmission operations using a memory buffer, and the transmitted data is checked for consistency. The transmitted data that passes the check is backed up or restored.

[0009] This application also provides a cloud disk data backup and recovery device, including:

[0010] The partitioning strategy configuration module is used to set the partitioning strategy parameters for the cloud disks to be backed up or restored.

[0011] The partitioning module is used to divide the partition on the cloud disk into multiple partition blocks according to the cloud disk partitioning strategy parameters when selecting to back up or restore cloud disk data.

[0012] The multi-threaded task allocation module is used to determine the number of threads in the thread pool based on the number of the partitioned blocks and the currently available computing resources, and to allocate the data of each partitioned block to a thread for processing.

[0013] The data transmission control module is used to respond to the thread startup by creating a temporary virtual cloud disk, mounting the temporary virtual cloud disk to the host machine, using an asynchronous input / output library to perform multi-threaded data transmission operations using a memory buffer, performing consistency checks on the transmitted data, and backing up or restoring the transmitted data that passes the checks.

[0014] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described cloud hard drive data backup and recovery methods.

[0015] Configure cloud disk partitioning strategy parameters for backup or recovery;

[0016] In response to selecting to back up or restore cloud disk data, the partition on the cloud disk is divided into multiple partition blocks according to the cloud disk partitioning strategy parameters;

[0017] The number of threads in the thread pool is determined based on the number of the partitioned blocks and the currently available computing resources, and the data of each partitioned block is assigned to a thread for processing.

[0018] In response to the thread starting, a temporary virtual cloud disk is created, the temporary virtual cloud disk is mounted on the host machine, the asynchronous input / output library is used to perform multi-threaded data transmission operations using a memory buffer, and the transmitted data is checked for consistency. The transmitted data that passes the check is backed up or restored.

[0019] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described cloud hard drive data backup and recovery methods:

[0020] Configure cloud disk partitioning strategy parameters for backup or recovery;

[0021] In response to selecting to back up or restore cloud disk data, the partition on the cloud disk is divided into multiple partition blocks according to the cloud disk partitioning strategy parameters;

[0022] The number of threads in the thread pool is determined based on the number of the partitioned blocks and the currently available computing resources, and the data of each partitioned block is assigned to a thread for processing.

[0023] In response to the thread starting, a temporary virtual cloud disk is created, the temporary virtual cloud disk is mounted on the host machine, the asynchronous input / output library is used to perform multi-threaded data transmission operations using a memory buffer, and the transmitted data is checked for consistency. The transmitted data that passes the check is backed up or restored.

[0024] This application employs a multi-segmentation and multi-threaded parallel processing approach for cloud disks, decomposing the backup and recovery tasks of large-capacity cloud disks into multiple sub-tasks for simultaneous processing. This significantly shortens the operation time. Determining the number of threads in the thread pool based on the number of segments and currently available computing resources improves the utilization of computing resources. The cloud disk segmentation strategy parameters can be flexibly configured according to business needs and cloud disk characteristics to meet data protection requirements in different scenarios. Consistency verification of transmitted data ensures the integrity and consistency of data during backup and recovery, reducing the risk of data loss. This achieves the goals of improving cloud disk backup and recovery efficiency, enhancing system reliability, and increasing the utilization of computing resources. Attached Figure Description

[0025] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is an application environment diagram of the cloud hard drive data backup and recovery method in one embodiment of this application;

[0027] Figure 2 This is a schematic diagram illustrating the operation of a cloud hard drive data backup and recovery method in one embodiment of this application;

[0028] Figure 3 This is a flowchart illustrating a cloud hard drive data backup and recovery method in one embodiment of this application;

[0029] Figure 4 This is a schematic diagram of the process of performing cloud disk data backup in one embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the cloud disk data recovery process in one embodiment of this application;

[0031] Figure 6 This is a structural block diagram of a cloud hard drive data backup and recovery device in one embodiment of this application;

[0032] Figure 7 This is an internal structural diagram of a computer device in one embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0034] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0035] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] The cloud disk data backup and recovery method provided in this application can be applied to, for example... Figure 1 The application environment shown is illustrated. Terminal 102 communicates with server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 104 can be a standalone server or a server cluster consisting of multiple servers. Terminal 102 can be used to back up and restore data on the cloud disks of server 104. An OpenStack platform is built on server 104, using the latest stable version (such as the Train version), and core components such as Nova compute service, Cinder block storage service, and Swift object storage service are installed and configured. Hardware resources for compute nodes and storage nodes are configured on server 104. Compute nodes are equipped with high-performance CPUs (such as Intel Xeon Platinum series), large-capacity memory (such as 128GB or more), and high-speed storage devices (such as SSD cloud disks); storage nodes are configured with large-capacity cloud disk arrays to meet backup data storage requirements.

[0037] like Figure 2 As shown, when users can perform backup and restore operations on the cloud disk data of server 104 through terminal 102, users log in to the OpenStack Horizon management interface and select the virtual machine cloud disk to be backed up or restored. In the cloud disk operation options, enter the cloud disk partitioning strategy configuration page. Users can choose to partition according to the cloud disk capacity ratio, such as dividing the cloud disk into 4 equal parts; they can also choose to partition according to the file system partition, and the system will automatically identify the partition situation on the cloud disk and perform partitioning; they can also choose to customize the byte number partitioning and enter the byte size of each partition block (such as 1GB). Users can choose the backup method, including incremental backup, full backup, and differential backup. They can also choose the number of nodes during backup. After clicking the save button, the system stores the partitioning strategy parameters in the OpenStack database (such as MySQL). The database table structure design includes fields such as cloud disk ID, partitioning strategy type, number of partition blocks, starting address and size of each partition block. Server 104 performs multi-threaded task allocation and obtains resource information such as the number of CPU cores and memory size of the compute nodes through OpenStack's Nova API. Based on the number of cloud disk partitions and computing resources, the thread pool size is determined using the formula: Thread pool size = min(Number of partitions, Number of available CPU cores * 2). For example, if the cloud disk is partitioned into 8 blocks and the system has 4 available CPU cores, the thread pool size is 8 (the smaller value). A thread pool is created using Python's thread pool module (such as `concurrent.futures.ThreadPoolExecutor`), and backup or restore tasks for each cloud disk partition are assigned to threads within the thread pool. A task identifier is set for each task, generated using a UUID (Universally Unique Identifier) ​​to ensure uniqueness.

[0038] like Figure 3 As shown, an embodiment of this application provides a cloud disk data backup and recovery method, including the following steps:

[0039] Step S1: Set the cloud disk partitioning strategy parameters for the cloud disk to be backed up or restored;

[0040] Step S2: In response to selecting to back up or restore cloud disk data, the partition on the cloud disk is divided into multiple partition blocks according to the cloud disk partitioning strategy parameters;

[0041] Step S3: Determine the number of threads in the thread pool based on the number of partitioned blocks and the currently available computing resources, and assign the data of each partitioned block to a thread for processing;

[0042] Step S4: In response to thread startup, a temporary virtual cloud disk is created, mounted on the host machine, and an asynchronous input / output library is used to perform multi-threaded data transmission operations using a memory buffer. The transmitted data is then checked for consistency, and the verified transmitted data is backed up or restored.

[0043] In this embodiment, setting the cloud disk partitioning strategy parameters includes:

[0044] The partitioning strategy type in the cloud disk partitioning strategy parameters includes partitioning the cloud disk equally according to the cloud disk capacity ratio, partitioning according to the existing file system partitions on the cloud disk, or partitioning according to the number of bytes in each partition block.

[0045] The cloud disk partitioning strategy parameters are stored in a database table structure. The database table structure contains fields such as cloud disk ID, partitioning strategy type, number of partition blocks, starting address and size of each partition block.

[0046] The cloud disk partitioning strategy configuration function includes: users can set cloud disk partitioning strategies for cloud disks to be backed up or restored through the OpenStack management interface or API. Partitioning strategies include, but are not limited to, partitioning by cloud disk capacity ratio (e.g., dividing the cloud disk into 2 or 4 equal parts), partitioning by file system partition (partitioning based on existing partitions on the cloud disk), and custom byte size partitioning (user-specified byte size for each partition). Based on the user-set partitioning strategy, the specific parameters for cloud disk partitioning are calculated, such as the number of partitions, the starting address and size of each partition, etc., and these parameters are stored in the OpenStack database for subsequent operations.

[0047] In this embodiment, determining the number of threads in the thread pool based on the number of partitioned blocks and currently available computing resources, and allocating the data of each partitioned block to a thread for processing includes:

[0048] Obtain cloud disk partitioning parameters, including the number of partitioned blocks, the starting address of each partitioned block, and its size;

[0049] Obtain currently available computing resources, including the number of available processor cores and memory size;

[0050] The number of threads in the thread pool is determined by the formula: Number of threads = min(Number of blocks, Number of available processor cores * 2).

[0051] Create a task for each thread to perform backup or restore operations on the data in the segmented blocks. Set the address at which each task reads data based on the starting address of each segmented block, and assign each task to it.

[0052] The multi-threaded task allocation function includes: After obtaining the cloud disk partitioning parameters, the system determines the thread pool size based on the number of partitioned blocks and the system's currently available computing resources (such as the number of CPU cores and memory size). The formula for calculating the thread pool size for each node is: Number of threads in the thread pool = min(Number of partitioned blocks, Number of available processor cores * 2). Setting the number of threads in the thread pool considers both the number of partitioned blocks and the number of available processor cores; setting the thread number based on the number of processor cores improves the utilization of available computing resources. Each cloud disk partitioned block is assigned to one thread in the thread pool for processing. An independent task is created for each thread, and the task content includes backup or restore operations on the specified cloud disk partitioned block. Simultaneously, a unique task identifier is assigned to each task for tracking and management in subsequent operations.

[0053] Understandably, determining the number of threads in the thread pool based on the number of data chunks and currently available computing resources, and allocating each data chunk to a thread for processing includes:

[0054] When setting cloud disk partitioning strategy parameters, the size level of each partition block is set for each partitioning strategy type. The size levels of the partition blocks include large capacity level, medium capacity level and low capacity level.

[0055] In response to determining the number of threads in the thread pool, obtain the partitioning strategy type from the cloud disk partitioning strategy parameters, and obtain the partitioning block size level from the partitioning strategy type;

[0056] When the size level of the partition block in the partitioning strategy type is large capacity, set the number of threads in the thread pool to min(number of partition blocks, number of available processor cores * n1);

[0057] When the size level of the partition block in the partitioning strategy type is medium capacity, set the number of threads in the thread pool to min(number of partition blocks, number of available processor cores * n2);

[0058] When the size level of the partition block in the partitioning strategy type is low capacity, the number of threads in the thread pool is set to min(number of partition blocks, number of available processor cores * n3); where n1, n2, and n3 are adjustment coefficients, and n1 < n2 < n3.

[0059] This configuration prioritizes setting the number of threads based on the number of available processor cores when the block size is large, minimizing the impact of the number of blocks on the number of threads and improving the utilization of available computing resources. When the block size is low, prioritizing setting the number of threads based on the number of blocks allows for improved data processing efficiency through multithreading.

[0060] In this embodiment, a temporary virtual cloud disk is created, mounted onto the host machine, and an asynchronous input / output library is used to perform multi-threaded data transfer operations using a memory buffer. Consistency checks are performed on the transferred data, and backups or restores the verified data.

[0061] In response to the backup operation of cloud disk data, a snapshot of the cloud disk is taken, a temporary virtual cloud disk is created from the snapshot, and the temporary virtual cloud disk is mounted on the host machine;

[0062] According to the segmented blocks, use the asynchronous input / output library to read data from the cloud disk data into the memory buffer, and compress the data in the memory buffer to form compressed backup data;

[0063] Upload the compressed backup data to the object storage. Compare the hash value of the compressed backup data with the hash value of the source data. If they match, the backup ends. If they do not match, the source data is backed up again.

[0064] The preferred hash value is the SHA256 value, which is a fixed-length hash value of 256 bits (32 bytes) generated by the SHA-256 algorithm and is usually represented as 64 hexadecimal characters.

[0065] In this embodiment, reading data from the cloud disk data into the memory buffer using the asynchronous input / output library according to the segmented blocks includes:

[0066] Obtain the backup method selected by the user, which may include incremental backup, full backup, or differential backup.

[0067] In response to the incremental backup method, the current metadata information of each segment is compared with the metadata information of the previous backup. Data of segments where data has changed is uploaded to the memory buffer and the metadata information of the data is recorded.

[0068] If the backup method is full backup, all non-zero data will be uploaded to the memory buffer;

[0069] If the backup method is differential backup, then the data in each partition is compared with the data in the first backup, and all the different data and metadata information are backed up.

[0070] like Figure 4As shown, the backup operation process includes: after each thread assigned to a backup task starts, it takes a snapshot of the virtual machine cloud disk through the CinderAPI, and creates a temporary virtual cloud disk based on the snapshot to ensure that the source virtual cloud disk service is not affected. If the OpenStack cluster has multiple control nodes, you can choose to perform parallel backups through multiple nodes, and then mount the temporary virtual cloud disks to multiple host machines. When restoring the backup, only one node is allowed to restore at a time, and simultaneous writing is not allowed, otherwise it will cause a conflict.

[0071] According to the set cloud disk partitioning strategy, the backup request is sent to multiple host machines (each host machine has its own data range to be backed up). The asynchronous I / O library (such as aiohttp) is used to read data from the cloud disk into the memory buffer. An appropriate buffer size (such as 64KB) is set to improve data reading efficiency.

[0072] Zero-skipping processing is performed on the read data. This involves checking if the read data chunk equals bytes(len(chunk)). If they are equal, compression is skipped, and the offset of the cloud disk is recorded in the backup metadata and uploaded to the backup backend. This zero-skipping operation on the backup data is compared with the differentiated data during recovery, further accelerating backup and recovery efficiency.

[0073] During a full backup, all non-zero data is uploaded to the backup backend. During an incremental backup, the metadata information from the previous backup is compared to identify which data has changed. If no data has changed, it is skipped; otherwise, the data block is backed up, and its metadata information is recorded. For differential backups, the data from the first backup is selected for comparison, and all differing data and metadata information are backed up.

[0074] The data in the memory buffer is compressed using LZ4, and the compression operation is implemented by calling Python's lz4 library. The S3 API is then used to upload the data to object storage, and the SHA256 values ​​of the backup data and the source data are calculated. If they match, the backup process ends; otherwise, the backup process is restarted for another backup.

[0075] Data consistency verification includes the following steps: After the backup operation is completed, the system performs consistency verification on the backup data. This involves calculating the hash value of the backup data (such as MD5 or SHA-256) and comparing it with the hash value of the original cloud disk data to ensure consistency. If verification fails, the backup operation is re-executed. Similarly, after the recovery operation is completed, the recovered data undergoes consistency verification. By mounting the recovered virtual machine cloud disk, a file system check tool (such as fsck) is run to check the integrity and consistency of the file system. If data errors or file system corruption are found, the recovery operation is re-executed. This consistency verification of the backup and restored data ensures the accuracy of the backup and restored data.

[0076] In this embodiment, a temporary virtual cloud disk is created, mounted onto the host machine, and an asynchronous input / output library is used to perform multi-threaded data transfer operations using a memory buffer. Consistency checks are performed on the transferred data, and backups or restores the verified data.

[0077] In response to a data recovery operation on a cloud disk, a temporary virtual cloud disk is created and mounted onto the host machine;

[0078] The asynchronous input / output library is used to decompress the recovered data, and the decompressed recovered data is restored to the memory buffer.

[0079] Read the recovery data from the memory buffer, copy the recovery data to the temporary virtual cloud disk, obtain the difference data between the temporary virtual cloud disk and the cloud disk to be recovered, and write the difference data.

[0080] The hash value of the written difference data is compared with the hash value of the source data. If they match, the recovery is complete; otherwise, the difference data is recovered again.

[0081] like Figure 5As shown, the recovery process includes: After each thread assigned to the backup and recovery task starts, it creates a new temporary virtual cloud disk through the Cinder API, then mounts it to the host machine, preparing for the next step of data writing. The compressed data is decompressed using an asynchronous I / O library, and the decompression function of the lz4 library is called to restore the data to the memory buffer. The breakpoint resume function is also implemented based on S3 object storage. The required data is read into the memory buffer using the S3 API, and the decompression function of the lz4 library is called to parse the data, analyzing whether the backup is a full backup, differential backup, or incremental backup. If it is a full backup, the backup data is directly compared with the cloud disk to be restored, and then the differential data is written. If it is a differential backup, the metadata information of the backup data needs to be compared, the parent backup is found and merged, and then the differential data is written. If it is an incremental backup, the cascaded backups need to be found sequentially according to the data backup order, then the data is merged and the differential data is written. The SHA256 value of the data is calculated and compared with the source data. If they match, the recovery is complete; otherwise, the recovery process is restarted and retried. During recovery, a file system check tool is run to check the file system integrity.

[0082] In this embodiment, the cloud disk data backup and recovery method further includes:

[0083] Monitor the task execution status of each thread in the thread pool in real time. The task execution status includes the task start time, current progress, and time elapsed.

[0084] In response to detecting that the execution speed of the target thread has slowed down or is blocked, the task of the target thread is assigned to other idle threads for processing, or the processor and memory resources of the computing node where the target thread is located are increased;

[0085] When a data read failure, compression / decompression error, or network connection abnormality is detected in the target thread, it is determined that an error has occurred in the target thread, the error information is recorded, and the task of the target thread is reassigned to other threads for processing;

[0086] In response to the determination that an error has occurred in the target thread, the error type, occurrence time, and task identifier are summarized to form an error report, which is then sent.

[0087] In this system, administrators can visually view the task execution status of each thread in the monitored thread pool through OpenStack dashboards or API interfaces. When a thread is found to be executing too slowly or is blocked, the system automatically adjusts resource allocation. For example, it may assign the thread's task to other idle threads or increase the CPU and memory resources of the compute node where the thread resides to improve task execution efficiency. During backup or recovery, if an error occurs in a thread (such as data read failure, compression / decompression error, network connection failure, etc.), the system immediately records the error information and reassigns the erroneous task to other threads. Simultaneously, a detailed error report is sent to the system administrator, including the error type, time of occurrence, and task identifier, to facilitate quick problem identification and resolution.

[0088] The task monitoring and resource scheduling implementation is as follows: Python's logging module is used to record the execution status information of each thread task, including start time, current progress (calculated as the ratio of processed data to total data), and elapsed time. The thread status in the thread pool is checked periodically (e.g., every 5 seconds). If the execution progress of a thread does not change or is too slow within a certain time (e.g., 30 seconds), the thread is considered blocked. For blocked threads, the system reassigns its task to other idle threads. Simultaneously, the resource allocation of the compute node where the thread resides is adjusted via OpenStack's Nova API, increasing CPU and memory quotas. When a thread encounters an error, the exception information is captured and logged to a log file. The erroneous task is re-added to the task queue, awaiting reassignment. An error report is also sent to the system administrator via email or SMS. The task execution status is monitored in real time, and resource allocation is dynamically adjusted to ensure full utilization of system resources and avoid waste. Backup and recovery tasks are added to the monitoring and resource scheduling functions to ensure high availability of backup and recovery tasks.

[0089] The aforementioned cloud disk data backup and recovery method employs a multi-threaded parallel processing approach, dividing the cloud disk into multiple partitions and decomposing the backup and recovery task of a large-capacity cloud disk into multiple sub-tasks for simultaneous processing. This significantly shortens the operation time. Determining the number of threads in the thread pool based on the number of partitions and the currently available computing resources can improve the utilization of computing resources. The cloud disk partitioning strategy parameters can be flexibly configured according to business needs and cloud disk characteristics to meet data protection requirements in different scenarios. Consistency verification of transmitted data ensures the integrity and consistency of data during backup and recovery, reducing the risk of data loss. This achieves the goals of improving cloud disk backup and recovery efficiency, enhancing system reliability, and improving the utilization of computing resources.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0091] In one embodiment, such as Figure 6 As shown, a cloud hard drive data backup and recovery device 10 is provided, including: a partitioning strategy configuration module 1, a partitioning processing module 2, a multi-threaded task allocation module 3, and a data transmission control module 4.

[0092] The partitioning strategy configuration module 1 is used to set the partitioning strategy parameters for the cloud disks to be backed up or restored.

[0093] The partitioning module 2 is used to split the partitions on the cloud disk into multiple partition blocks according to the cloud disk partitioning strategy parameters when selecting to back up or restore cloud disk data.

[0094] The multi-threaded task allocation module 3 is used to determine the number of threads in the thread pool based on the number of partitioned blocks and the currently available computing resources, and to allocate the data of each partitioned block to a thread for processing.

[0095] The data transmission control module 4 is used to respond to the thread startup by creating a temporary virtual cloud disk, mounting the temporary virtual cloud disk to the host machine, using the asynchronous input / output library to perform multi-threaded data transmission operations using the memory buffer, performing consistency verification on the transmitted data, and backing up or restoring the transmitted data that passes the verification.

[0096] In this embodiment, setting the cloud disk partitioning strategy parameters includes:

[0097] The partitioning strategy type in the cloud disk partitioning strategy parameters includes partitioning the cloud disk equally according to the cloud disk capacity ratio, partitioning according to the existing file system partitions on the cloud disk, or partitioning according to the number of bytes in each partition block.

[0098] The cloud disk partitioning strategy parameters are stored in a database table structure. The database table structure contains fields such as cloud disk ID, partitioning strategy type, number of partition blocks, starting address and size of each partition block.

[0099] In this embodiment, determining the number of threads in the thread pool based on the number of partitioned blocks and currently available computing resources, and allocating the data of each partitioned block to a thread for processing includes:

[0100] Obtain cloud disk partitioning parameters, including the number of partitioned blocks, the starting address of each partitioned block, and its size;

[0101] Obtain currently available computing resources, including the number of available processor cores and memory size;

[0102] The number of threads in the thread pool is determined by the formula: Number of threads = min(Number of blocks, Number of available processor cores * 2).

[0103] Create a task for each thread to perform backup or restore operations on the data in the segmented blocks. Set the address at which each task reads data based on the starting address of each segmented block, and assign each task to it.

[0104] In this embodiment, a temporary virtual cloud disk is created, mounted onto the host machine, and an asynchronous input / output library is used to perform multi-threaded data transfer operations using a memory buffer. Consistency checks are performed on the transferred data, and backups or restores the verified data.

[0105] In response to the backup operation of cloud disk data, a snapshot of the cloud disk is taken, a temporary virtual cloud disk is created from the snapshot, and the temporary virtual cloud disk is mounted on the host machine;

[0106] According to the segmented blocks, use the asynchronous input / output library to read data from the cloud disk data into the memory buffer, and compress the data in the memory buffer to form compressed backup data;

[0107] Upload the compressed backup data to object storage. Compare the hash value of the compressed backup data (SHA256 value is a fixed-length 256-bit (32-byte) hash value generated by the SHA-256 algorithm, usually represented as 64 hexadecimal characters) with the hash value of the source data. If they match, the backup ends; otherwise, the source data is backed up again.

[0108] In this embodiment, reading data from the cloud disk data into the memory buffer using the asynchronous input / output library according to the segmented blocks includes:

[0109] Obtain the backup method selected by the user, which may include incremental backup, full backup, or differential backup.

[0110] In response to the incremental backup method, the current metadata information of each segment is compared with the metadata information of the previous backup. Data of segments where data has changed is uploaded to the memory buffer and the metadata information of the data is recorded.

[0111] If the backup method is full backup, all non-zero data will be uploaded to the memory buffer;

[0112] If the backup method is differential backup, then the data in each partition is compared with the data in the first backup, and all the different data and metadata information are backed up.

[0113] In this embodiment, a temporary virtual cloud disk is created, mounted onto the host machine, and an asynchronous input / output library is used to perform multi-threaded data transfer operations using a memory buffer. Consistency checks are performed on the transferred data, and backups or restores the verified data.

[0114] In response to a data recovery operation on a cloud disk, a temporary virtual cloud disk is created and mounted onto the host machine;

[0115] The asynchronous input / output library is used to decompress the recovered data, and the decompressed recovered data is restored to the memory buffer.

[0116] Read the recovery data from the memory buffer, copy the recovery data to the temporary virtual cloud disk, obtain the difference data between the temporary virtual cloud disk and the cloud disk to be recovered, and write the difference data.

[0117] The hash value of the written difference data is compared with the hash value of the source data. If they match, the recovery is complete; otherwise, the difference data is recovered again.

[0118] In this embodiment, as Figure 6 As shown, the cloud disk data backup and recovery device 10 also includes a task monitoring and resource scheduling module 5. The task monitoring and resource scheduling module 5 is used for:

[0119] Monitor the task execution status of each thread in the thread pool in real time. The task execution status includes the task start time, current progress, and time elapsed.

[0120] In response to detecting that the execution speed of the target thread has slowed down or is blocked, the task of the target thread is assigned to other idle threads for processing, or the processor and memory resources of the computing node where the target thread is located are increased;

[0121] When a data read failure, compression / decompression error, or network connection abnormality is detected in the target thread, it is determined that an error has occurred in the target thread, the error information is recorded, and the task of the target thread is reassigned to other threads for processing;

[0122] In response to the determination that an error has occurred in the target thread, the error type, occurrence time, and task identifier are summarized to form an error report, which is then sent.

[0123] The aforementioned cloud disk data backup and recovery device employs a multi-threaded parallel processing approach, dividing the cloud disk into multiple partitions and processing them in parallel. This decomposes the backup and recovery task of a large-capacity cloud disk into multiple sub-tasks for simultaneous processing, significantly shortening the operation time. Determining the number of threads in the thread pool based on the number of partitions and the currently available computing resources can improve the utilization of computing resources. The cloud disk partitioning strategy parameters can be flexibly configured according to business needs and cloud disk characteristics to meet data protection requirements in different scenarios. Consistency verification of transmitted data ensures the integrity and consistency of data during backup and recovery, reducing the risk of data loss. This achieves the goals of improving cloud disk backup and recovery efficiency, enhancing system reliability, and improving the utilization of computing resources.

[0124] For a description of the features in the embodiment of the cloud hard drive data backup and recovery device, please refer to the relevant description of the embodiment of the cloud hard drive data backup and recovery method, which will not be repeated here.

[0125] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described cloud hard drive data backup and recovery method embodiments.

[0126] In one embodiment, the electronic device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores cloud disk data backup and recovery data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a cloud disk data backup and recovery method.

[0127] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described cloud hard drive data backup and recovery method embodiments when running.

[0128] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0129] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described cloud hard drive data backup and recovery method embodiments.

[0130] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described cloud hard drive data backup and recovery method embodiments.

[0131] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] The foregoing has provided a detailed description of a cloud hard drive data backup and recovery method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for data backup and recovery on a cloud hard drive, characterized in that, include: Configure cloud disk partitioning strategy parameters for backup or recovery; In response to selecting to back up or restore cloud disk data, the partition on the cloud disk is divided into multiple partition blocks according to the cloud disk partitioning strategy parameters; The number of threads in the thread pool is determined based on the number of the partitioned blocks and the currently available computing resources, and the data of each partitioned block is assigned to a thread for processing. In response to the thread starting, a temporary virtual cloud disk is created, the temporary virtual cloud disk is mounted on the host machine, the asynchronous input / output library is used to perform multi-threaded data transmission operations using a memory buffer, and the transmitted data is checked for consistency. The transmitted data that passes the check is backed up or restored.

2. The cloud hard drive data backup and recovery method according to claim 1, characterized in that, The parameters for setting the cloud disk partitioning strategy include: The partitioning strategy type in the cloud disk partitioning strategy parameters includes partitioning the cloud disk equally according to the cloud disk capacity ratio, partitioning according to the existing file system partitions on the cloud disk, or partitioning by customizing the number of bytes in each partition block. The cloud disk partitioning strategy parameters are stored in a database table structure. The database table structure includes fields such as cloud disk ID, partitioning strategy type, number of partition blocks, starting address and size of each partition block.

3. The cloud hard drive data backup and recovery method according to claim 2, characterized in that, The step of determining the number of threads in the thread pool based on the number of partitioned blocks and the currently available computing resources, and allocating the data of each partitioned block to a thread for processing includes: Obtain cloud disk partitioning parameters, which include the number of partitioned blocks, the starting address of each partitioned block, and its size; Obtain currently available computing resources, including the number of available processor cores and memory size; The number of threads in the thread pool is determined by the formula: Number of threads = min(Number of blocks, Number of available processor cores * 2). Create a task for each thread to back up or restore the data of the segmented block. Set the address for each task to read data based on the starting address of each segmented block. Assign each task to a thread in the thread pool for processing. Assign a unique task identifier to each task.

4. The cloud hard drive data backup and recovery method according to claim 1, characterized in that, The process of creating a temporary virtual cloud disk, mounting the temporary virtual cloud disk to the host machine, using an asynchronous input / output library to perform multi-threaded data transfer operations with a memory buffer, performing consistency checks on the transferred data, and backing up or restoring the verified data includes: In response to a backup operation on cloud disk data, a snapshot is taken of the cloud disk, a temporary virtual cloud disk is created using the snapshot, and the temporary virtual cloud disk is mounted onto the host machine; According to the segmented blocks, use the asynchronous input / output library to read data from the cloud disk data into the memory buffer, and compress the data in the memory buffer to form compressed backup data; The compressed backup data is uploaded to the object storage. The hash value of the compressed backup data is compared with the hash value of the source data. If they match, the backup ends. If they do not match, the source data is backed up again.

5. The cloud hard drive data backup and recovery method according to claim 4, characterized in that, The step of reading data from the cloud disk data into the memory buffer using an asynchronous input / output library according to the segmented blocks includes: The backup method is specified, including incremental backup, full backup, or differential backup. In response to the backup method being incremental backup, the current metadata information of each segment is compared with the metadata information of the previous backup. Data of segments where data has changed is uploaded to the memory buffer and the metadata information of the data is recorded. If the backup method is a full backup, then all non-zero data will be uploaded to the memory buffer. If the backup method is differential backup, then the data in each partition is compared with the data in the first backup, and all the different data and metadata information are backed up.

6. The cloud hard drive data backup and recovery method according to claim 1, characterized in that, The process of creating a temporary virtual cloud disk, mounting the temporary virtual cloud disk to the host machine, using an asynchronous input / output library to perform multi-threaded data transfer operations with a memory buffer, performing consistency checks on the transferred data, and backing up or restoring the verified data includes: In response to a data recovery operation on a cloud disk, a temporary virtual cloud disk is created and mounted onto the host machine; The asynchronous input / output library is used to decompress the recovered data, and the decompressed recovered data is restored to the memory buffer. Read the recovery data from the memory buffer, copy the recovery data to the temporary virtual cloud disk, obtain the difference data between the temporary virtual cloud disk and the cloud disk to be recovered, and write the difference data. The hash value of the written difference data is compared with the hash value of the source data. If they match, the recovery is complete; otherwise, the difference data is recovered again.

7. The cloud hard drive data backup and recovery method according to claim 1, characterized in that, The cloud disk data backup and recovery method also includes: Real-time monitoring of the task execution status of each thread in the thread pool, including the task's start time, current progress, and elapsed time; In response to detecting a slowdown or blockage in the execution speed of the target thread, the task of the target thread is assigned to other idle threads for processing, or the processor and memory resources of the computing node where the target thread is located are increased; In response to the detection of data read failure, compression / decompression error, or network connection abnormality in the target thread, the system determines that the target thread has encountered an error, records the error information, and reassigns the task of the target thread to other threads for processing; in response to the determination that the target thread has encountered an error, the system summarizes the error type, occurrence time, and task identifier to form an error report and sends the error report.

8. A cloud hard drive data backup and recovery device, characterized in that, include: The partitioning strategy configuration module is used to set the partitioning strategy parameters for the cloud disks to be backed up or restored. The partitioning module is used to divide the partition on the cloud disk into multiple partition blocks according to the cloud disk partitioning strategy parameters when selecting to back up or restore cloud disk data. The multi-threaded task allocation module is used to determine the number of threads in the thread pool based on the number of the partitioned blocks and the currently available computing resources, and to allocate the data of each partitioned block to a thread for processing. The data transmission control module is used to respond to the thread startup by creating a temporary virtual cloud disk, mounting the temporary virtual cloud disk to the host machine, using an asynchronous input / output library to perform multi-threaded data transmission operations using a memory buffer, performing consistency checks on the transmitted data, and backing up or restoring the transmitted data that passes the checks.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the cloud hard drive data backup and recovery method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the cloud hard drive data backup and recovery method as described in any one of claims 1 to 7.

Images