Data storage and backup recovery system based on block storage

Abstract

The invention provides a data storage and backup recovery system based on block storage, which belongs to the field of data storage and is composed of three main components: a scheduling module is responsible for global backup and recovery task scheduling and monitoring; the backup execution module executes data extraction, transmission and backup management; the storage modules are distributed at different nodes to ensure persistent and secure storage of data. According to the system, communication between components is optimized by utilizing an innovative communication protocol, and various message types are supported to realize efficient cooperative work, including intelligent component registration, heartbeat detection, backup, recovery instruction processing and the like. According to the invention, through highly refined data block management, a strong storage resource control capability is provided for a user.

Description

Technical Field

[0001] The present invention relates to the field of data storage, and in particular to a data storage and backup recovery system based on block storage. Background Art

[0002] As a mature data storage solution, block storage uses highly refined "data blocks" as the basic unit, giving users the ability to control storage resources. Each block represents an uninterrupted hard disk capacity. Cloud hard disk is a virtualized block storage service designed for virtual machines. By configuring reserved storage space on demand, it not only maximizes cost-effectiveness, but also ensures high reliability and nearly unlimited expansion potential of the service.

[0003] With the rapid development of information technology, enterprises and individuals are increasingly dependent on data resources, and data security and immediate availability have become core issues that cannot be ignored. In this context, efficient data backup mechanisms and rapid data recovery technologies have become the lifeline for ensuring business continuity and maintaining data security. However, traditional backup methods are often limited by the drawbacks of slow recovery speed and excessive resource consumption. When responding to emergencies, solutions that can achieve recovery in seconds are particularly critical, highlighting the urgent need for innovation and upgrading of modern backup technology.

[0004] With the rapid development of information technology, enterprises and individuals are increasingly dependent on data. Data security and immediate accessibility have become the core elements of business survival. Therefore, building an efficient data backup mechanism and rapid recovery capabilities has become the key to maintaining data security and ensuring business continuity. Traditional backup technology is limited by its slow recovery speed and high resource consumption, making it incapable of coping with emergencies. This has prompted the industry to urgently demand modern backup technology that can achieve second-level recovery. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a data storage and backup recovery system based on block storage. By integrating the flexibility of block storage with the advantages of cloud technology, a scalable, highly reliable and easy-to-manage data protection system is constructed. It is particularly suitable for various organizations and individual users with strict requirements on data security and business continuity, and provides strong support for data management in the digital age.

[0006] The technical solution of the present invention is:

[0007] A data storage and backup recovery system based on block storage, characterized by comprising:

[0008] a) A scheduling module, which is responsible for receiving backup instructions, intelligently scheduling backup tasks, processing recovery instructions, implementing recovery operation scheduling, and maintaining heartbeat detection with the backup execution module and storage module;

[0009] b) at least one backup execution module, which is used to receive and execute backup and recovery tasks, including precise data extraction, secure transmission and backup record management;

[0010] c) A set of storage modules distributed on each storage node of the backup cluster, focusing on data storage and analysis to ensure the integrity and security of backup data;

[0011] d) Among them, each module communicates efficiently through a customized communication protocol based on the TCP protocol. The protocol defines a series of message types such as component registration request, heartbeat detection request and response, snapshot creation request and response, backup creation request and response, metadata acquisition request and response, and recovery request and response, so as to realize the collaborative operation of various components in the system.

[0012] Furthermore,

[0013] The scheduling module further includes an automatic fault detection and recovery mechanism, which can automatically reschedule tasks when any component is detected to be abnormal, thereby ensuring service continuity.

[0014] The backup execution module adopts multi-threaded segment processing technology, dynamically adjusts the processing strategy according to the size of the data block to optimize the backup efficiency, and supports incremental data identification and duplicate data deletion to reduce storage occupancy.

[0015] The storage module can adaptively identify the backup performance level it provides according to the storage node configuration, including but not limited to standard and high-performance, so as to realize on-demand allocation of storage resources.

[0016] A communication framework based on IO multiplexing technology, especially the Netty framework, is adopted to efficiently manage multiple TCP connections with a single thread, optimize system resource utilization, especially the concurrent performance in backup and recovery task processing.

[0017] Going further,

[0018] Provides a detailed backup and recovery process, covering the entire process management of component initialization, backup task creation and execution, recovery task execution, and block storage use, ensuring high automation and intelligence of data protection and business continuity.

[0019] in,

[0020] Startup of each component: First, configure and start the scheduling module and backup execution module on the management node. Then, configure and start the storage module on the storage node of each backup cluster to form a distributed data protection network. The scheduling module should be run first, followed by the backup execution module and storage module. During this process, the regist instruction in the protocol is used to report its own IP address and component identity to the scheduling module. When registering, the storage module clearly identifies the backup performance level it provides according to the different configurations of the storage node, whether it is standard or high-performance. At the same time, the scheduling module will periodically send heart_check_request messages to all registered components to check the operating status of each component. If the corresponding heart_check_response is not received within the specified time, the component will be automatically identified as offline and removed from the list of active components.

[0021] Create a backup task: When the scheduling module receives a backup request, it immediately sends the create_snapshot_request command to the backup execution module; after receiving the message, the backup execution module creates a temporary snapshot in the corresponding block storage of the production cluster to ensure the data at the corresponding time and ensure the consistency of the backup data; after successful creation, it returns a create_snapshot_response message to the scheduling module to confirm that the creation is complete; after receiving the message, the scheduling module selects a storage module with corresponding performance according to storage requirements, and sends the create_backup_request command to the backup execution module, and the message body encapsulates the snapshot information that was successfully created and the selected storage module information; after receiving the message, the backup execution module starts to execute the backup and returns a create_backup_response message to the scheduling module.

[0022] Backup task execution: The backup execution module first uses the successfully created snapshot to create a temporary volume and mounts the temporary volume to itself; it uses a multi-threaded approach to read data from the volume in segments and divides the processing tasks according to the size of the data block: if the block size exceeds 100G, 100G is used as the processing unit, and each thread processes 100G / X bytes of data. When all 100G are processed, the next segment of data is processed; if the data block is less than 100G, each thread processes the block size / 10 / X bytes of data; during the data processing process, the backup execution module needs to send the get_metadata_request instruction to the storage module to obtain the most recent metadata record of the volume ; After receiving the message, the storage module returns get_metadata_response to carry the required message; After the backup execution module obtains the message, it determines whether the read data is incremental data. If it is incremental data, it calculates the md5 value of the data and compares it with the fingerprint value of the deduplication database; When the data processing is completed, the create_backup_request command is sent to the storage module. The message not only encapsulates the data to be backed up, but also includes the backup index address. The storage module returns create_backup_response after successfully writing the data to the disk; When all data is written to the disk, the temporary volume and snapshot are recovered and the backup status is updated successfully.

[0023] For the data that has been deduplicated, only the index position is recorded in the backup index address.

[0024] Execution of recovery task: The scheduling module sends a restore_request instruction to the storage module based on the backup information. After receiving the message, the storage module first creates a logical block storage, then traverses the metadata records from the backup to its root backup, that is, the full backup on the backup chain, and translates each block of data on the logical block into the data index position specified by the storage node corresponding to the metadata record; after all blocks are translated, the restore_response is returned to the scheduling module and the storage node location information of the block mapping is updated, and the production cluster is switched to the backup cluster; at the same time, the storage module performs data synchronization asynchronously, by creating several snapshots and continuously comparing snapshot difference data blocks, the difference data is continuously written to the corresponding position of the original production cluster. When the difference data is empty, the data synchronization is completed, and the restore_response message is reported to the scheduling module again to update the storage node location information of the block mapping, and the backup cluster is switched to the production cluster.

[0025] Block storage usage: When the virtual machine is mounted, the scheduling module maps the block storage device to the virtual machine according to the storage node location information of the block mapping; if the block storage is already in the mounted state, when the schedule receives the updated block mapping location information, it is also necessary to perform a swap operation to update the storage information on the virtual machine to the block information of the new location.

[0026] IO multiplexing technology is used to allow a single thread to monitor and process several TCP connections at the same time; the child thread pool of the backup execution module and the storage module is limited to contain only one thread.

[0027] The beneficial effects of the present invention are

[0028] a. Enhanced data security and recovery capabilities: High-efficiency backup mechanisms combined with second-level recovery technology ensure that data can be quickly restored in the face of system failures or disaster events, reducing business interruption time and improving service reliability and customer satisfaction.

[0029] b. Intelligent and automated operation and maintenance scheduling: The scheduling module (schedule component) realizes the intelligent scheduling of backup and recovery processes, reduces the burden of manual intervention, and improves operation and maintenance efficiency.

[0030] c. High scalability and flexibility: The backup execution module (backup component) and storage module (storage component) are designed to be horizontally scalable, which can easily adapt to the rapid growth of data volume and meet future expansion needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the principle of implementing cloud hard disk backup and second-level recovery. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] The present invention provides a data storage and backup recovery system based on block storage, which provides users with powerful storage resource control capabilities through highly refined data block management. The core system architecture consists of three main components: the scheduling module is responsible for global backup and recovery task scheduling and monitoring; the backup execution module performs data extraction, transmission and backup management; the storage module is distributed in different nodes to ensure the persistent and secure storage of data. The system uses innovative communication protocols to optimize communication between components and supports multiple message types to achieve efficient collaborative work, including intelligent component registration, heartbeat detection, backup and recovery instruction processing, etc.

[0034] The system design places special emphasis on the efficiency of backup and recovery, and uses multi-threaded segmentation processing, incremental data identification and deduplication technology to greatly improve backup speed and storage efficiency. Through the use of IO multiplexing technology and the Netty framework, the system can effectively cope with high-concurrency scenarios and ensure the stability and efficiency of services. In addition, the system provides a detailed backup and recovery process, from component initialization, task scheduling, data processing to the final block storage application, with full automated management, ensuring the immediacy of data protection and business continuity.

[0035] In general, the solution system consists of three core modules: a scheduling module (referred to as schedule component), a backup execution module (referred to as backup component) that can be horizontally expanded according to actual conditions, and a group of storage modules (referred to as storage components) distributed on each storage node of the backup cluster. As the command center of the entire architecture, the schedule component plays a key role in the nervous system, and is in charge of the precise orchestration of the backup and recovery process. Its core responsibilities include receiving backup instructions, intelligently scheduling backup tasks, processing recovery instructions, implementing recovery operation scheduling, and maintaining heartbeat detection between the Backup component and the Storage component. As the main executor of the system, the backup component takes the initiative to receive and execute backup and recovery tasks, covering the precise extraction and processing of data, secure transmission, and rigorous management of backup records. It is the backbone to ensure the smooth implementation of data protection tasks. In contrast, the storage component focuses on the storage and analysis of data. By deploying this component on each storage node, it realizes the dual reinforcement of the integrity and security of backup data, and builds an indestructible defense line for the long-term preservation of data. This carefully designed architecture ensures a perfect combination of efficiency and security in every link, from policy formulation to execution and data storage.

[0036] In order to ensure stable and efficient collaboration between components within the system, an innovative communication protocol is adopted, which relies on the solid reliability of the TCP protocol to provide a solid guarantee for information transmission. The message structure of this protocol is sophisticated and is divided into two major components: the header and the message body. The header encapsulates core metadata such as the message type to provide guidance for the classification and orientation of information; while the message body directly carries the specific substantive content of the communication, ensuring the effective transmission of data. The protocol specifically defines a series of detailed message types, aiming to comprehensively cover all aspects of component interaction, including but not limited to: component registration request (regist), used to initialize identification and connection between components; heartbeat detection request (heart_check_request) and response (heart_check_response), to ensure the activity of the service and network connectivity; create snapshot request (create_snapshot_request) and its response (create_snapshot_response), to provide instructions and confirmation for instantaneous capture of data status; create backup request (create_backup_request) and response (create_backup_response), to ensure smooth scheduling and feedback of backup operations; get metadata request (get_metadata_request) and response (get_metadata_response), to support efficient retrieval of data structure information; and restore request (restore_request) and response (restore_response), to accurately guide the data recovery process and provide real-time feedback on the execution status. This series of carefully designed message types together builds a precise execution framework for system operation instructions, ensuring that each operation step can receive a timely and accurate response, thereby improving the operating efficiency and reliability of the entire system.

[0037] A detailed backup and recovery process follows rigorous steps, as follows:

[0038] Startup of each component: The first step is to configure and start the schedule component and backup component on the management node. Then, the storage component needs to be configured and started on the storage node of each backup cluster to form a distributed data protection network. The key to the startup process is that the schedule component should be run first, followed by the backup component and storage component. In this process, the regist instruction in the above protocol is used to report its own IP address and component identity to the schedule component. It is worth noting that when the storage component is registered, it can clearly identify the backup performance level it provides according to the different configurations of the storage node, whether it is standard or high-performance. At the same time, the schedule component will periodically send heart_check_request messages to all registered components to check the operating status of each component. If the corresponding heart_check_response is not received within the specified time, the component will be automatically identified as offline and removed from the list of active components to ensure effective management of system resources and immediate response to faults.

[0039] Create a backup task: When the schedule component receives a backup request, it immediately sends the create_snapshot_request command to the backup component; after receiving the message, the backup component quickly creates a temporary snapshot in the corresponding block storage of the production cluster to ensure the data at the corresponding moment and the consistency of the backup data. After successful creation, it will return a create_snapshot_response message to the schedule component to confirm the completion of the creation; after receiving the message, the schedule component selects a storage component with corresponding performance according to the storage requirements, and sends the create_backup_request command to the backup component. The message body should encapsulate the above-mentioned snapshot information of successful creation and the selected storage component information; after receiving the message, the backup component starts to perform the backup and returns a create_backup_response message to the schedule component.

[0040] Backup task execution: The backup component first creates a temporary volume using the previously successfully created snapshot and mounts the temporary volume to itself. It uses a multi-threaded approach to read data from the volume in segments, and finely divides the processing tasks according to the size of the data block: if the block size exceeds 100G, 100G is used as the processing unit, and each thread processes 100G / X bytes of data. When all 100G are processed, the next segment of data is processed; if the data block is less than 100G, each thread processes the block size / 10 / X bytes of data. During the data processing process, the backup component needs to send the get_metadata_request instruction to the storage component to obtain the most recent metadata record of the volume. After receiving the message, the storage component returns get_metadata_response with the required message; after the backup component obtains the message, it determines whether the read data is incremental data. If it is incremental data, the md5 value of the data is calculated and compared with the fingerprint value of the deduplication database. After the data processing is completed, the create_backup_request command is sent to the storage component. The message not only encapsulates the data to be backed up, but also includes the backup index address (for data that has been deduplicated, only its index position is recorded). The storage component returns create_backup_response after the data is successfully written to the disk. When all data is written to the disk, the temporary volume and snapshot are recovered and the backup status is updated successfully.

[0041] Execution of recovery task: The schedule component sends a restore_request instruction to the storage component based on the backup information. After receiving the message, the storage component first creates a logical block storage, and then traverses the metadata records of all backups from the backup to its root backup (full backup on the backup chain), and translates each block of data on the logical block into the data index location specified by the storage node corresponding to the metadata record. After all blocks are translated, the schedule component restore_response is returned and the storage node location information of the block mapping is updated (from the production cluster to the backup cluster). At the same time, the storage component performs data synchronization asynchronously, and continuously writes the difference data to the corresponding location of the original production cluster by creating multiple snapshots and continuously comparing the snapshot difference data blocks. When the difference data is empty, the data synchronization is completed, and the restore_response message is reported to the schedule component again to update the storage node location information of the block mapping (from the backup cluster to the production cluster).

[0042] Block storage usage: When the virtual machine is mounted, the schedule maps the block storage device to the virtual machine according to the storage node location information of the block mapping. If the block storage is already mounted, when the schedule receives the updated block mapping location information, it is also necessary to perform a swap operation to update the storage information on the virtual machine to the block information of the new location.

[0043] In the early days of network communication, Java widely adopted the traditional BIO (Blocking I / O) model. This model works in a blocking manner, which means that when a thread processes a TCP connection, it will focus entirely on the operation of the connection until the read and write operations on the connection are completed. Under this mechanism, a thread can only correspond to one TCP connection and cannot handle multiple connection requests at the same time. As the number of backup components and storage components increases, the schedule component faces a surge in the number of threads. To solve this problem, the performance testing system uses IO multiplexing technology, which allows a single thread to efficiently monitor and process multiple TCP connections at the same time. Netty is a popular Java language RPC (Remote Procedure Call) framework. When initialized, the server configures two dedicated thread pools: one for managing the establishment and closing of connections (parent thread pool), and the other for processing business logic (child thread pool). Considering the serial execution requirements of a single backup and recovery, the child thread pools of the backup component and the storage component are limited to contain only one thread. Such a configuration ensures that only one task of a component is executed in the system at any time, which not only prevents OOM situations caused by excessive use of resources, but also maintains the consistency and reliability of backup execution.

[0044] The above description is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data storage and backup recovery system based on block storage, characterized in that: include: The scheduling module is responsible for receiving backup instructions, intelligently scheduling backup tasks, processing recovery instructions, implementing recovery operation scheduling, and maintaining heartbeat detection with the backup execution module and storage module; The backup execution module is used to receive and execute backup and recovery tasks, including precise data extraction, secure transmission and backup record management; The storage modules distributed on each storage node of the backup cluster are used for data storage and analysis to ensure the integrity and security of the backup data; The modules communicate with each other through a customized communication protocol based on the TCP protocol. The protocol defines component registration requests, heartbeat detection requests and responses, snapshot creation requests and responses, backup creation requests and responses, metadata acquisition requests and responses, and recovery requests and responses to achieve collaborative operation of various components in the system.

2. The system according to claim 1, characterized in that The scheduling module includes an automatic fault detection and recovery mechanism, which automatically reschedules tasks when any component is detected to be abnormal, ensuring service continuity.

3. The system according to claim 1, characterized in that The backup execution module adopts multi-threaded segment processing technology, dynamically adjusts the processing strategy according to the size of the data block to optimize the backup efficiency, and supports incremental data identification and duplicate data deletion to reduce storage occupancy.

4. The system according to claim 1, characterized in that Startup of each component: First, configure and start the scheduling module and backup execution module on the management node. Then, configure and start the storage module on the storage node of each backup cluster to form a distributed data protection network. The scheduling module should be run first, followed by the backup execution module and storage module. During this process, the regist instruction in the protocol is used to report its own IP address and component identity to the scheduling module. When registering, the storage module clearly identifies the backup performance level it provides according to the different configurations of the storage node, whether it is standard or high-performance. At the same time, the scheduling module will periodically send heart_check_request messages to all registered components to check the operating status of each component. If the corresponding heart_check_response is not received within the specified time, the component will be automatically identified as offline and removed from the list of active components.

5. The system according to claim 4, characterized in that Create a backup task: When the scheduling module receives a backup request, it immediately sends the create_snapshot_request command to the backup execution module; After receiving the message, the backup execution module creates a temporary snapshot in the corresponding block storage of the production cluster to ensure the data at the corresponding time and ensure the consistency of the backup data; after successful creation, it returns the create_snapshot_response message to the scheduling module to confirm the completion of the creation; after receiving the message, the scheduling module selects a storage module with corresponding performance according to the storage requirements, and sends the create_backup_request command to the backup execution module. The message body encapsulates the snapshot information of the successful creation and the selected storage module information; after receiving the message, the backup execution module starts to execute the backup and returns the create_backup_response message to the scheduling module.

6. The system according to claim 5, characterized in that Backup task execution: The backup execution module first uses the successfully created snapshot to create a temporary volume and mounts the temporary volume to itself; it uses a multi-threaded approach to read data from the volume in segments and divides the processing tasks according to the size of the data block: if the block size exceeds 100G, 100G is used as the processing unit, and each thread processes 100G / X bytes of data. When all 100G are processed, the next segment of data is processed; if the data block is less than 100G, each thread processes the block size / 10 / X bytes of data; during the data processing process, the backup execution module needs to send the get_metadata_request instruction to the storage module to obtain the most recent metadata record of the volume ; After receiving the message, the storage module returns get_metadata_response to carry the required message; After the backup execution module obtains the message, it determines whether the read data is incremental data. If it is incremental data, it calculates the md5 value of the data and compares it with the fingerprint value of the deduplication database; When the data processing is completed, the create_backup_request command is sent to the storage module. The message not only encapsulates the data to be backed up, but also includes the backup index address. The storage module returns create_backup_response after successfully writing the data to the disk; When all data is written to the disk, the temporary volume and snapshot are recovered and the backup status is updated successfully.

7. The system according to claim 6, characterized in that For the data that has been deduplicated, only the index position is recorded in the backup index address.

8. The system according to claim 6 or 7, characterized in that: Execution of recovery task: The scheduling module sends a restore_request instruction to the storage module based on the backup information. After receiving the message, the storage module first creates a logical block storage, then traverses the metadata records from the backup to its root backup, that is, the full backup on the backup chain, and translates each block of data on the logical block into the data index position specified by the storage node corresponding to the metadata record; after all blocks are translated, the restore_response is returned to the scheduling module and the storage node location information of the block mapping is updated, and the production cluster is switched to the backup cluster; at the same time, the storage module performs data synchronization asynchronously, by creating several snapshots and continuously comparing snapshot difference data blocks, the difference data is continuously written to the corresponding position of the original production cluster. When the difference data is empty, the data synchronization is completed, and the restore_response message is reported to the scheduling module again to update the storage node location information of the block mapping, and the backup cluster is switched to the production cluster.

9. The system according to claim 8, characterized in that Block storage usage: When the virtual machine is mounted, the scheduling module maps the block storage device to the virtual machine according to the storage node location information of the block mapping; if the block storage is already in the mounted state, when the schedule receives the updated block mapping location information, it is also necessary to perform a swap operation to update the storage information on the virtual machine to the block information of the new location.

10. The system according to claim 1, characterized in that IO multiplexing technology is used to allow a single thread to monitor and process several TCP connections at the same time; the child thread pool of the backup execution module and the storage module is limited to contain only one thread.

Images