Data disaster recovery conversion method and system based on distributed storage, medium and product

Abstract

The application discloses a kind of data disaster recovery conversion methods, systems, medium and product based on distributed storage, it is related to data disaster recovery conversion technical field.Authenticity verification is carried out to source distributed node and target distributed node;After passing through authentication verification, the mirror data list of the data to be migrated is obtained from source distributed node;Data block size is determined based on the data storage type corresponding to source distributed node and the data storage type corresponding to target distributed node, and each mirror data in the mirror data list is blocked based on data block size;All blocked mirror data is written into network transmission pipeline, so that network transmission pipeline sends all blocked mirror data to target distributed node;Through target distributed node, the received blocked mirror data is format converted, deduplication pool filtered and data is written.The method, system, medium and product disclosed by the application can realize data disaster recovery conversion between heterogeneous nodes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data disaster recovery conversion, and particularly relates to a data disaster recovery conversion method and system based on distributed storage, a medium and a product. BACKGROUND

[0002] Data disaster recovery conversion refers to a process of migrating data from a primary storage system (or a source node) to a backup storage system (or a target node) to ensure the availability, consistency and business continuity of data in a disaster scenario. In a distributed storage system, data disaster recovery is a core mechanism for ensuring service availability. An open-source distributed storage system represented by Ceph realizes basic disaster recovery capability through technologies such as multi-copy replication and erasure coding.

[0003] However, in a distributed storage system, when data migration or disaster recovery switching is performed between heterogeneous nodes (such as different hardware architectures, storage media, Ceph versions or third-party storage systems), data disaster recovery conversion between the heterogeneous nodes is often difficult to achieve due to data format conflicts.

[0004] Therefore, how to provide an effective solution to realize data disaster recovery conversion between heterogeneous nodes in a Ceph distributed storage system has become a difficult problem to be solved in the prior art. SUMMARY

[0005] The application aims to provide a data disaster recovery conversion method and system based on distributed storage, and a medium and a product, to solve the above problems existing in the prior art.

[0006] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0007] In a first aspect, the application provides a data disaster recovery conversion method based on distributed storage, which is used to realize data disaster recovery conversion between heterogeneous nodes in a Ceph distributed storage system, and includes the following steps:

[0008] Authenticating and verifying the source distributed node and the target distributed node;

[0009] After the authentication and verification are passed, obtaining a mirror data list of to-be-migrated data from the source distributed node;

[0010] Determining a data block size based on a data storage type corresponding to the source distributed node and a data storage type corresponding to the target distributed node, and blocking each mirror data in the mirror data list based on the data block size;

[0011] Writing all the blocked mirror data into a network transmission pipeline, so that the network transmission pipeline sends all the blocked mirror data to the target distributed node;

[0012] The target distributed node performs format conversion, deduplication pool filtering and data writing on the received chunked mirror data to realize data migration.

[0013] Based on the above disclosure, the application verifies the source distributed node and the target distributed node; obtains a mirror data list of data to be migrated from the source distributed node after the verification; determines a data chunk size based on a data storage type corresponding to the source distributed node and a data storage type corresponding to the target distributed node, and chunks each mirror data in the mirror data list based on the data chunk size; writes all the chunked mirror data to a network transmission pipeline, so that the network transmission pipeline sends all the chunked mirror data to the target distributed node; and the target distributed node performs format conversion, deduplication pool filtering and data writing on the received chunked mirror data to realize data migration. In this way, the divided data block size can conform to the storage limit of the target distributed node during data disaster recovery conversion, and the data format is the format of the storage type corresponding to the target distributed node, so that the data in the source distributed node of the Ceph distributed storage system can be migrated and backed up to the heterogeneous target distributed node, and data disaster recovery conversion between heterogeneous nodes can be realized.

[0014] In one possible design, the authentication verification of the source distributed node and the target distributed node includes:

[0015] Obtains first authentication information of the source distributed node and second authentication information of the target distributed node entered by a user;

[0016] Authenticates and verifies the source distributed node and the target distributed node based on the first authentication information and the second authentication information;

[0017] The first authentication information includes an IP address, a port PORT and an access key of the source distributed node, and the second authentication information includes an IP address, a port PORT and an access key of the target distributed node.

[0018] In one possible design, after obtaining the mirror data list of data to be migrated from the source distributed node, the method further includes:

[0019] In response to a setting operation of a user, determines a task thread number and a network transmission bandwidth of mirror migration;

[0020] Generates a task list corresponding to the mirror data list.

[0021] In one possible design, the writing of all the chunked mirror data to the network transmission pipeline includes:

[0022] Start at least one task in the task list based on the number of task threads;

[0023] Based on the network transmission bandwidth, the image data corresponding to each task in the at least one task is written into the network transmission pipeline after being divided into blocks.

[0024] Once any task is completed, check if there are any unstarted tasks in the task list. If so, start one of the unstarted tasks, until there are no unstarted tasks in the task list.

[0025] In one possible design, after initiating at least one task in the task list, the method further includes:

[0026] Obtain the metadata of the image data corresponding to the started task, and record the metadata in the transaction log.

[0027] In one possible design, determining the data block size based on the data storage type corresponding to the source distributed node and the data storage type corresponding to the target distributed node includes:

[0028] If the data storage type corresponding to the source distributed node is the same as the data storage type corresponding to the target distributed node, then the data block size is determined based on the size of the data block in the source distributed node.

[0029] If the data storage type corresponding to the source distributed node is different from the data storage type corresponding to the target distributed node, and the data storage type corresponding to the target distributed node is of type S3, then the data block size is determined to be 4MiB.

[0030] If the data storage type corresponding to the source distributed node is different from the data storage type corresponding to the target distributed node, and the data storage type corresponding to the target distributed node is local disk storage, then the data block size is determined to be 16MiB.

[0031] In one possible design, the target distributed node writes data by invoking the storage write adapter.

[0032] Secondly, this invention provides a data disaster recovery and transformation system based on distributed storage, used to realize data disaster recovery and transformation between heterogeneous nodes in a Ceph distributed storage system, including:

[0033] The authentication and verification unit is used to authenticate and verify the source distributed node and the target distributed node.

[0034] The acquisition unit is used to obtain a list of mirror data of the data to be migrated from the source distributed node after the authentication and verification are passed.

[0035] The determining unit is used to determine the data block size based on the data storage type corresponding to the source distributed node and the data storage type corresponding to the target distributed node;

[0036] A segmentation unit is used to segment each image data in the image data list into blocks based on the data block size;

[0037] The writing unit is used to write all the segmented image data into the network transmission pipeline so that the network transmission pipeline can send all the segmented image data to the target distributed node.

[0038] The data processing unit is used to perform format conversion, deduplication pool filtering, and data writing on the received block-based mirror data through the target distributed node to achieve data migration.

[0039] Thirdly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the data disaster recovery transformation method based on distributed storage as described in the first aspect or any possible design of the first aspect.

[0040] Fourthly, the present invention provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform a data disaster recovery transformation method based on distributed storage as described in the first aspect or any possible design of the first aspect.

[0041] Beneficial effects:

[0042] The data disaster recovery conversion method, system, medium and product based on distributed storage provided by this invention can ensure that the size of the divided data blocks meets the storage limit of the target distributed node and the data format is the format of the storage type corresponding to the target distributed node during data disaster recovery conversion. This enables the migration and backup of data in the source distributed node of the Ceph distributed storage system to the heterogeneous target distributed node, realizing data disaster recovery conversion between heterogeneous nodes, which is convenient for practical application and promotion. Attached Figure Description

[0043] Figure 1 A flowchart illustrating the data disaster recovery and transformation method based on distributed storage provided in this application embodiment;

[0044] Figure 2 This is a block diagram of a data disaster recovery and transformation system based on distributed storage provided in an embodiment of this application. Detailed Implementation

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0046] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0047] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0048] Example:

[0049] like Figure 1 As shown, the first aspect of this embodiment provides a data disaster recovery transformation method based on distributed storage. This method can be applied to a Ceph distributed storage system to achieve data disaster recovery transformation between heterogeneous nodes in the Ceph distributed storage system. Figure 1 As shown, the data disaster recovery transformation method based on distributed storage may include, but is not limited to, the following steps S101 to S105.

[0050] Step S101. Perform authentication and verification on the source distributed node and the target distributed node.

[0051] The source distributed node can be a node in the Ceph distributed storage system to which data needs to be migrated and backed up, and the target distributed node can be a node in the Ceph distributed storage system to which data needs to be migrated.

[0052] When performing disaster recovery migration of data between heterogeneous nodes in a Ceph distributed storage system, users can enter the first authentication information of the source distributed node and the second authentication information of the target distributed node. Authentication and verification of the source and target distributed nodes can then be performed based on the first and second authentication information. The first authentication information may include, but is not limited to, the IP address (Internet Protocol Address), port (a logical channel used to identify different services on the same device), and access key of the source distributed node. The second authentication information may include, but is not limited to, the IP address, port, and access key of the target distributed node.

[0053] Step S102. After successful authentication, obtain the list of mirror data of the data to be migrated from the source distributed node.

[0054] After successful authentication, the distributed system can obtain mirror data of all data from the source distributed nodes. At this point, the user can manually select the mirror data that needs to be migrated and backed up. For ease of explanation, in this embodiment, all the selected mirror data that needs to be migrated and backed up are referred to as the mirror data list.

[0055] Understandably, if the user does not manually select the image data to be migrated and backed up, the system may default to using a portion or all of the image data as the image data to be migrated and backed up.

[0056] In one or more embodiments, the user can manually set the number of task threads and network transmission bandwidth for image migration. The system can then respond to the user's settings and determine the number of task threads and network transmission bandwidth for image migration. Simultaneously, a task list corresponding to the image data list can be generated. Tasks in the task list can correspond one-to-one with image data in the image data list, or one task in the task list can correspond to multiple image data in the image data list. That is, each task corresponds to the migration and backup process of one or more image data.

[0057] Step S103. Determine the data block size based on the data storage type corresponding to the source distributed node and the data storage type corresponding to the target distributed node, and divide each mirror data in the mirror data list into blocks based on the data block size.

[0058] Specifically, if the data storage type corresponding to the source distributed node is the same as the data storage type corresponding to the target distributed node, the data block size can be determined based on the size of the data block in the source distributed node.

[0059] If the data storage type corresponding to the source distributed node is different from the data storage type corresponding to the target distributed node, and the data storage type corresponding to the target distributed node is S3 (Amazon S3, Amazon Simple Storage Service), then the data block size is determined to be 4MiB.

[0060] If the data storage type corresponding to the source distributed node is different from the data storage type corresponding to the target distributed node, and the data storage type corresponding to the target distributed node is local disk storage, then the data block size is determined to be 16MiB.

[0061] In this way, the size of the data blocks can be dynamically adjusted to ensure that the size of the divided data blocks meets the storage limits of the target distributed nodes.

[0062] Step S104. Write all the segmented image data into the network transmission pipeline so that the network transmission pipeline can send all the segmented image data to the target distributed node.

[0063] Specifically, writing all the segmented image data into the network transmission pipeline may include, but is not limited to, the following steps S1041-S1043.

[0064] Step S1041. Start at least one task in the task list according to the number of task threads.

[0065] For example, if the number of task threads is 16, then 16 tasks in the task list can be started.

[0066] Step S1042. Based on the network transmission bandwidth, write the image data corresponding to each task in the at least one task into the network transmission pipeline after being divided into blocks.

[0067] That is, the data transmission speed is controlled according to the network transmission bandwidth, and the image data corresponding to each task in at least one task is written into the network transmission pipeline so that the image data corresponding to each task in at least one task is transmitted to the target distributed node through the network transmission pipeline.

[0068] Step S1043. When any task is completed, check if there are any unstarted tasks in the task list. If there are unstarted tasks, start one of them until there are no unstarted tasks in the task list.

[0069] When any of the started tasks is completed, the task list can be checked to see if there are any unstarted tasks. If there are unstarted tasks, one of the unstarted tasks will be started, until there are no unstarted tasks in the task list.

[0070] In one or more embodiments, after starting at least one task in the task list, the system may also obtain the metadata of the image data corresponding to the started task and record the metadata of the image data corresponding to the started task in the transaction log.

[0071] Step S105. The target distributed node performs format conversion, deduplication pool filtering, and data writing on the received segmented mirror data to achieve data migration.

[0072] After sending the segmented mirror data to the target distributed node, the distributed system can convert the received segmented mirror data into a format compatible with the data storage type of the target distributed node. Then, the converted mirror data is filtered by a deduplication pool (a mechanism that filters or processes data according to predefined rules before it enters a storage pool with deduplication / removal capabilities; its core purpose is to optimize storage efficiency, reduce unnecessary deduplication calculations, and ensure the security and compliance of critical data). Finally, the target distributed node calls the storage write adapter to write the data obtained after deduplication pool filtering to achieve data migration. After the data is written, the transaction log can be committed and the task list updated.

[0073] In one or more embodiments, before the received segmented image data is format-converted by the target distributed node, the received segmented image data can also be verified.

[0074] The data disaster recovery and conversion method based on distributed storage provided by this invention involves: authenticating and verifying the source and target distributed nodes; after successful authentication, obtaining a list of mirrored data to be migrated from the source distributed node; determining the data block size based on the data storage type of the source and target distributed nodes, and then dividing each mirrored data in the list into blocks based on the block size; writing all the blocked mirrored data into a network transmission pipeline so that the network transmission pipeline can send all the blocked mirrored data to the target distributed node; and finally, the target distributed node performs format conversion, deduplication pool filtering, and data writing on the received blocked mirrored data to achieve data migration. This ensures that the size of the divided data blocks conforms to the storage limitations of the target distributed node, and the data format is consistent with the storage type of the target distributed node during data disaster recovery and conversion. This enables the migration and backup of data from the source distributed node of the Ceph distributed storage system to the heterogeneous target distributed node, solving the "protocol gap" problem between heterogeneous storage media, achieving data disaster recovery and conversion between heterogeneous nodes, and facilitating practical application and promotion.

[0075] Please see Figure 2 The second aspect of this application provides a data disaster recovery and transformation system based on distributed storage, used to realize data disaster recovery and transformation between heterogeneous nodes in a Ceph distributed storage system. The data disaster recovery and transformation system based on distributed storage includes:

[0076] The authentication and verification unit is used to authenticate and verify the source distributed node and the target distributed node.

[0077] The acquisition unit is used to obtain a list of mirror data of the data to be migrated from the source distributed node after the authentication and verification are passed.

[0078] The determining unit is used to determine the data block size based on the data storage type corresponding to the source distributed node and the data storage type corresponding to the target distributed node;

[0079] A segmentation unit is used to segment each image data in the image data list into blocks based on the data block size;

[0080] The writing unit is used to write all the segmented image data into the network transmission pipeline so that the network transmission pipeline can send all the segmented image data to the target distributed node.

[0081] The data processing unit is used to perform format conversion, deduplication pool filtering, and data writing on the received block-based mirror data through the target distributed node to achieve data migration.

[0082] The working process, working details and technical effects of the data disaster recovery and transformation system based on distributed storage provided in the second aspect of this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0083] This embodiment's third aspect provides a computer-readable storage medium storing instructions comprising the data disaster recovery transformation method based on distributed storage as described in the first aspect of the embodiment. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the data disaster recovery transformation method based on distributed storage as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0084] The fourth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the data disaster recovery transformation method based on distributed storage as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0085] It should be understood that specific details are provided in the following description to facilitate a complete understanding of the exemplary embodiments. However, those skilled in the art will understand that the exemplary embodiments can be implemented without these specific details. For example, the system may be shown in block diagrams to avoid obscuring the example with unnecessary details. In other instances, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the exemplary embodiments.

[0086] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data disaster recovery and transformation method based on distributed storage, used to realize data disaster recovery and transformation between heterogeneous nodes in a Ceph distributed storage system, characterized in that, include: Perform authentication and verification on the source distributed nodes and the target distributed nodes; After successful authentication, obtain the mirror data list of the data to be migrated from the source distributed nodes; The data block size is determined based on the data storage type corresponding to the source distributed node and the data storage type corresponding to the target distributed node, and each mirror data in the mirror data list is divided into blocks based on the data block size; All the segmented image data is written into the network transmission pipeline so that the network transmission pipeline can send all the segmented image data to the target distributed node; The target distributed node performs format conversion, deduplication pool filtering, and data writing on the received block-based mirror data to achieve data migration. The process of determining the data block size based on the data storage type corresponding to the source distributed node and the data storage type corresponding to the target distributed node includes: If the data storage type corresponding to the source distributed node is the same as the data storage type corresponding to the target distributed node, then the data block size is determined based on the size of the data block in the source distributed node. If the data storage type corresponding to the source distributed node is different from the data storage type corresponding to the target distributed node, and the data storage type corresponding to the target distributed node is of type S3, then the data block size is determined to be 4MiB. If the data storage type corresponding to the source distributed node is different from the data storage type corresponding to the target distributed node, and the data storage type corresponding to the target distributed node is local disk storage, then the data block size is determined to be 16MiB.

2. The data disaster recovery and transformation method based on distributed storage according to claim 1, characterized in that, The authentication and verification of the source distributed node and the target distributed node includes: Obtain the first authentication information of the source distributed node and the second authentication information of the target distributed node entered by the user; The source distributed node and the target distributed node are authenticated and verified based on the first authentication information and the second authentication information. The first authentication information includes the IP address, port, and access key of the source distributed node, and the second authentication information includes the IP address, port, and access key of the target distributed node.

3. The data disaster recovery and transformation method based on distributed storage according to claim 1, characterized in that, After obtaining the mirror data list of the data to be migrated from the source distributed nodes, the method further includes: Responding to user settings, determine the number of task threads and network transmission bandwidth for image migration; Generate a task list corresponding to the image data list.

4. The data disaster recovery and transformation method based on distributed storage according to claim 3, characterized in that, The step of writing all the segmented mirror data into the network transmission pipeline includes: Start at least one task in the task list based on the number of task threads; Based on the network transmission bandwidth, the image data corresponding to each task in the at least one task is written into the network transmission pipeline after being divided into blocks. Once any task is completed, check if there are any unstarted tasks in the task list. If so, start one of the unstarted tasks, until there are no unstarted tasks in the task list.

5. The data disaster recovery and transformation method based on distributed storage according to claim 4, characterized in that, After initiating at least one task in the task list, the method further includes: Obtain the metadata of the image data corresponding to the started task, and record the metadata in the transaction log.

6. The data disaster recovery and transformation method based on distributed storage according to claim 1, characterized in that, The target distributed node writes data by calling the storage write adapter.

7. A data disaster recovery and transformation system based on distributed storage, used to realize data disaster recovery and transformation between heterogeneous nodes in a Ceph distributed storage system, characterized in that, include: The authentication and verification unit is used to authenticate and verify the source distributed node and the target distributed node. The acquisition unit is used to obtain a list of mirror data of the data to be migrated from the source distributed node after the authentication and verification are passed. The determining unit is used to determine the data block size based on the data storage type corresponding to the source distributed node and the data storage type corresponding to the target distributed node; A segmentation unit is used to segment each image data in the image data list into blocks based on the data block size; The writing unit is used to write all the segmented image data into the network transmission pipeline so that the network transmission pipeline can send all the segmented image data to the target distributed node. The data processing unit is used to perform format conversion, deduplication pool filtering, and data writing on the received block-based mirror data through the target distributed node to achieve data migration. When determining the data block size based on the data storage type corresponding to the source distributed node and the data storage type corresponding to the target distributed node, the determining unit is specifically used for: If the data storage type corresponding to the source distributed node is the same as the data storage type corresponding to the target distributed node, then the data block size is determined based on the size of the data block in the source distributed node. If the data storage type corresponding to the source distributed node is different from the data storage type corresponding to the target distributed node, and the data storage type corresponding to the target distributed node is of type S3, then the data block size is determined to be 4MiB. If the data storage type corresponding to the source distributed node is different from the data storage type corresponding to the target distributed node, and the data storage type corresponding to the target distributed node is local disk storage, then the data block size is determined to be 16MiB.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, perform the data disaster recovery transformation method based on distributed storage as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the data disaster recovery transformation method based on distributed storage as described in any one of claims 1 to 6.

Images