A distributed storage dynamic data protection system and method

Abstract

This invention belongs to the field of computer storage technology, specifically relating to a distributed storage dynamic data protection system and method. It includes a storage engine client, a single-machine storage engine module, and a storage engine cluster management module. The storage engine cluster management module publishes data views, determines the protection mode and read / write method of the data views, and propagates the data views to the storage engine client and the single-machine storage engine module. The storage engine client accesses the data views in the storage engine cluster management module through the protection mode and synchronizes the data views to the single-machine storage engine module. The storage engine client writes to the data views through the read / write method and migrates the migrateable data views through the protection mode. The storage engine cluster management module switches the migrated data views and propagates them to the storage engine client and the single-machine storage engine module, enabling seamless switching for business operations and avoiding cutovers.

Description

Technical Field

[0001] This invention belongs to the field of computer storage technology, specifically relating to a distributed storage dynamic data protection system and method. Background Technology

[0002] Distributed storage, also known as software-defined storage in the storage industry, is defined by the Storage Networking Industry Association (SNIA) as: a type of virtualized storage with a service management interface. SDS includes storage pooling functionality and allows for the definition of data service characteristics within the storage pool through the service management interface. Furthermore, SNIA proposes that software-defined storage (SDS) should possess the following characteristics:

[0003] High degree of automation, reducing the operational overhead of storage infrastructure by simplifying management; standard interfaces, supporting API management, publishing, and operation and maintenance of storage devices and services; virtualized data paths, supporting multiple standard protocols, allowing applications to write data through block storage, file storage, or object storage interfaces; scalability, the storage architecture has the ability to seamlessly scale up without affecting availability or causing performance degradation; transparency, storage should provide users with management and monitoring of available storage resources and overhead.

[0004] The most important aspect of the definition of virtualized storage is the virtualization of storage hardware. Compared to traditional storage, distributed storage no longer relies on proprietary hardware (such as storage controllers) and proprietary networks (such as FC networks). It can provide professional storage services through general-purpose hardware and networks.

[0005] Distributed storage technology is now widely used in traditional data centers, public cloud data centers, private cloud data centers, and hyperconverged workstations, playing an increasingly important role. Among these, the distributed storage engine provides high-performance, high-reliability, and efficient data access capabilities, and is the core component of distributed storage. Distributed block, file, object, and database functionalities are products built upon the distributed storage engine.

[0006] Currently, the storage engines used in well-known distributed storage products do not support switching from a 3-replica protection mode to an EC (erasure coding) protection mode, or from a small-scale non-folding EC to a large-scale EC protection mode. This means that if a 3-replica solution is used when initially building a distributed storage cluster, subsequent business changes (such as an increase in high-bandwidth services where a larger proportion of EC performs better) or cluster expansion (often involving hundreds or thousands of nodes where the reliability of 3 replicas is insufficient) require significant manpower, time, and resources for business cutover, which also carries considerable risk. Summary of the Invention

[0007] The purpose of this invention is to provide a distributed storage dynamic data protection system that enables seamless business switching and avoids the need for cutovers, thereby solving the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a distributed storage dynamic data protection system, comprising a storage engine client, a storage single-machine engine module, and a storage engine cluster management module, wherein: the storage engine cluster management module is used to publish a data view, determine the protection mode and read / write method of the data view, and propagate the data view to the storage engine client and the storage single-machine engine module; the storage engine client accesses the data view in the storage engine cluster management module through the protection mode and synchronizes the data view to the storage single-machine engine module; the storage engine client writes to the data view through the read / write method and migrates the migrateable data view through the protection mode; the storage engine cluster management module performs view switching on the migrated data view and propagates it to the storage engine client and the storage single-machine engine module.

[0009] Preferably, the storage single-machine engine module is used to provide local data persistence and persistent data access for the storage engine client.

[0010] Preferably, the read / write method is used to divide the data view into read-only data and read-write output data.

[0011] Preferably, the migrated data in the data view is read-only data; the non-migrated data in the data view is readable and writable output data.

[0012] Preferably, the protection mode is used to perform authoritative sharding on the data view.

[0013] Preferably, the authoritative shard includes an authoritative node disk and a non-authoritative set.

[0014] Preferably, the view switching includes: selecting the data view to be switched and switching the protection mode; selecting nodes that intersect with the authoritative node disk in the switched data view to construct a non-authoritative set; publishing the non-authoritative set as the first new view; initiating a migration command, marking the authoritative shards participating in the migration in the first new view as read-only, reading the data of the authoritative shards, and writing it to the non-authoritative set according to EC; after the migration, performing a consistency check on the first new view to form a new set of authoritative shards; re-converting the data view to writable data, publishing the second new view, and spreading the second new view to the cluster through a cluster view diffusion scheme.

[0015] Preferably, the data view is generated by the storage engine cluster management module.

[0016] Preferably, the first new view is read-only during the migration, and no new data is written during the migration process.

[0017] Preferably, the cluster includes: the storage engine cluster management module deploying the single-machine storage engine to a general-purpose server via the Paxos or Raft protocol.

[0018] On the other hand, this invention proposes a distributed storage dynamic data protection method, comprising:

[0019] Publish the data view, determine its protection mode and read / write method, and distribute the data view to the storage engine client and storage single-machine engine module;

[0020] Access the data view in the storage engine cluster management module through protected mode, and synchronize the data view to the storage single-machine engine module;

[0021] Data is written to the data view via read / write, and the migrateable data view is migrated via protected mode;

[0022] Perform view switching on the migrated data view and extend it to the storage engine client and storage single-machine engine module.

[0023] Technical effects and advantages of the present invention: The distributed storage dynamic data protection system proposed in this invention has the following advantages compared with the prior art:

[0024] The storage engine cluster management module of this invention is used to publish data views, determine the protection mode and read / write method of the data views, and propagate the data views to storage engine clients and storage single-machine engine modules. The storage engine client accesses the data views in the storage engine cluster management module through the protection mode and synchronizes the data views to the storage single-machine engine module. The storage engine client writes to the data views through the read / write method and migrates the migrateable data views through the protection mode. The storage engine cluster management module switches the migrated data views and propagates them to the storage engine clients and storage single-machine engine modules, enabling seamless switching for business operations and avoiding cutovers due to this situation. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a distributed storage dynamic data protection system according to the present invention;

[0026] Figure 2 This is a schematic diagram of the initial stage of the copy protection switching EC of the present invention;

[0027] Figure 3This is a schematic diagram of the final stage of the copy protection switching EC of this invention;

[0028] Figure 4 This is a schematic diagram illustrating the switching from a small-scale EC to a large-scale EC according to the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Storage engines generally consist of three main components: the storage engine client, the storage single-machine engine module, and the storage engine cluster management module.

[0031] Storage engine clients access data in a single engine according to the data protection methods determined by the storage engine cluster management or the storage engine designer. For example, a storage user may specify EC4+2, collapsed EC20+4:1, or replica protection mode in the cluster management GUI. The cluster management generates a view, and the protection information is synchronized to the engine client through view diffusion. The engine client then determines the data protection mode based on the view.

[0032] The storage single-machine engine provides local data persistence and persistent data access capabilities.

[0033] Storage engine cluster management determines the data protection mode and read / write methods to ensure data consistency and high-performance data access within the cluster, providing efficient data storage capabilities. The storage cluster management system provides a GUI and allows customers to select the protection mode. For example, if a customer selects a collapsed EC20+4:1 system with the fault domain being the rack, the storage cluster management system will generate a view based on the protection mode. The three main components within the storage system then determine which rack and which disks to read / write based on the PG settings in the view.

[0034] This invention provides a distributed storage dynamic data protection system, including a storage engine client, a storage single-machine engine module, and a storage engine cluster management module.

[0035] For example, the storage engine cluster management module is used to publish data views, determine the protection mode and read / write method of the data views, and spread the data views to the storage engine client and the storage single-machine engine module.

[0036] Specifically, read / write mode is used to divide a data view into read-only data and read-write output data. Migrated data in a data view is read-only data; non-migrated data in a data view is read-write output data. Protection mode is used to perform authoritative sharding on a data view. Authoritative shards include authoritative node disks and non-authoritative sets.

[0037] In addition, the data view is generated by the storage engine cluster management module.

[0038] For example, the storage engine client accesses the data view in the storage engine cluster management module through protected mode and synchronizes the data view to the storage single-machine engine module.

[0039] For example, the storage engine client writes to the data view in a read-write manner and migrates the migrateable data view through protected mode.

[0040] For example, the storage engine cluster management module switches the view of the migrated data and spreads it to the storage engine client and the storage single-machine engine module.

[0041] Specifically, the view switching process includes the following steps: selecting the data view to be switched and switching the protection mode; constructing a non-authoritative set by selecting nodes that intersect with the authoritative node disk in the switched data view; publishing the non-authoritative set as the first new view; initiating the migration command, marking the authoritative shards participating in the migration in the first new view as read-only, reading the data from the authoritative shards, and writing it to the non-authoritative set according to EC (Economic Consistency); after the migration, performing a consistency check on the first new view and forming a new set of authoritative shards; re-converting the data view to writable data, publishing the second new view, and disseminating the second new view to the cluster through the cluster view diffusion scheme.

[0042] In addition, the first new view is read-only during the migration, and no new data is written during the migration process.

[0043] Furthermore, the storage single-machine engines are deployed on general-purpose servers, and the storage engine cluster management is deployed as a cluster via Paxos or Raft protocols. The storage engine client is integrated into the product as an SDK by business applications (such as block, file, and object storage). Both the storage single-machine engines and the storage engine client have access to the storage cluster management, and the storage engine cluster management can manage all storage single-machine engines.

[0044] For example, the storage standalone engine module is used to provide local data persistence and persistent data access for storage engine clients.

[0045] On the other hand, this embodiment of the invention provides a distributed storage dynamic data protection method, including:

[0046] Publish the data view, determine its protection mode and read / write method, and distribute the data view to the storage engine client and storage single-machine engine module;

[0047] Access the data view in the storage engine cluster management module through protected mode, and synchronize the data view to the storage single-machine engine module;

[0048] Data is written to the data view via read / write, and the migrateable data view is migrated via protected mode;

[0049] The migrated data view is switched and then distributed to the storage engine client and the storage standalone engine module.

[0050] The specific embodiments of the above-mentioned distributed storage dynamic data protection method are as follows:

[0051] Taking a storage engine that uses views as the sole basis, and a star-structured, append-only storage engine as an example:

[0052] PG: Data Protection Group;

[0053] Authoritative Sharding: Shards within a PG that are considered reliable by the cluster management system, indicating that the data within the PG has reached consensus and has been kept online and alive by the cluster management system.

[0054] (1) The storage engine cluster management module publishes a view, which is quickly disseminated to the storage engine client and the storage single-machine engine module. A. PGs that need to be migrated are read-only; B. PGs that are not involved are read-write.

[0055] (2) Storage engine cluster management ensures that all shards designed for the PG during migration are read-only, and that the views have been synchronized to the corresponding storage single-machine engines. Since view control is implemented, view generation is uniformly handled by the cluster management system. At the start of migration, it is ensured that the views have been synchronized to the relevant single-machine engines (this can be achieved through a logical clock). Therefore, regardless of the version the client receives during the migration process, no new data can be written to that PG.

[0056] (3) When the storage engine client writes new data, it uses a writable PG.

[0057] (4) When a storage engine client needs to read data from a migrating PG, it uses the protection mode corresponding to the authoritative shard. Referring to the explanation of authoritative shards above, the set of authoritative shards in a PG corresponds to a replica. This is a certain protection mode of the EC. The replica method reads data through the replica, while the EC reads data by calculating the location of the data in the shard. In case of hardware failure, it performs a repair read.

[0058] (5) After the migration is completed, the storage engine cluster management switches the view and spreads it to the storage engine client and the storage single engine.

[0059] The entire view change process is as follows: Taking a node failure domain and a 3-replica switchover to EC 4+2 as an example:

[0060] 1. Currently, all PGs are in 3-player dungeons;

[0061] 2. Select a number of PGs (e.g., 2) at a time to switch protection modes;

[0062] 3. The selected PG selects 6 disks from another node (without overlap with the current authoritative node disks) according to EC 4+2, forming a non-authoritative set of EC 4+2;

[0063] 4. Publish the new view;

[0064] 5. Initiate the migration command, requiring the authoritative shards participating in the migration to be read-only. After reading the data from the authoritative shards, write it to the non-authoritative set in the EC (Extended Conversion) manner.

[0065] 6. After the migration is complete and the PG consistency check is performed, the EC 4+2 set replaces the 3-replica set to form a new authoritative shard set;

[0066] 7. The PG becomes writable again, a new view is published, and the view is spread to the cluster through a cluster view diffusion scheme, thereby realizing the change of the PG;

[0067] 8. Repeat steps 1 to 7 to complete the switchover of the entire cluster.

[0068] Because write-on-demand redirection is used and a new PG is immediately switched after a write failure, the read and write operations of upper-layer services are not affected. Since some PGs handle large volumes of data while others handle small volumes, cluster load balancing is considered separately from a cluster load balancing perspective.

[0069] By setting the PG (Post Group) to read-only during migration, no new data is written during the PG migration process, making the migration simple and reliable, and allowing migration to a high proportion of erasure coding. The storage engine dynamically switches from three-replica protection mode to EC (Erasure Coding) protection mode. The storage engine also dynamically switches from a low proportion EC (Erasure Coding) protection mode to EC (Erasure Coding) protection mode. During the dynamic switching process, the business is unaware of the changes.

[0070] Furthermore, this embodiment also provides a terminal device. The distributed storage dynamic data protection method involved in this embodiment is mainly applied to the terminal device, which can be a PC, portable computer, mobile terminal or other device with display and processing functions.

[0071] Specifically, the terminal device may include a processor (e.g., CPU), a communication bus, a user interface, a network interface, and memory. The communication bus is used to enable communication between these components; the user interface may include a display screen or an input unit such as a keyboard; the network interface may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface); the memory may be high-speed RAM or stable non-volatile memory, such as disk storage, and may also optionally be a storage device independent of the aforementioned processor.

[0072] The memory stores a readable storage medium, which contains a distributed storage dynamic data protection program. The processor can call the dynamic data protection program stored in the memory and execute the distributed storage dynamic data protection method provided in this embodiment of the invention.

[0073] Understandably, a readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example—but not limited to—an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage medium as used herein is not to be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0074] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0075] Computer program instructions used to perform operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0076] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A distributed storage dynamic data protection system, characterized in that, This includes a storage engine client, a single-machine storage engine module, and a storage engine cluster management module, among which: The storage engine cluster management module is used to publish data views, determine the protection mode and read / write mode of the data views, and spread the data views to the storage engine client and the storage single-machine engine module; the read / write mode is used to divide the data views into read-only data and read-write data; the protection mode is used to perform authoritative sharding on the data views; authoritative sharding includes authoritative node disks and non-authoritative sets; The storage engine client accesses the data view in the storage engine cluster management module through the protection mode, and synchronizes the data view to the storage single-machine engine module; The storage engine client writes to the data view using the read / write method and migrates the migrateable data view using the protection mode; The storage engine cluster management module performs view switching on the migrated data view and spreads it to the storage engine client and the storage single-machine engine module; The view switching process includes: selecting the data view to be switched and switching its protection mode; constructing a non-authoritative set by selecting nodes that intersect with the authoritative node disk in the switched data view; publishing the non-authoritative set as the first new view; initiating a migration command, marking the authoritative shards participating in the migration in the first new view as read-only, reading the data from the authoritative shards, and writing it to the non-authoritative set according to EC (Economic Consistency); after the migration, performing a consistency check on the first new view to form a new set of authoritative shards; re-making the data view writable, publishing the second new view, and disseminating the second new view to the cluster through a cluster view diffusion scheme.

2. The distributed storage dynamic data protection system according to claim 1, characterized in that, The storage single-machine engine module is used to provide local data persistence and persistent data access for the storage engine client.

3. The distributed storage dynamic data protection system according to claim 2, characterized in that, The migrated data in the data view is read-only data; the non-migrated data in the data view is readable and writable data.

4. The distributed storage dynamic data protection system according to claim 3, characterized in that, The data view is generated by the storage engine cluster management module.

5. A distributed storage dynamic data protection system according to claim 4, characterized in that, The first new view is read-only during the migration, and no new data is written during the migration process.

6. A method for dynamic data protection in distributed storage, characterized in that, include: Publish the data view, determine its protection mode and read / write method, and distribute the data view to the storage engine client and storage single-machine engine module; Read / write mode is used to divide the data view into read-only data and read-write data; protection mode is used to perform authoritative sharding on the data view; authoritative sharding includes authoritative node disks and non-authoritative sets; Access the data view in the storage engine cluster management module through protected mode, and synchronize the data view to the storage single-machine engine module; Data is written to the data view via read / write, and the migrateable data view is migrated via protected mode; The migrated data view is switched and then distributed to the storage engine client and the storage single-machine engine module; View switching includes: selecting the data view to be switched and switching the protection mode; selecting nodes that intersect with the authoritative node disk in the switched data view to construct a non-authoritative set; publishing the non-authoritative set as the first new view; initiating a migration command, marking the authoritative shards participating in the migration in the first new view as read-only, reading the data from the authoritative shards, and writing it to the non-authoritative set according to EC (Economic Consistency) method; after migration, performing a consistency check on the first new view to form a new set of authoritative shards; re-making the data view writable, publishing the second new view, and disseminating the second new view to the cluster through a cluster view diffusion scheme.

Images