Distributed storage data read-write method and device, equipment, medium and program product

Abstract

The invention relates to the technical field of storage, and discloses a distributed storage data reading and writing method and device, equipment, a medium and a program product. A data block distribution view corresponding to each storage pool is generated according to the distributed cluster information, the data block distribution view comprises a plurality of protection groups, and each protection group comprises corresponding disk information; and querying the data block distribution view according to the received service data, and determining the protection group distributed by each data block so as to perform data read operation or write operation. According to the method, complex concepts of various partitions and DHT calculation of multiple partitions are avoided, the concept of a protection group is abstracted in a view, mapping is made in advance, operation is replaced by table look-up, and in an ultra-high-performance scene, less calculation resources are occupied, and the software stack time delay is lower.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a distributed storage data reading and writing method, device, equipment, medium and program product. Background Art

[0002] Distributed storage is a data storage technology that stores data in multiple nodes. Each node can be a computer, server or other storage device. When storing data, it is usually divided into multiple blocks or objects and distributed on different storage nodes. Each node can independently store and access data.

[0003] Distributed storage technology is now widely used in traditional data centers, public cloud data centers, private cloud data centers and hyper-converged workstation products. The distributed storage engine provides high-performance, high-reliability and efficient data access capabilities for distributed storage and is the core component of distributed storage.

[0004] Currently, there are many implementation schemes for distributed storage engines, such as Google's publicly available distributed storage engine GFS (Google File System, GFS for short), which can provide an append-write file system. GFS is managed by a unified master to manage the metadata of file-to-space mapping. Chunk information is queried by the master on specific chunk servers at startup. The master manages chunk information for the entire cluster. When the cluster becomes larger, managing chunk information for the entire cluster not only occupies master resources but also has slow response.

[0005] Another widely used one is ceph. Ceph is a distributed storage that uses a distributed algorithm (CRUSH algorithm) for data distribution. It can distribute data objects according to the weight of each storage device, so that the distribution is close to uniform distribution. Ceph obtains a set of OSD (Object Storage Device, referred to as OSD) lists through calculation for reading and writing. Since these OSD lists are calculated and ceph is a new write that overwrites, in order to ensure data consistency and prevent write disorder and data inconsistency, there are PG locks and object locks, and the software stack is complex. When a cluster fails, PG must make a consistency decision to determine the new OSD master in PG, which will provide read and write services. In certain unstable network scenarios, long-term suspended IO may cause business interruption. In addition, the logic of ceph's in-situ overwriting is not suitable for the scenario after the popularization of SSD media, resulting in the inability to exert the performance of SSD media. Summary of the invention

[0006] In view of this, the present invention provides a distributed storage data reading and writing method, device, equipment, medium and program product to solve the problems of complex software stack and high computing resource occupancy in the prior art.

[0007] In a first aspect, the present invention provides a distributed storage data reading and writing method, the method comprising:

[0008] Get distributed cluster information;

[0009] Generate a data block distribution view corresponding to each storage pool according to the distributed cluster information, the data block distribution view includes multiple protection groups, and each protection group includes corresponding disk information;

[0010] The data block distribution view is queried according to the received business data to determine the protection group of each data block distribution to perform data read or write operations.

[0011] The present invention does not have the complex concepts of various partitions and DHT calculations of multiple partitions. The concept of protection groups is abstracted in the view, mapping is done in advance, and calculations are replaced by table lookups. In ultra-high performance scenarios, less computing resources are occupied and the software stack delay is lower.

[0012] In an optional implementation, querying a data block distribution view according to received service data and determining a protection group for each data block distribution to perform a data write operation includes:

[0013] Determine the data block identifier corresponding to the business data;

[0014] Determine the protection group identifier mapped to the data block identifier according to the data block identifier;

[0015] According to the protection group ID, query the disks included in the protection group ID;

[0016] Write business data to the corresponding disk.

[0017] In this embodiment, chunk id can be used directly after application like virtual built-in address, physical space is mapped when writing, thin allocation is supported naturally, simpler software stack is provided through simple read-write logic, single write lock logic and fault decision logic design, thus higher reliability is achieved. It is more adaptable to the redirection feature of SSD media when writing, and the full performance of SSD is brought into play under lower configuration CPU, thus saving cost.

[0018] In an optional implementation, when at least one disk fails, the method further includes:

[0019] Determine whether there is an authoritative set in the protection group. The authoritative set is a set of shards whose disk status remains normal.

[0020] If there is an authoritative set, determine whether the authoritative set includes a recovery set and the number of shards in the recovery set is greater than or equal to the number of data shards. The recovery set is the shard to which the business data is successfully written.

[0021] If the authoritative set includes a recovery set and the number of shards in the recovery set is greater than or equal to the number of data shards, the recovery set is selected for data recovery.

[0022] In an optional implementation, in the absence of an authoritative set, all shards in the protection group are selected for data recovery.

[0023] The selection of the data consistency consensus set is simple and reliable, and it is easy to mathematically prove the correctness of the set selection, making fault data consistency processing more reliable.

[0024] In an optional implementation, the method further includes: after the data blocks in the disk are migrated, reclaiming the migrated data blocks from the data block distribution view.

[0025] In this embodiment, data blocks are recycled, which can effectively prevent data block leakage and reduce performance loss caused by data redundancy, thereby improving the overall responsiveness of the system.

[0026] In an optional implementation, before querying the data block distribution view according to the received business data, the process includes: determining whether the client request information is consistent with the view information in the data block distribution view.

[0027] In a second aspect, the present invention provides a distributed storage data reading and writing device, the device comprising:

[0028] Collection module, used to obtain distributed cluster information;

[0029] A generation module, used to generate a data block distribution view corresponding to each storage pool according to the distributed cluster information, the data block distribution view includes multiple protection groups, and each protection group includes corresponding disk information;

[0030] The query module is used to query the data block distribution view according to the received business data, determine the protection group of each data block distribution, and perform data read or write operations.

[0031] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the distributed storage data reading and writing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for enabling a computer to execute the distributed storage data reading and writing method of the first aspect or any corresponding embodiment thereof.

[0033] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the distributed storage data reading and writing method of the first aspect or any corresponding embodiment thereof.

[0034] It should be noted that the distributed storage data reading and writing device, computer equipment and computer-readable storage medium provided by the present invention correspond to the above-mentioned distributed storage data reading and writing method. Therefore, for the beneficial effects of the distributed storage data reading and writing device, computer equipment and computer-readable storage medium, please refer to the description of the corresponding beneficial effects of the distributed storage data reading and writing method above, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 is a flow chart of a distributed storage data reading and writing method according to an embodiment of the present invention;

[0037] Figure 2 is a schematic diagram of the structure of a storage pool according to an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of a logical topology according to an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of a physical hardware topology according to an embodiment of the present invention;

[0040] Figure 5 is a schematic diagram of a data block distribution view according to an embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of the composition of a chunk id according to an embodiment of the present invention;

[0042] Figure 7 is a schematic diagram of the composition of pg id according to an embodiment of the present invention;

[0043] Figure 8is a schematic diagram of an authoritative set being a non-empty set according to an embodiment of the present invention;

[0044] Fig. 9 is a schematic diagram of an authoritative set being an empty set according to an embodiment of the present invention;

[0045] Fig.10 is a schematic diagram of chunk recycling according to an embodiment of the present invention;

[0046] Fig.11 is a schematic diagram of a chunk client according to an embodiment of the present invention;

[0047] Fig.12 is a structural block diagram of a distributed storage data reading and writing device according to an embodiment of the present invention;

[0048] Fig.13 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0050] The following are some explanations of the names of the nouns that appear in this invention:

[0051] EC: Erasure Code, which means erasure code, is a coding technology that adds m copies of data to n copies of original data, and can restore the original data through any n copies of data among the n+m copies. Mathematically, encoding is expressed as constructing a multivariate linear equation, and decoding is solving the multivariate linear equation. Further abstraction is matrix operation. Current correction coding technology is mainly divided into two categories: one is correction coding based on Galois field operation (such as RS); the other is erasure code (LDPC) based on "XOR" operation.

[0052] Chunk: means data block, which is uniquely identified by chunk id. It provides a byte stream space that supports appending, similar to the write-time mapping function of virtual address memory. The maximum size is usually 128M. It provides chunk operation interfaces for reading, appending, closing, deleting, and querying length.

[0053] Pool: used to describe a virtual logical address space with continuous chunk ids. Pool id is used as a unique identifier. When a business uses a chunk, it needs to apply for a chunk id from the pool.

[0054] PG: Protect Group, meaning logical protection group, uniquely identified by pg id. In order to simplify the management of chunk id in pool, PG is abstracted between chunk and pool, and chunk id is mapped to fixed pg id through a simple and reliable algorithm. PG describes a group of independent virtual hard disks, which are organized together by EC or replicas through the protection strategy formulated by pool to provide reliability guarantee, provide logical protection group of read and write space to the outside, and simplify cluster management.

[0055] Strongly consistent write: Data written to all shards is returned successfully, ensuring that data will not be lost under any circumstances.

[0056] Chunkserver: refers to a stand-alone storage server, which is used in the present invention to describe a stand-alone storage server, manage the storage space on the stand-alone server, and manage the stand-alone byte stream chunk space.

[0057] Chunkmaster: means cluster management center, which is responsible for a series of cluster management functions such as data distribution, data balancing, fault recovery, cluster expansion and contraction, cluster monitoring, virtual space allocation and virtual space conversion to physical space. Different storage engines have different naming methods, also known as master and CM (cluster master).

[0058] chunkclient: means cluster read and write SDK (Software Development Kit).

[0059] In the current distributed storage data reading and writing methods, there are often the following problems:

[0060] 1. Fault handling takes a long time. For example, in order to ensure data reliability, when a Ceph storage cluster encounters a fault, it needs to decide on an authoritative node before reading and writing, and this time is uncontrollable.

[0061] 2. Ceph's reading and writing methods require locking, which results in the inability to bring out the performance of the new hardware. Compared with the leading products in this field, Ceph's latency is many times higher.

[0062] 3. To achieve write-once-multiple-reads, many products use relatively complex control methods. For example, Ceph downgrades writing when encountering hardware failures, and the background catches up with the data differences, which reduces reliability.

[0063] In view of this, according to an embodiment of the present invention, a distributed storage data reading and writing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0064] In this embodiment, a distributed storage data reading and writing method is provided, which can be executed by a server, a terminal, a mobile terminal and other devices. Figure 1 is a flow chart of a distributed storage data reading and writing method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0065] Step S101, obtaining distributed cluster information.

[0066] Step S102: Generate a data block distribution view corresponding to each storage pool according to the distributed cluster information. The data block distribution view includes multiple protection groups PG, and each protection group includes corresponding disk information.

[0067] Step S103, querying the data block distribution view according to the received business data, and determining the protection group of each data block chunk distribution, so as to perform a data read operation or a data write operation.

[0068] The above process can be performed by the chunk master in the server and other devices. The chunk master controls the data distribution of the entire cluster. It can be a raft (a distributed consistency algorithm) cluster or a paxos (a distributed consistency algorithm) cluster, or it can rely on the cluster management capabilities provided by an external raft cluster (such as etcd (a highly available distributed key-value database)). As the metadata management center of the entire cluster, the chunk master can receive the distributed cluster information reported by the chunkserver, including: distributed cluster physical disk information and server information, and report the cluster's unique disk identifier to the chunkserver. The chunk master does not directly manage the chunks of the entire cluster, but manages the abstract protection group PG above the chunk.

[0069] Based on the physical disk information and server information reported by the chunk server, a logical topology can be constructed. Figure 2 As shown, the cluster is divided into several storage pools, and PG management is performed in each storage pool. PG management includes data balancing, fault handling, PG migration, generating data block distribution view cdmap, chunk allocation, disk selection, view analysis, etc. Further, refer to Figure 3 As shown, it is the logical topology structure corresponding to the storage pool: starting from root, there are multiple logical levels including AZ (data center), Row (the same group of aggregation switches), Rack (the same group of access switches), Host, and Vdisk. Figure 4 The physical hardware topology of the cluster.

[0070] You can create multiple sets of protection strategies on the logical topology, such as Row fault domain + 3 replicas, Rack fault domain + EC 4+2:1, Host fault domain + EC 4+2, etc. In addition, in the process of creating a storage pool, create a PG according to the protection strategy on the topology, such as Rack fault domain + EC 4+2:1. In the process of creating the PG, fill the disk into the authoritative shard set of the PG according to the protection strategy. After the storage pool is successfully created, the cdmap is generated. After the cdmap is generated, the chunk master notifies the chunk server of the storage pool designed by the chunk server during the process of keeping the physical disk alive, and the cdmap needs to be updated. When there is a hardware failure in the cluster, such as disk, node, or network failure, the cluster's fault handling will also change the cdmap according to the hardware failure to ensure the accuracy of chunk allocation.

[0071] The data block distribution view corresponding to one of the storage pools refers to Figure 5 As shown, the data block distribution view mainly consists of four parts: cdmap_ver, pool information, PG information (i.e. Figure 5 Available PG set in) and disk information (i.e. Figure 5 The cluster disk collection in .

[0072] Pool information includes: pool_id, pool_ver, avail_chunk_cnt, pg_cnt, avail_pg_cnt, flags. pool_id indicates a virtual logical space with continuous chunk ids, including 2 to the 56th power of virtual chunk spaces; pool_ver, combined with the logic of the SDK, indicates whether the current chunk space belongs to the pool space under the current view time. Chunk id is mapped to chunk through simple calculations to achieve data reading and writing. avail_chunk_cnt indicates the number of available chunks, pg_cnt indicates the number of pgs, avail_pg_cnt indicates the number of available pgs, and flags records some marks: such as allowing non-authoritative reads, etc.

[0073] PG information includes: pg_id, pg_ver, status, init_set, auth_set, r_set. pg_id indicates the unique protection group space under a certain logical clock in the cluster. The protection group breaks up the physical space of multiple disks in the cluster and organizes them into protection spaces according to the protection strategy. The protection space may be multiple copies, EC 4+2 or EC 20+4, etc.; pg_ver indicates the protection group version number, which is used for cluster failure recovery. When restoring a specific protection group (PG), it ensures that the PGs restored by each chunk server service are consistent; status is used to mark the current read and write status of the PG; init_set indicates the pg original disk set; auth_set indicates the authoritative disk set; r_set indicates the recovery disk set; m_set indicates the migration disk set. The original disk information included in each protection group PG can be determined when selecting disks for balancing.

[0074] The disk information records the unique information of all disks in the cluster. The vdisk_id corresponds to the vdisk_id recorded in the PG. That is, the identifier 1 of vdisk1 in the disk information corresponds to 1 in init_set[(1,13,25,37)(49,61)] in the PG information.

[0075] Finally, the data block distribution view is used to query the data block distribution and determine the final data read and write address.

[0076] The present invention provides a storage engine read-write method with write-time redirection, which does not have the complex concepts of various partitions and DHT calculations of multiple partitions, abstracts the concept of protection groups in the view, makes mappings in advance, replaces calculations with table lookups, occupies less computing resources in ultra-high performance scenarios, and has lower software stack latency.

[0077] The distributed storage data reading and writing method provided by the present invention can be used as a data reading and writing method for the storage base of a public cloud and a private cloud, and can be used in an append-write scenario. It can also be used as a data reading and writing method for a software-defined storage product, and can be used in an append-write scenario. It can also be used as a persistent data reading and writing method for a cluster, and can be used in an append-write scenario.

[0078] In some optional implementations, querying a data block distribution view according to received business data and determining a protection group for each data block distribution to perform a data write operation includes:

[0079] Determine the data block identifier corresponding to the business data, that is, the chunk id in this embodiment;

[0080] According to the data block identifier, determine the protection group identifier mapped to the data block identifier, that is, pg id in this embodiment;

[0081] According to the protection group ID, query the disk disk included in the protection group ID;

[0082] Write the business data to the corresponding disk.

[0083] Reference Figure 6 As shown in the figure, the chunk id consists of a 1-byte pool_id and a 7-byte chunk index. The chunkid represents a virtual logical space. The maximum value is determined by the view, such as 256M. When used, it is mapped to the actual physical space. It is 8 bytes long and unique within the cluster. The chunk index represents the protection group to which the chunk belongs. Figure 7 As shown, pg id consists of 1-byte pool_id and 3-byte pg index. pg_index=chunk index&(pg_cnt-1); pg_cnt is an integer power of 2.

[0084] Business data is written by applying for chunk id through chunk client. Chunk client caches chunk id. If there is no chunk id cache, writing is not allowed.

[0085] When reading and writing business data, the following information must be carried: cdmap_ver, pool_ver, pg_ver, shard_ver, and chunk_id. Among them: cdmap_ver is the version number of cdmap. Chunk client and chunk server interact to determine whether the local view version is the latest, so as to achieve the purpose of view diffusion. pool_ver is the pool version number, which is globally unique in the entire cluster. It checks whether the cdmap of chunk client and the cdmap of chunkserver are in the same pool. pg_ver is the PG version number. It checks whether the PG in the cdmap of chunk client and the PG in the cdmap of chunk server are consistent. shard_ver is the shard version number in PG. It is the version number when PG selects a disk. It is globally unique and is used to avoid the problem that the same disk is selected at different time points and different shard positions during recovery in PG, resulting in the indistinguishability of chunks. chunk_id is the unique identifier of the chunk. The information contains pool_id and pg_id, which are the most important information.

[0086] According to the protection group identifier pg id, query the disk disk corresponding to the protection group identifier in cdmap, such as: PG (ProtectGroup) pg_id1, pg_ver, status, init_setl (1, 13, 25, 37) (49, 61)], auth_set [(1, 13, 25.37) (49, 61)], r setl.m set [], where 1, 13, 25, 37 are the disks corresponding to pg_id1, and write the data to the corresponding disk. The database chunk space in this embodiment is similar to a memory virtual address, which can be used after application. When the data is actually written, the actual physical space is mapped. The write permission of the chunk is controlled by the memory of the client chunk client SDK. When the pool version changes, the chunk client sdk directly clears the chunk memory, prohibits all chunks from being written, and ensures that only a single thread writes to the chunk. In addition, the size of the chunk can be synchronized to the chunk client SDK through the management granularity limit of the disk by the chunk server through the CDMap generated by the chunk master.

[0087] The reading based on cdmap is as follows: according to the chunk id passed in by the business data, the PG is calculated according to the cdmap, and the data at the specified location is read according to the protection strategy and sharding information in the PG. When it is found that the PG is not readable or writable, it is necessary to wait for 1 second and try to update whether it is based on the latest cdmap. In the event of an abnormality in the cluster, the pause is normal and will not cause pressure on the chunkmaster.

[0088] In this embodiment, chunk id can be used directly after application like virtual built-in address, physical space is mapped when writing, and thin (a way to dynamically allocate disk) allocation is naturally supported. Through the design of simple read-write logic, single write lock logic and fault decision logic, a simpler software stack is provided, thus having higher reliability. It is more adaptable to the redirection feature of SSD media when writing, and the full performance of SSD is brought into play under a lower configuration CPU, thus saving costs.

[0089] In some optional implementations, when at least one disk fails, the method further includes:

[0090] Determine whether there is an authoritative set in the protection group. The authoritative set is a set of shards whose disk status has always remained normal. The authoritative shards are shards whose disk status has remained normal after the PG reaches full shards and consensus is reached.

[0091] If there is an authoritative set, determine whether the authoritative set includes a recovery set and the number of shards in the recovery set is greater than or equal to the number of data shards. The recovery set is the shard to which the business data is successfully written.

[0092] If the authoritative set includes a recovery set and the number of shards in the recovery set is greater than or equal to the number of data shards, the recovery set is selected for data recovery.

[0093] The storage cluster contains multiple firmwares, and the annual damage rate of the disk is <= 5%. However, failures still occur in the cluster. After a failure, follow the following principles to select a set of disks that can make PG consistency decisions. This can ensure the availability of data when encountering a single point of failure and improve the reliability of the overall system.

[0094] Reference Figure 8 As shown, the total set: A = {n|n is the initial shard created by PG: data shard + verification shard}; the authoritative set: B = {x|x is the shard that has been in normal status since PG reached a consensus}; the recoverable set: C = {y|y is the shard in pg, the number of sets >= the number of data shards}.

[0095] When the authoritative shard contains data on successful business writes, when the length consensus set selects C∈D and D∈B, the D set will definitely be able to consensus on the data on successful business writes.

[0096] The above is for the situation where some disks and nodes in the cluster cannot be recovered after failure.

[0097] In some optional embodiments, in the absence of an authoritative set, all shards in the protection group are selected for data recovery.

[0098] Reference Fig. 9 As shown, the total set: A = {n|n is the initial shard created by PG: data shard + verification shard}; the authoritative set: B = {x|x is an empty set} and the recoverable set: C = {yly is the shard within pg, the number of sets >= the number of data shards}.

[0099] When the authoritative shard is an empty set, when the length consensus set selects D=A, that is, the full set of PG, the full set must contain the data of successful business writes and can be consensused.

[0100] The above is for scenarios where the cluster can recover after a large-scale failure, especially a network outage.

[0101] That is, when there are enough authoritative shards, the set of consistency decisions selects {x|x is the authoritative shard, and the number of authoritative shards>=the number of data shards}. When there are not enough authoritative shards, the set of consistency decisions selects the entire set. Clarifying the disk set for consistency decisions can ensure the availability of data when encountering a single point of failure and improve the reliability of the overall system. The selection of the data consistency consensus set is simple and reliable, and it is easy to prove the correctness of the set selection through mathematics, making the consistency processing of faulty data more reliable.

[0102] In some optional embodiments, the method further comprises:

[0103] After the data blocks in the disk are migrated, the corresponding data blocks are reclaimed from the data block distribution view.

[0104] There are two ways to reclaim chunk space: one is to reclaim it by deleting the chunk by the business; the other is to reclaim the chunk after the cdmap changes. The business reclaims the chunk after receiving the chunk deletion command, and the consistency negotiation after a failure will continue to delete it.

[0105] In this embodiment, the recycling is based on cdmap, which includes the distribution of chunks. When the utilization rate of a disk is too high, it is necessary to migrate the PG in the disk, that is, to migrate the chunks mapped in the PG. After the chunk is migrated, the cdmap changes, and the migrated chunks can be recycled through the chunk server. Fig.10 As shown, when the chunk in the 8th PG in pool1 needs to be recovered, the PG1.8 can be migrated to other disks, such as disk 4, and PG1.3 can be migrated from disk 4. Then the chunk server reclaims the chunk space belonging to PG1.3 according to cdmap.

[0106] In this embodiment, data blocks are recycled, which can effectively prevent data block leakage and reduce performance loss caused by data redundancy, thereby improving the overall responsiveness of the system.

[0107] In some optional implementations, before querying the data block distribution view according to the received business data, it includes: determining whether the client request information is consistent with the view information in the data block distribution view. In this embodiment, the client request information includes the client's chunk client cdmap version number, pool version number, data sharding location, etc., as follows.

[0108] The chunk master controls the changes of all version numbers in the cluster. All version numbers in chunk servers and chunk client components come from the chunk master. The chunk master can also control whether the data on the chunk server needs to be recycled in the background through CDMAP. In addition, the chunk master also provides the function of allocating chunk_id. The business applies for chunk_id from the chunk master through the chunk client. Application is not allowed without the chunk client.

[0109] The chunk server manages the storage media on the storage nodes in the cluster and is the stand-alone storage engine in the present invention. It is responsible for discovering the local storage media, bringing the local storage media into management, and synchronizing the information to the cluster management chunkmaster. After the chunk master generates the cdmap view, it updates the cdmap to the local. After receiving the read and write request from the client chunk client, the following operations are also required:

[0110] Check the chunk client cdmap version number and the local version number. The larger version number is more authoritative and does not affect reading and writing. If the cdmap version number is lower, pull a new cdmap from the chunk master.

[0111] Check whether the pool_ver in the cdmap is consistent to determine whether the pool has been deleted. If the pool_ver is inconsistent, it will affect reading and writing, and the lagging end of the cdmap needs to be updated. Check whether the pool_ver in the cdmap is consistent to prevent the chunk client from not updating the cdmap after the pool_id is reused. When the pool_ver changes, clear the chunk id memory set, and the chunk id held by the upper-layer business will not be writable.

[0112] Calculate the PG ID based on the chunk ID, get the PG status, and check whether the chunk is writable, readable, or not readable or writable.

[0113] When reading a chunk, it is important to compare the shard ver with the actual shard ver on the chunk, check whether shard_ver is consistent, and determine whether the data shard location considered by the client is consistent with the shard location considered locally. If shard_ver is inconsistent, it will affect reading and writing, and the lagging end of cdmap needs to be updated. Shard_ver can be used to distinguish whether a disk is repeatedly selected into pg, but at different shard locations in EC.

[0114] When pg_ver mainly checks and recovers, PG requires that all chunk servers in PG are at the same logical clock.

[0115] When the chunk server manages chunks, it needs to persist pool_ver along with the chunks to facilitate future deletion.

[0116] In addition, regarding the chunk client, the chunk client is the SDK provided by the read and write method mentioned in this embodiment for the service, and mainly provides the following functions, see Fig.11 As shown:

[0117] Provides an interface for opening a pool. When a business opens a pool, it synchronizes the cdmap of the pool from the chunk master.

[0118] Provides an interface for applying for chunks. When a business applies for a chunk id, the chunk id is stored in the current working thread.

[0119] When the business thread writes data to a chunk, it checks whether the chunk id written by the customer is in the memory of the current thread. If it does not exist, writing is not allowed, thus ensuring that only a single thread writes to the chunk.

[0120] The chunk client retries writing according to the situation. If the chunk client cannot retry writing, an error code is returned to suggest that the business change the chunk to write.

[0121] Chunk client provides an interface for closing chunks. Closing chunks is not restricted and can be done by any client. Closing operations are idempotent. Writing is not allowed after a chunk is closed.

[0122] The chunk client provides a query length interface. The queried length is not guaranteed to be the latest length and may be outdated.

[0123] The chunk client provides a deletion interface. After the deletion is called, the chunk becomes untrustworthy and cannot be used again.

[0124] When a chunk client uses a pool for business, it first updates the cdmap by opening the pool. Then, it checks whether the cdmap version number is outdated through operations such as chunk reading and writing and applying for chunk ID, and then updates the cdmap.

[0125] The chunk client applies for a chunk id on the pool. Since the chunk master does not directly manage chunks, the chunk master generates a chunk id on the PG in the pool.

[0126] The present invention proposes a view-based data reading and writing method. All servers install the service chunkserver of the present invention and a unified cluster management chunkmaster. After the chunkserver monitors the hardware to generate a view, the business reads and writes data in the cluster through the chunk client SDK provided by the present invention. The present invention is simpler and more efficient through the data reading and writing method. When using the RMDA network, the read and write latency can reach 50us on NVMe. Unified view management, each component obtains data distribution through a similar table lookup. When adding a protection mode, it only needs to change the table lookup method of the view to expand different protection modes, which is more flexible. As a self-developed distributed engine, the reading and writing method can replace the currently used open source distributed engine ceph.

[0127] In the present embodiment, a distributed storage data reading and writing device is also provided, and the device is used to implement the above-mentioned embodiment and preferred implementation mode, and the description has been made no further. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0128] This embodiment provides a distributed storage data reading and writing device, such as Fig.12 As shown, the device comprises:

[0129] Collection module 201, used to obtain distributed cluster information;

[0130] A generating module 202, configured to generate a data block distribution view corresponding to each storage pool according to the distributed cluster information, wherein the data block distribution view includes multiple protection groups, and each protection group includes corresponding disk information;

[0131] The query module 203 is used to query the data block distribution view according to the received business data, and determine the protection group of each data block distribution to perform a data read operation or a data write operation.

[0132] In some optional implementations, the query module 302 is also used to determine the data block identifier corresponding to the business data; determine the protection group identifier mapped by the data block identifier based on the data block identifier; query the disk included in the protection group identifier based on the protection group identifier; and write the business data to the corresponding disk.

[0133] In some optional embodiments, the device further comprises:

[0134] The fault handling module is used to determine whether there is an authoritative set in the protection group. The authoritative set is a set of shards whose disk status has always remained normal. If there is an authoritative set, it is determined whether the authoritative set includes a recovery set and the number of shards in the recovery set is greater than or equal to the number of data shards. The recovery set is the shard to which business data has been successfully written. If the authoritative set includes a recovery set and the number of shards in the recovery set is greater than or equal to the number of data shards, the recovery set is selected for data recovery. If there is no authoritative set, all shards in the protection group are selected for data recovery.

[0135] The recycling module is used to recycle the migrated data blocks from the data block distribution view after the data blocks in the disk are migrated.

[0136] The checking module is used to determine whether the client request information is consistent with the view information in the data block distribution view.

[0137] The distributed storage data reading and writing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0138] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0139] The embodiment of the present invention also provides a computer device having the above Fig.12 The distributed storage data reading and writing device shown.

[0140] See also Fig.13 , Fig.13 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.13 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.13 A processor 10 is taken as an example.

[0141] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0142] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0143] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0145] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0146] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0147] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0148] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A distributed storage data reading and writing method, characterized in that: The method comprises: Get distributed cluster information; Generate a data block distribution view corresponding to each storage pool according to the distributed cluster information, wherein the data block distribution view includes multiple protection groups, and each protection group includes corresponding disk information; The data block distribution view is queried according to the received business data to determine the protection group of each data block distribution so as to perform a data read operation or a data write operation.

2. The method according to claim 1, characterized in that The querying the data block distribution view according to the received business data to determine the protection group of each data block distribution to perform the data write operation includes: Determine a data block identifier corresponding to the business data; Determine, according to the data block identifier, a protection group identifier mapped to the data block identifier; According to the protection group identifier, query the disk included in the protection group identifier; The business data is written to the corresponding disk.

3. The method according to claim 1, characterized in that In the event of at least one disk failure, the method further comprises: Determine whether there is an authoritative set in the protection group, where the authoritative set is a set of shards whose disk status always remains normal; In the case where the authoritative set exists, determining whether the authoritative set includes a recovery set and the number of shards in the recovery set is greater than or equal to the number of data shards, and the recovery set is the shards to which the business data is successfully written; When the authoritative set includes the recovery set and the number of shards in the recovery set is greater than or equal to the number of data shards, the recovery set is selected for data recovery.

4. The method according to claim 3, characterized in that In the absence of the authoritative set, all shards in the protection group are selected for data recovery.

5. The method according to claim 1, characterized in that The method further comprises: After the data blocks in the disk are migrated, the migrated data blocks are recovered from the data block distribution view.

6. The method according to claim 1, characterized in that Before querying the data block distribution view according to the received business data, the method includes: It is determined whether the client request information is consistent with the view information in the data block distribution view.

7. A distributed storage data reading and writing device, characterized in that: The device comprises: Collection module, used to obtain distributed cluster information; A generation module, configured to generate a data block distribution view corresponding to each storage pool according to the distributed cluster information, wherein the data block distribution view includes a plurality of protection groups, and each protection group includes corresponding disk information; The query module is used to query the data block distribution view according to the received business data, and determine the protection group of each data block distribution to perform data reading or writing operations.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the distributed storage data reading and writing method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the distributed storage data reading and writing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that It includes computer instructions, and the computer instructions are used to enable a computer to execute the distributed storage data reading and writing method according to any one of claims 1 to 6.

Images