Distributed storage method and system based on centralized metadata

Abstract

The embodiment of the invention discloses a distributed storage method and system based on centralized metadata, after metadata is generated by a metadata master node, the metadata is stored in a shared memory pool, and the metadata master node and slave nodes can obtain metadata information by accessing the shared memory pool. Through a CXL.Mem protocol, a shared memory as high as 2TB is realized, so that a data server in the distributed file system can share a metadata space of the 2TB, and the expansibility of the system is remarkably improved. Due to the low-delay characteristic of the CXL.Mem protocol, the communication performance is greatly improved, the loss of a communication part is further reduced, and the expansion capability of the system is enhanced; according to the method, the storage performance can be remarkably improved, the data access delay is reduced, the expansibility of the system is enhanced while the requirements of high-performance calculation and big data processing are met, and the continuously increasing data volume and service requirements can be easily coped with; the utilization rate of metadata storage resource space is improved, and the cost is reduced; and the reliability and the consistency of the metadata are ensured.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer storage technology, and in particular to a distributed storage method and system based on centralized metadata. Background Art

[0002] With the explosive growth of data volume and the increasing demand for data processing speed, traditional distributed storage architectures are facing performance bottlenecks and scalability limitations. Problems such as insufficient memory bandwidth, high data transmission latency, and complex storage resource management have seriously affected the overall performance of data centers and high-performance computing systems.

[0003] In distributed storage applications, storage software cannot build a perfect distributed storage system due to the limitations of the distributed CAP theory. When building a distributed storage system, users hope that the system they build can continue to provide more reliable services, so it needs to meet the characteristics of data consistency, system availability, and network partition tolerance. However, the actual situation in software architecture is a swing between consistency and availability, that is, the choice between CP and AP.

[0004] Since the single server of distributed storage is limited by memory expansion, the shared memory of the storage system is limited, and the storage space of the metadata server is the bottleneck. If the memory cluster is built, the system architecture will be limited by the CAP theory, which will greatly reduce the user experience. Summary of the invention

[0005] To this end, an embodiment of the present invention provides a distributed storage method and system based on centralized metadata to solve the technical problems that the prior art is susceptible to limitations of memory expansion and CAP theory and has poor user experience.

[0006] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0007] According to a first aspect of an embodiment of the present invention, a distributed storage method based on centralized metadata is provided, the method being applied to a shared memory pool and comprising:

[0008] S1. Obtain user demand and determine the type of the demand. When the user demand is a write data request, the client initiates a write data request through the network and distributes the write data request to the metadata storage node through load balancing. After receiving the write data request, the metadata storage node first checks the permission of the write data request and obtains the data file to be written, allocates data blocks to the file to be written according to the current block configuration information, generates data block allocation information and sends it to the client.

[0009] S2. After receiving the data block allocation information, the client establishes a pipeline connection with a group of data storage nodes of the first data block, prepares to write data, and sends data to the first data storage node in the pipeline. After receiving the data, the data storage node writes the received data into a local storage medium and transmits the received data to the next data storage node in the pipeline. The data storage node sends heartbeat information to the metadata storage node at a preset period. The metadata storage node updates the data block status using the heartbeat information. When all data blocks have successfully written data and the metadata has been updated, the complete metadata is saved in the shared memory pool.

[0010] S3. When the user demand is a data read request, the data read request is distributed to the metadata storage node through load balancing and the corresponding file path is obtained. After receiving the request from the client, the metadata storage node uses the corresponding file path to search for the corresponding metadata in the shared memory pool and obtain the data block information of the file data to be read, extracts the data block location information from the data block information and sends it to the client;

[0011] S4. After receiving the data block location information, the client establishes a connection with the corresponding data storage node, and the user terminal reads the data block through the data storage node.

[0012] Further, after receiving the write data request, the metadata storage node first checks the permission of the write data request and obtains the data file to be written, and also includes:

[0013] The client initiates a write data request through the network, and the metadata storage node receives the write data request and determines whether the write data request is legal;

[0014] If the data write request is a legitimate request, allocating a data block for the file to be written according to the current block configuration information;

[0015] If the write data request is an illegal request, the write data request is rejected.

[0016] Furthermore, the metadata storage node is used to record the allocated data block information in the corresponding metadata.

[0017] Furthermore, the information in the metadata includes: a data block identifier and a data storage node location.

[0018] Furthermore, the metadata storage node updates the data block status in the metadata using the heartbeat information.

[0019] Furthermore, the data block information includes a data block identifier and data storage node location information of the data block.

[0020] Furthermore, when the current data block storage node fails / cannot respond, the replica location information corresponding to the metadata storage node is obtained, and the replica data block is read from the available data storage node.

[0021] According to a second aspect of an embodiment of the present invention, there is provided a distributed storage system based on centralized metadata, the system comprising:

[0022] A demand classification module is used to obtain user demand and determine the type of the demand. When the user demand is a write data request, the client initiates a write data request through the network and distributes the write data request to the metadata storage node through load balancing. After receiving the write data request, the metadata storage node first checks the permission of the write data request and obtains the data file to be written, allocates data blocks to the file to be written according to the current block configuration information, generates data block allocation information and sends it to the client;

[0023] A data writing module is used for establishing a pipeline connection with a group of data storage nodes of the first data block after the client receives the data block allocation information, and sending data to the first data storage node in the pipeline after preparing to write data. After receiving the data, the data storage node writes the received data into a local storage medium and transmits the received data to the next data storage node in the pipeline. The data storage node sends heartbeat information to the metadata storage node at a preset period, and the metadata storage node updates the data block status using the heartbeat information. When all data blocks have successfully written data and the metadata has been updated, the complete metadata is saved in the shared memory pool;

[0024] The first reading module is used for distributing the data reading request to the metadata storage node through load balancing and obtaining the corresponding file path when the user demand is a data reading request. After receiving the request from the client, the metadata storage node uses the corresponding file path to search for the corresponding metadata in the shared memory pool and obtain the data block information of the file data to be read, extracting the data block location information from the data block information and sending it to the client;

[0025] The second reading module is used for the client to establish a connection with the corresponding data storage node after receiving the data block location information, and the user end reads the data block through the data storage node.

[0026] The embodiments of the present invention have the following advantages:

[0027] After the metadata master node in the embodiment of the present invention generates metadata, it saves the metadata to the shared memory pool. Both the metadata master node and the slave node can obtain metadata information by accessing the shared memory pool. Through the CXL.Mem protocol, up to 2TB of shared memory is achieved, so that the data servers in the distributed file system can share a 2TB metadata space, thereby significantly improving the scalability of the system. The low latency characteristics of the CXL.Mem protocol greatly improve communication performance, further reduce the loss of the communication part, and enhance the expansion capability of the system; the present invention can significantly improve storage performance, reduce data access latency, meet the needs of high-performance computing and big data processing, and enhance the scalability of the system, and can easily cope with the growing amount of data and business needs; improve the utilization rate of metadata storage resource space, reduce costs; and ensure the reliability and consistency of metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0029] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0030] Figure 1 A schematic diagram of the logical structure of a distributed storage system based on centralized metadata provided by an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of a distributed storage method based on centralized metadata provided by an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of the deployment of traditional distributed storage based on centralized metadata provided by an embodiment of the present invention;

[0033] Figure 4 A schematic diagram of deployment of distributed storage based on distributed metadata in a distributed storage method based on centralized metadata provided by an embodiment of the present invention;

[0034] Figure 5A schematic diagram of a deployment structure in a distributed storage method based on centralized metadata provided by an embodiment of the present invention;

[0035] Figure 6 A schematic diagram of the internal structure design of the shared CXL.mem memory pool hardware in a distributed storage method based on centralized metadata provided by an embodiment of the present invention;

[0036] Figure 7 A schematic diagram of the CXL principle in a distributed storage method based on centralized metadata provided by an embodiment of the present invention;

[0037] Figure 8 A schematic diagram of the PCIEport principle in a distributed storage method based on centralized metadata provided by an embodiment of the present invention;

[0038] Figures 9 to 12 A schematic diagram of the DDRport principle in a distributed storage method based on centralized metadata provided by an embodiment of the present invention;

[0039] Figures 13 to 15 A schematic diagram of the SSD port principle in a distributed storage method based on centralized metadata provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In traditional distributed storage based on centralized metadata, metadata is data about data, such as the file name, size, creation time, storage location, and other information. In a centralized environment, metadata is usually centralized and managed in a node or cluster. Centralized metadata management has a dedicated master node responsible for maintaining metadata. The traditional centralized metadata deployment structure is shown in the figure. Figure 3 shown.

[0042] In order to solve the technical problem that the above-mentioned method of avoiding radar interference through passive resistance cannot ensure the effectiveness of avoiding interference.

[0043] refer to Figure 1 The embodiment of the present invention discloses a distributed storage system based on centralized metadata, which includes: a demand classification module 1; a data writing module 2; a first reading module 3; and a second reading module 4.

[0044] Business application network: A network shared by enterprise services, user terminals, management terminals, etc. Devices that support Ethernet, RoCE, IB, and other types of networks can be used.

[0045] Distributed storage network: A network shared by various types of nodes (management nodes, metadata nodes, data nodes, etc.) of a distributed storage system. Network devices that support Ethernet, RoCE, IB, etc. can be used.

[0046] Storage node module: It consists of multiple storage devices, connected by cables to share the CXL.mem memory pool:

[0047] The storage device can be an SSD, HDD or other new storage media.

[0048] Supports hot-swapping and automatic fault switching of storage devices to improve system reliability.

[0049] Distributed storage management module: responsible for resource management, task scheduling and data allocation of the entire distributed storage system.

[0050] Use consistent hashing algorithms or other distributed algorithms to achieve uniform data distribution and load balancing.

[0051] Supports dynamic expansion and contraction of storage nodes to adapt to business changes.

[0052] Shared CXL.mem memory pool: Through the CXL.mem protocol, the host can access the memory connected to the device.

[0053] Each storage node device can share a unified metadata space to improve the scalability of the system.

[0054] Take advantage of the low latency characteristics of the CXL.Mem protocol to improve system communication performance.

[0055] A unified metadata space improves the performance of querying data information in distributed storage systems.

[0056] CXL backend network: The network shared by the storage nodes of the distributed storage system and the nodes that share the CXL.mem memory pool.

[0057] Corresponding to the above-disclosed distributed storage system based on centralized metadata, an embodiment of the present invention further discloses a distributed storage method based on centralized metadata. The following describes in detail a distributed storage method based on centralized metadata disclosed in an embodiment of the present invention in combination with the above-described distributed storage system based on centralized metadata.

[0058] refer to Figure 2and Figure 3 The present invention discloses a distributed storage method based on centralized metadata, which is applied to a shared memory pool and includes:

[0059] S1. Obtain user demand and determine the type of the demand. When the user demand is a write data request, the client initiates a write data request through the network and distributes the write data request to the metadata storage node through load balancing. After receiving the write data request, the metadata storage node first checks the permission of the write data request and obtains the data file to be written, allocates data blocks to the file to be written according to the current block configuration information, generates data block allocation information and sends it to the client.

[0060] When a client initiates a write request to a distributed storage system through the network, the request is distributed to the metadata storage node through load balancing. After receiving the request, the metadata storage node of the storage system first checks the permissions of the request to determine whether it has the right to create a file in the specified path, and then allocates a series of data blocks to the file based on the file size and the current block configuration. Each data block usually has a default size.

[0061] The metadata storage node records the allocated data block information in its metadata, which includes the data block identifier, the data storage node location, etc. and synchronizes it to the metadata storage slave node.

[0062] S2. After receiving the data block allocation information, the client establishes a pipeline connection with a group of data storage nodes of the first data block, prepares to write data and sends data to the first data storage node in the pipeline. After receiving the data, the data storage node writes the received data into the local storage medium and transmits the received data to the next data storage node in the pipeline. The data storage node sends heartbeat information to the metadata storage node at a preset period. The metadata storage node updates the data block status using the heartbeat information. When all data blocks are successfully written and the metadata is updated, the complete metadata is saved to the shared memory pool.

[0063] After receiving the allocation information of the data block, the client establishes a pipeline connection with a group of data storage nodes of the first data block and prepares to write data. The client starts to send data to the first data storage node in the pipeline. After receiving the data, the data storage node writes the data to the local storage medium on the one hand, and passes the data to the next data storage node in the pipeline on the other hand. The data storage node periodically sends heartbeat messages to the metadata storage node to report the write status of the data block. The metadata storage node updates the data block status in the metadata based on these messages.

[0064] When all data blocks are successfully written and the metadata is updated, the client notifies the metadata storage node that the writing is complete. The complete metadata is then saved to the shared memory pool to ensure metadata consistency.

[0065] S3. When the user demand is a data read request, the data read request is distributed to the metadata storage node through load balancing and the corresponding file path is obtained. After receiving the request from the client, the metadata storage node uses the corresponding file path to search for the corresponding metadata in the shared memory pool and obtain the data block information of the file data to be read, extracts the data block location information from the data block information and sends it to the client.

[0066] When a client initiates a data read request, the request is distributed to the metadata storage node through load balancing, and the file path to be read is specified. After the metadata storage node receives the client's request, it searches for the corresponding metadata in the shared memory pool according to the file path and obtains the file's data block information, including the identifier of the data block and the location of the data storage node that stores the data blocks.

[0067] S4. After receiving the data block location information, the client establishes a connection with the corresponding data storage node, and the user terminal reads the data block through the data storage node.

[0068] After the metadata master node generates metadata, it saves the metadata to the shared memory pool. Both the metadata master node and the slave node can obtain metadata information by accessing the shared memory pool.

[0069] Furthermore, after receiving the write data request, the metadata storage node first checks the permission of the write data request and obtains the data file to be written, and also includes: the client initiates a write data request through the network, the metadata storage node receives the write data request and determines whether the write data request is legal; if the write data request is a legal request, allocates data blocks to the file to be written according to the current block configuration information; if the write data request is an illegal request, rejects the write data request.

[0070] Furthermore, the metadata storage node is used to record the allocated data block information in the corresponding metadata.

[0071] Furthermore, the information in the metadata includes: a data block identifier and a data storage node location.

[0072] Furthermore, the metadata storage node updates the data block status in the metadata using the heartbeat information.

[0073] Furthermore, the data block information includes a data block identifier and data storage node location information of the data block.

[0074] Furthermore, when the current data block storage node fails / cannot respond, the replica location information corresponding to the metadata storage node is obtained, and the replica data block is read from the available data storage node.

[0075] The metadata storage node returns the location information of the data block to the client. Based on the received data block location information, the client establishes a connection with the data storage node that stores the corresponding data block. The client then reads the data block from the data storage node. If a data storage node fails or fails to respond, the client will try to read the replica data block from other available data storage nodes based on the other replica location information provided by the metadata storage node.

[0076] refer to Figure 5 The business application network is a network that provides storage services to enterprise users. The distributed storage network is used to build the internal network of the distributed system. The network consists of multiple storage node devices. The CXL backend network is used to connect the host and the shared CXL.mem memory pool for metadata storage. Storage nodes are devices with different roles in the distributed system and can be flexibly deployed according to the actual distributed system requirements and business needs.

[0077] Use hybrid resource cards to store metadata, such as Figure 6 As shown in the figure, the host sees the continuous SSD space through the CXL switch and writes metadata to the SSD. The MMU in the CXL hybrid resource card passes the write operation to the NVME controller and also to the MC. The metadata will be written to the SSD and DDR at the same time. When reading metadata, the MMU passes the read command to the MC, and the data is read directly from the DDR. In general, the write latency is equivalent to SSD write, and the read latency is equivalent to DDR read. In the process of frequent read queries, the performance is greatly improved. The read latency is reduced from 100us to 1us.

[0078] refer to Figures 7 and 8 , the pins with the network name FPGA_PCIE_Finger* provide the chip with the signal of the PCIE bus. TX means sending from the chip to the gold finger, and RX means sending from the gold finger to the chip. This type of pin is a high-speed serial serdes pin. The level type is CML level, which can provide a signal rate of 32Gbps. Here, the PCIE port bus implements the PCIE bus of GEN5X8LANE. Provides a system-side interface for the entire hybrid resource card.

[0079] refer to Figures 9 to 12, DDR port is the interface provided by the chip to interconnect with RDIMM. DDR4_CH* is the signal of DDR4 channel, and the level type is POD type. The signals here are divided into two categories, data signal and address control signal. DQ* and DQS* are data signals, and the rest are address and control signals. This chip provides 2 sets of DDR4 buses for memory expansion, and each set of DDR4 buses connects two RDIMM memory sticks. A total of 4 RDIMM memories are provided, and by plugging in 64GB RDIMM, 256GB memory capacity can be provided.

[0080] SSD port is the PCIE port provided by the chip to connect to NVMe SSD. Figures 13 to 15 FPGA_PCIE* is the PCIE signal pin of the chip, which is a high-speed serial serdes signal with a level type of CML. Two sets of GEN4X4LANE PCIE buses and NVMe SSD interconnection are provided here. PCIE_M.2* signals are PCIE signals of NVMe SSD. These signals are connected to FPGA_PCIE* to realize storage resource expansion on the chip.

[0081] The embodiments of the present invention have the following beneficial effects:

[0082] 1) Significantly improve storage performance, reduce data access latency, and meet the needs of high-performance computing and big data processing;

[0083] 2) Enhance the scalability of the system to easily cope with the growing amount of data and business needs;

[0084] 3) Improve the utilization of metadata storage resource space and reduce costs;

[0085] 4) Ensure the reliability and consistency of metadata.

[0086] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0087] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A distributed storage method based on centralized metadata, characterized in that: The method is applied to a shared memory pool and comprises: S1. Obtain user demand and determine the type of the demand. When the user demand is a write data request, the client initiates a write data request through the network and distributes the write data request to the metadata storage node through load balancing. After receiving the write data request, the metadata storage node first checks the permission of the write data request and obtains the data file to be written, allocates data blocks to the file to be written according to the current block configuration information, generates data block allocation information and sends it to the client. S2. After receiving the data block allocation information, the client establishes a pipeline connection with a group of data storage nodes of the first data block, prepares to write data, and sends data to the first data storage node in the pipeline. After receiving the data, the data storage node writes the received data into a local storage medium and transmits the received data to the next data storage node in the pipeline. The data storage node sends heartbeat information to the metadata storage node at a preset period. The metadata storage node updates the data block status using the heartbeat information. When all data blocks have successfully written data and the metadata has been updated, the complete metadata is saved in the shared memory pool. S3. When the user demand is a data read request, the data read request is distributed to the metadata storage node through load balancing and the corresponding file path is obtained. After receiving the request from the client, the metadata storage node uses the corresponding file path to search for the corresponding metadata in the shared memory pool and obtain the data block information of the file data to be read, extracts the data block location information from the data block information and sends it to the client; S4. After receiving the data block location information, the client establishes a connection with the corresponding data storage node, and the user terminal reads the data block through the data storage node.

2. A distributed storage method based on centralized metadata as claimed in claim 1, characterized in that: After receiving the write data request, the metadata storage node first checks the permission of the write data request and obtains the data file to be written, and further includes: The client initiates a write data request through the network, and the metadata storage node receives the write data request and determines whether the write data request is legal; If the data write request is a legitimate request, allocating a data block for the file to be written according to the current block configuration information; If the write data request is an illegal request, the write data request is rejected.

3. A distributed storage method based on centralized metadata as claimed in claim 2, characterized in that: The metadata storage node is used to record the allocated data block information in the corresponding metadata.

4. A distributed storage method based on centralized metadata as claimed in claim 3, characterized in that: The information in the metadata includes: the data block identifier and the data storage node location.

5. A distributed storage method based on centralized metadata as claimed in claim 4, characterized in that: The metadata storage node updates the data block status in the metadata using the heartbeat information.

6. A distributed storage method based on centralized metadata as claimed in claim 5, characterized in that: The data block information includes a data block identifier and data storage node location information of the data block.

7. A distributed storage method based on centralized metadata as claimed in claim 6, characterized in that: When the current data block storage node fails / cannot respond, the replica location information corresponding to the metadata storage node is obtained, and the replica data block is read from the available data storage node.

8. A distributed storage system based on centralized metadata, characterized in that: The system comprises: A demand classification module is used to obtain user demand and determine the type of the demand. When the user demand is a write data request, the client initiates a write data request through the network and distributes the write data request to the metadata storage node through load balancing. After receiving the write data request, the metadata storage node first checks the permission of the write data request and obtains the data file to be written, allocates data blocks to the file to be written according to the current block configuration information, generates data block allocation information and sends it to the client; A data writing module is used for establishing a pipeline connection with a group of data storage nodes of the first data block after the client receives the data block allocation information, and sending data to the first data storage node in the pipeline after preparing to write data. After receiving the data, the data storage node writes the received data into a local storage medium and transmits the received data to the next data storage node in the pipeline. The data storage node sends heartbeat information to the metadata storage node at a preset period, and the metadata storage node updates the data block status using the heartbeat information. When all data blocks have successfully written data and the metadata has been updated, the complete metadata is saved in the shared memory pool; The first reading module is used for distributing the data reading request to the metadata storage node through load balancing and obtaining the corresponding file path when the user demand is a data reading request. After receiving the request from the client, the metadata storage node uses the corresponding file path to search for the corresponding metadata in the shared memory pool and obtain the data block information of the file data to be read, extracts the data block location information from the data block information and sends it to the client; The second reading module is used for the client to establish a connection with the corresponding data storage node after receiving the data block location information, and the user end reads the data block through the data storage node.

Images