Distributed storage method and system based on distributed metadata

By adopting CXL memory data synchronization protocol and shared memory pool technology in distributed storage systems, the poor user experience and scalability problems caused by CAP theory are solved, and a high-performance and high-scaling distributed storage system is realized to meet the needs of large-scale Data processing requirements.

CN119179441APending Publication Date: 2024-12-24GUOSHUJILIAN (SHANGHAI) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411294092.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-14
Publication Date
2024-12-24

AI Technical Summary

Technical Problem

Traditional distributed storage systems based on distributed metadata are limited by CAP theory and cannot have high consistency and availability at the same time, resulting in poor user experience and limited storage scalability.

Method used

Using CXL memory data synchronization protocol, metadata is stored between multiple nodes through a shared memory pool, metadata sharing in distributed storage systems is realized, load balancing and DNS polling are used to optimize data reading, ensuring the consistency of metadata and reliability.

Benefits of technology

It significantly improves the scalability and response performance of the storage system, reduces data access latency, meets the needs of high-performance computing and big data processing, and enhances the scalability of the system and the utilization of metadata storage resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119179441A_ABST
    Figure CN119179441A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a distributed storage method and system based on distributed metadata, the metadata storage space of the distributed storage of the distributed metadata in the embodiment of the invention can be expanded to 2TB to 16TB, so that the distributed system can share the unified metadata storage space, and the expansibility of the storage system is obviously improved. According to the technical scheme, each node can search full metadata in the shared storage pool, so that when the system reads the data, the polling search time of each node is shortened, and the response performance of the system is improved. From the distribution of storage nodes of the architecture, a decentralized architecture is adopted, object data is scattered and stored in a plurality of hard disks of different nodes, and unified namespace access is provided for the outside. And the metadata adopts a memory sharing architecture, so that each storage node can access the total metadata, and the response delay time of the storage system is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of data storage, and in particular to a distributed storage method and system based on distributed metadata. Background Art

[0002] In traditional distributed storage based on distributed metadata, metadata is data about data, such as the name, size, creation time, storage location, etc. of a file. In a distributed environment, metadata is usually managed by distributing it across multiple nodes.

[0003] With the explosive growth of data volume and the increasing demand for data processing speed, traditional distributed storage architectures are facing performance bottlenecks and scalability limitations. Problems such as insufficient memory bandwidth, high data transmission latency, and complex storage resource management have seriously affected the overall performance of data centers and high-performance computing systems.

[0004] In distributed storage applications, storage software is limited by the distributed CAP theory. When building a distributed storage system, users hope that the system they build can continue to provide more reliable services. At this time, it is necessary to meet the characteristics of data consistency, system availability, and network partition tolerance. However, the actual situation in the software architecture is that it swings between consistency and availability, and it is impossible to have high consistency and availability at the same time.

[0005] In the original practice, the single server of distributed storage was limited by memory expansion, resulting in limited shared memory of the storage system and insufficient storage space of the metadata server. If the memory cluster is continued to be built, the system architecture will be limited by the CAP theory, which will greatly reduce the user experience. Summary of the invention

[0006] To this end, an embodiment of the present invention provides a distributed storage method and system based on distributed metadata to solve the technical problems that the existing distributed storage is limited by the CAP theory, resulting in poor user experience and inability to simultaneously have high consistency and availability.

[0007] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, a distributed storage method based on distributed metadata is provided, the method being applied to a CXL memory data synchronization protocol and comprising:

[0009] S1. Obtain user demand and determine the type of demand. When the user demand is to write data, the client initiates writing data to the distributed storage network through the network. The distributed storage network obtains the data to be written and metadata as objects and stores the data to be written in the local storage medium, and saves the metadata in the shared memory pool.

[0010] S2. When the user needs to upload a file / object, the distributed storage network receives the upload request from the client and obtains the object data to be uploaded, generates a unique data key for the object data to be uploaded, combines the object data to be uploaded and the corresponding metadata, and writes them into the corresponding storage location;

[0011] S3. When the user demand is a data read request, the request is distributed to the server through load balancing or DNS polling. After receiving the request, the service point obtains the data key from the data read request and uses the data key to search for the metadata of the corresponding data object in the shared memory pool and sends the metadata and the corresponding object data to the client.

[0012] Furthermore, the information contained in the metadata includes: data key, data value and data attribute information.

[0013] Furthermore, the attribute information includes: name, size, modification time and storage type of the object.

[0014] Furthermore, when the user needs to write data, the client initiates writing data to the distributed storage network through the network, and the distributed storage network obtains the data to be written and metadata as objects and stores the data to be written in the local storage medium, and saves the metadata in the shared memory pool, including:

[0015] Get the data to be written and generate a unique identifier for each data object;

[0016] The corresponding metadata is saved in the shared memory pool of the corresponding node, and the data object is stored in the storage medium of the corresponding node.

[0017] Further, combining the object data to be uploaded and the corresponding metadata and writing them into a corresponding storage location includes:

[0018] Combining the object data to be uploaded and the corresponding metadata and obtaining a distributed architecture and a data placement strategy;

[0019] Using the distributed architecture and data placement strategy to obtain the server nodes and corresponding storage locations that need to be stored;

[0020] The object data to be uploaded and the metadata are written into a corresponding storage location together.

[0021] Furthermore, when reading data, it also includes performing coding verification on the coding block, and sending the metadata and corresponding object data to the client after coding verification.

[0022] According to a second aspect of an embodiment of the present invention, a distributed storage system based on distributed metadata is provided, the system comprising:

[0023] A user demand acquisition module is used to acquire user demand and determine the type of the demand. When the user demand is to write data, the client initiates writing data to the distributed storage network through the network. The distributed storage network acquires the data to be written and metadata as objects and stores the data to be written in the local storage medium, and saves the metadata in the shared memory pool;

[0024] The upload module is used for receiving the upload request from the client and obtaining the object data to be uploaded when the user needs to upload a file / object, generating a unique data key for the object data to be uploaded, combining the object data to be uploaded and the corresponding metadata and writing them into the corresponding storage location;

[0025] The reading module is used to distribute the request to the server through load balancing or DNS polling when the user demand is a data reading request. After receiving the request, the service point obtains the data key from the data reading request and uses the data key to search the metadata of the corresponding data object in the shared memory pool and send the metadata and the corresponding object data to the client.

[0026] Furthermore, the system further comprises:

[0027] The network module consists of a business application network and a distributed storage network;

[0028] The storage node module consists of multiple storage devices, each of which is connected to the shared memory pool through cables;

[0029] Distributed storage management module, used for resource management, task scheduling and data distribution;

[0030] Shared memory pool, which enables the host to access the memory connected to the device through the CXL memory data synchronization protocol;

[0031] CXL backend network, a network shared by storage nodes and shared memory pool nodes.

[0032] The embodiments of the present invention have the following advantages:

[0033] The metadata storage space of the distributed storage of the distributed metadata in the embodiment of the present invention can be expanded to 2TB to 16TB in size, so that the distributed system shares a unified metadata storage space, significantly improving the scalability of the storage system. Each node can search for the full amount of metadata in the shared storage pool, so that when the system reads data, it reduces the polling search time of each node and improves the system response performance. From the distribution of the storage nodes of the architecture, it can be seen that a decentralized architecture is adopted, and the object data is scattered and stored in multiple hard disks of different nodes, providing a unified namespace access to the outside world. The metadata adopts a memory sharing architecture, so that each storage node can access the full amount of metadata, reducing the response delay time of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0035] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0036] Figure 1 A schematic diagram of the logical structure of a distributed storage system based on distributed metadata provided by an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a distributed storage method based on distributed metadata provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the deployment of traditional distributed storage based on distributed metadata provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of deployment of distributed storage based on distributed metadata in a distributed storage method based on distributed metadata provided by an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a deployment structure in a distributed storage method based on distributed metadata provided by an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of the hardware internal structure design of a shared CXL.mem memory pool in a distributed storage method based on distributed metadata provided by an embodiment of the present invention;

[0042] Figure 7 A schematic diagram of the CXL principle in a distributed storage method based on distributed metadata provided by an embodiment of the present invention;

[0043] Figure 8 A schematic diagram of the PCIEport principle in a distributed storage method based on distributed metadata provided by an embodiment of the present invention;

[0044] Figures 9 to 12 A schematic diagram of the DDRport principle in a distributed storage method based on distributed metadata provided by an embodiment of the present invention;

[0045] Figures 13 to 15 A schematic diagram of the SSD port principle in a distributed storage method based on distributed metadata provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] With the explosive growth of data volume and the increasing demand for data processing speed, traditional distributed storage architectures are facing performance bottlenecks and scalability limitations. Problems such as insufficient memory bandwidth, high data transmission latency, and complex storage resource management have seriously affected the overall performance of data centers and high-performance computing systems.

[0048] In distributed storage applications, storage software is limited by the distributed CAP theory. When building a distributed storage system, users hope that the system they build can continue to provide more reliable services. At this time, it is necessary to meet the characteristics of data consistency, system availability, and network partition tolerance. However, the actual situation in the software architecture is that it swings between consistency and availability, and it is impossible to have high consistency and availability at the same time.

[0049] In traditional distributed storage based on distributed metadata, metadata is data about data, such as the name, size, creation time, storage location, etc. of the file. In a distributed environment, metadata is usually managed on multiple nodes, such as Figure 3 shown.

[0050] When a client initiates writing data to distributed storage, the storage system writes the data and metadata together as an object, without the need to use a separate metadata storage node or library. Metadata contains various information about the data object, such as the data key (the unique identifier of the object in the storage pool, similar to the file path), the data value (that is, the uploaded data itself, similar to the file content), and other data attributes (such as the file modification time, storage type, etc.).

[0051] When writing data, the storage system generates a unique identifier for each data object and stores the relevant metadata together with the data object. This ensures that the data and its corresponding metadata are always associated and consistent.

[0052] When a client uploads a file or object, the storage system receives the client's upload request, obtains the object data and related information to be uploaded, and generates a unique data key for the object. The object data and the corresponding metadata are then combined. Based on the storage system's distributed architecture and data placement strategy, the server nodes and specific storage locations where the object is stored are determined. Finally, the object data and metadata are written together to the corresponding storage location.

[0053] From the distribution of data and metadata in the architecture, it is not difficult to see that a decentralized shared-nothing architecture is adopted, in which object data is broken up and stored on multiple hard disks in different nodes, providing external access to a unified namespace.

[0054] When a client initiates a data read request, the request is distributed to each node of the storage service through load balancing or DNS polling. After receiving the request, the server searches for the corresponding data object in the local or distributed storage according to the key of the data in the request (similar to the file path). After finding the data object, the metadata of the data object is obtained. The metadata contains various attribute information of the object, such as the name, size, modification time, storage type, etc. of the object.

[0055] The server uses the information in the metadata to read the data, such as determining the storage location and encoding method of the data. When reading data, the storage system may perform encoding verification on the encoding block to ensure the correctness of each encoding. Finally, the data and related metadata are returned to the client.

[0056] In order to solve the technical problem that the above-mentioned distributed storage is limited by the CAP theory, resulting in poor user experience and the inability to have high consistency and availability at the same time.

[0057] refer to Figure 1The embodiment of the present invention discloses a distributed storage system based on distributed metadata, which includes: a user demand acquisition module 1; an upload module 2; and a reading module 3.

[0058] Furthermore, the system also includes: a network module 4, which is composed of a business application network and a distributed storage network; a storage node module 5, which is composed of multiple storage devices, and each storage device is connected to a shared memory pool through a cable; a distributed storage management module 6, which is used for resource management, task scheduling and data allocation; a shared memory pool 7, which enables the host to access the memory connected to the device through the CXL memory data synchronization protocol; and a CXL backend network 8, which is a network shared by storage nodes and shared memory pool nodes.

[0059] Business application network: A network shared by enterprise services, user terminals, management terminals, etc. Devices that support Ethernet, RoCE, IB, and other types of networks can be used.

[0060] Distributed storage network: A network shared by various types of nodes (management nodes, metadata nodes, data nodes, etc.) of a distributed storage system. Network devices that support Ethernet, RoCE, IB, etc. can be used.

[0061] The storage device of the storage node module 5 can be an SSD, HDD or other new storage media, which supports hot plugging and automatic failure switching of storage devices to improve the reliability of the system.

[0062] The distributed storage management module 6 adopts a consistent hashing algorithm or other distributed algorithms to achieve uniform distribution of data and load balancing, and supports dynamic expansion and contraction of storage nodes to adapt to business changes.

[0063] Each storage node device in the shared memory pool 7 can share a unified metadata space to improve the scalability of the system, utilize the low latency characteristics of the CXL.Mem protocol to improve the system communication performance, and the unified metadata space improves the distributed storage system's data information query performance.

[0064] Corresponding to the above-disclosed distributed storage system based on distributed metadata, the embodiment of the present invention further discloses a distributed storage method based on distributed metadata. The following describes in detail a distributed storage method based on distributed metadata disclosed in the embodiment of the present invention in combination with the above-described distributed storage system based on distributed metadata.

[0065] In traditional distributed storage systems, the scalability of the file system is limited due to the bottleneck of the metadata server. The present invention uses the CXL.Mem protocol to achieve a shared memory of up to 2TB, so that the data servers in the distributed file system can share a 2TB metadata space, thereby significantly improving the scalability of the system.

[0066] The low latency characteristics of the CXL.Mem protocol greatly improve communication performance, further reduce the loss of the communication part, and enhance the expansion capability of the system. The present invention provides an innovative solution and reference design standard to optimize the performance and scalability of distributed storage systems.

[0067] refer to Figure 2 and Figure 4 The present invention discloses a distributed storage method based on distributed metadata, which is applied to the CXL memory data synchronization protocol and includes:

[0068] S1. Obtain user demand and determine the type of demand. When the user demand is to write data, the client initiates writing data to the distributed storage network through the network. The distributed storage network obtains the data to be written and metadata as objects and stores the data to be written in the local storage medium, and saves the metadata in the shared memory pool.

[0069] When a client initiates writing data to distributed storage through a storage network, the storage system writes the data and metadata together as objects, without the need to use a separate metadata storage node or library. The data object of each node is stored in the local storage medium, and the metadata is saved in the shared memory pool. Metadata contains various information about the data object, such as the key of the data (the unique identifier of the object in the storage pool, similar to the file path), the value of the data (that is, the uploaded data itself, similar to the file content), and other attributes of the data (such as the modification time of the file, storage type, etc.).

[0070] S2. When the user needs to upload a file / object, the distributed storage network receives the upload request from the client and obtains the object data to be uploaded, generates a unique data key for the object data to be uploaded, combines the object data to be uploaded and the corresponding metadata, and writes them into the corresponding storage location.

[0071] When a client uploads a file or object, the storage system receives the client's upload request, obtains the object data and related information to be uploaded, and generates a unique data key for the object. The object data and the corresponding metadata are then combined. Based on the storage system's distributed architecture and data placement strategy, the server nodes and specific storage locations where the object is stored are determined. Finally, the object data and metadata are written together to the corresponding storage location.

[0072] S3. When the user demand is a data read request, the request is distributed to the server through load balancing or DNS polling. After receiving the request, the service point obtains the data key from the data read request and uses the data key to search for the metadata of the corresponding data object in the shared memory pool and sends the metadata and the corresponding object data to the client.

[0073] When a client initiates a data read request through the storage network, the request is distributed to a node on the storage service side through load balancing or DNS polling. After receiving the request, the server searches for the metadata of the corresponding data object in the shared memory pool according to the key of the data in the request (similar to the file path). The metadata contains various attribute information of the object, such as the name, size, modification time, storage type, etc.

[0074] In the innovative distributed storage based on distributed metadata, after the system generates metadata, it is saved in the metadata memory block of the shared memory pool. Under the CXL.mem protocol, the memory gaps are opened up to form a unified memory space. The stored metadata will be managed uniformly, thus ensuring the consistency of metadata. The metadata storage node obtains metadata information by accessing the shared memory pool.

[0075] Furthermore, the information contained in the metadata includes: data key, data value and data attribute information.

[0076] Furthermore, the attribute information includes: name, size, modification time and storage type of the object.

[0077] From the distribution of storage nodes in the architecture, it is not difficult to see that the decentralized architecture is adopted, and the object data is scattered and stored in multiple hard disks of different nodes, providing unified namespace access to the outside world. The metadata adopts a memory sharing architecture, so that each storage node can access the full amount of metadata, reducing the response delay time of the storage system.

[0078] Furthermore, when the user's demand is to write data, the client initiates writing data to the distributed storage network through the network. The distributed storage network obtains the data to be written and metadata as objects and stores the data to be written in the local storage medium, and saves the metadata in the shared memory pool, including: obtaining the data to be written and generating a unique identifier for each data object; saving the corresponding metadata in the shared memory pool of the corresponding node, and storing the data object in the storage medium of the corresponding node.

[0079] When writing data, the storage system generates a unique identifier for each data object and saves the related metadata to the shared memory pool of the node where the data object is stored in the storage medium. This ensures that the data and its corresponding metadata are always associated and consistent.

[0080] Furthermore, the object data to be uploaded and the corresponding metadata are combined and written into a corresponding storage location, including: combining the object data to be uploaded and the corresponding metadata and obtaining a distributed architecture and a data placement strategy; using the distributed architecture and the data placement strategy to obtain a server node to be stored and a corresponding storage location; writing the object data to be uploaded and the metadata together into a corresponding storage location.

[0081] Furthermore, when reading data, it also includes performing coding verification on the coding block, and sending the metadata and corresponding object data to the client after coding verification.

[0082] The server uses the information in the metadata to read the data, such as determining the storage location and encoding method of the data. When reading data, the storage system may perform encoding verification on the encoding block to ensure the correctness of each encoding. Finally, the data and related metadata are returned to the client.

[0083] refer to Figure 5 The business application network is a network that provides storage services to enterprise users. The distributed storage network is used to build the internal network of the distributed system. The network consists of multiple storage node devices. The CXL backend network is used to connect the host and the shared CXL.mem memory pool for metadata storage. Storage nodes are devices with different roles in the distributed system and can be flexibly deployed according to the actual distributed system requirements and business needs.

[0084] Use hybrid resource cards to store metadata, such as Figure 6 As shown in the figure, the host sees the continuous SSD space through the CXL switch and writes metadata to the SSD. The MMU in the CXL hybrid resource card passes the write operation to the NVME controller and also to the MC. The metadata will be written to the SSD and DDR at the same time. When reading metadata, the MMU passes the read command to the MC, and the data is read directly from the DDR. In general, the write latency is equivalent to SSD write, and the read latency is equivalent to DDR read. In the process of frequent read queries, the performance is greatly improved. The read latency is reduced from 100us to 1us.

[0085] refer to Figures 7 and 8, the pins with the network name FPGA_PCIE_Finger* provide the chip with the signal of the PCIE bus. TX means sending from the chip to the gold finger, and RX means sending from the gold finger to the chip. This type of pin is a high-speed serial serdes pin. The level type is CML level, which can provide a signal rate of 32Gbps. Here, the PCIE port bus implements the PCIE bus of GEN5X8LANE. Provides a system-side interface for the entire hybrid resource card.

[0086] refer to Figures 9 to 12 , DDR port is the interface provided by the chip to interconnect with RDIMM. DDR4_CH* is the signal of DDR4 channel, and the level type is POD type. The signals here are divided into two categories, data signal and address control signal. DQ* and DQS* are data signals, and the rest are address and control signals. This chip provides 2 sets of DDR4 buses for memory expansion, and each set of DDR4 buses connects two RDIMM memory sticks. A total of 4 RDIMM memories are provided, and by plugging in 64GB RDIMM, 256GB memory capacity can be provided.

[0087] SSD port is the PCIE port provided by the chip to connect to NVMe SSD. Figures 3 to 15 FPGA_PCIE* is the PCIE signal pin of the chip, which is a high-speed serial serdes signal with a level type of CML. Two sets of GEN4X4LANE PCIE buses and NVMe SSD interconnection are provided here. PCIE_M.2* signals are PCIE signals of NVMe SSD. These signals are connected to FPGA_PCIE* to realize storage resource expansion on the chip.

[0088] The present invention has the following beneficial effects:

[0089] 1) Significantly improve storage performance, reduce data access latency, and meet the needs of high-performance computing and big data processing;

[0090] 2) Enhance the scalability of the system to easily cope with the growing amount of data and business needs;

[0091] 3) Improve the utilization of metadata storage resource space and reduce costs;

[0092] 4) Ensure the reliability and consistency of metadata.

[0093] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A distributed storage method based on distributed metadata, characterized in that: The method is applied to the CXL memory data synchronization protocol, and includes: S1. Obtain user demand and determine the type of demand. When the user demand is to write data, the client initiates writing data to the distributed storage network through the network. The distributed storage network obtains the data to be written and metadata as objects and stores the data to be written in the local storage medium, and saves the metadata in the shared memory pool. S2. When the user needs to upload a file / object, the distributed storage network receives the upload request from the client and obtains the object data to be uploaded, generates a unique data key for the object data to be uploaded, combines the object data to be uploaded and the corresponding metadata, and writes them into the corresponding storage location; S3. When the user demand is a data read request, the request is distributed to the server through load balancing or DNS polling. After receiving the request, the service point obtains the data key from the data read request and uses the data key to search for the metadata of the corresponding data object in the shared memory pool and sends the metadata and the corresponding object data to the client.

2. A distributed storage method based on distributed metadata as claimed in claim 1, characterized in that: The information contained in metadata includes: data key, data value and data attribute information.

3. A distributed storage method based on distributed metadata as claimed in claim 2, characterized in that: The attribute information includes: the name, size, modification time and storage type of the object.

4. A distributed storage method based on distributed metadata as claimed in claim 3, characterized in that: When the user needs to write data, the client initiates writing data to the distributed storage network through the network. The distributed storage network obtains the data to be written and metadata as objects and stores the data to be written in the local storage medium, and saves the metadata in the shared memory pool, including: Get the data to be written and generate a unique identifier for each data object; The corresponding metadata is saved in the shared memory pool of the corresponding node, and the data object is stored in the storage medium of the corresponding node.

5. A distributed storage method based on distributed metadata as claimed in claim 4, characterized in that: Combining the object data to be uploaded and the corresponding metadata and writing them into a corresponding storage location includes: Combining the object data to be uploaded and the corresponding metadata and obtaining a distributed architecture and a data placement strategy; Using the distributed architecture and data placement strategy to obtain the server nodes and corresponding storage locations that need to be stored; The object data to be uploaded and the metadata are written into a corresponding storage location together.

6. A distributed storage method based on distributed metadata as claimed in claim 5, characterized in that: When reading data, it also includes encoding verification of the encoding block, and after the encoding verification, the metadata and the corresponding object data are sent to the client.

7. A distributed storage system based on distributed metadata, characterized in that: The system comprises: A user demand acquisition module is used to acquire user demand and determine the type of the demand. When the user demand is to write data, the client initiates writing data to the distributed storage network through the network. The distributed storage network acquires the data to be written and metadata as objects and stores the data to be written in the local storage medium, and saves the metadata in the shared memory pool; The upload module is used for receiving the upload request from the client and obtaining the object data to be uploaded when the user needs to upload a file / object, generating a unique data key for the object data to be uploaded, combining the object data to be uploaded and the corresponding metadata and writing them into the corresponding storage location; The reading module is used to distribute the request to the server through load balancing or DNS polling when the user demand is a data reading request. After receiving the request, the service point obtains the data key from the data reading request and uses the data key to search the metadata of the corresponding data object in the shared memory pool and send the metadata and the corresponding object data to the client.

8. A distributed storage system based on distributed metadata as claimed in claim 7, characterized in that: The system further comprises: The network module consists of a business application network and a distributed storage network; The storage node module consists of multiple storage devices, each of which is connected to the shared memory pool through cables; Distributed storage management module, used for resource management, task scheduling and data allocation; shared memory pool, through CXL memory data synchronization protocol, the host can access the memory connected to the device; CXL backend network, a network shared by storage nodes and shared memory pool nodes.