Distributed storage system and distributed reading and writing method

Abstract

Embodiments of the present invention provide a distributed storage system and a distributed reading and writing method, which relate to the field of supercomputing technology. The distributed storage system includes a switch and multiple computing servers. Each computing server includes a data processor and multiple data memories. The data processor is deployed with a storage server and a first storage client, and the computing server is deployed with a second storage client. Each data processor is communicatively connected through the switch. The computing server is used to send read and write instructions to the data processor based on the second storage client. The data processor is used to receive the read and write instructions based on the first storage client, and based on the read and write instructions, perform data interaction with the data memory based on the storage server. By offloading the access to the data memory in the computing server to the data processor of this node, the isolation between the storage service and the computing service is ensured, thereby saving computing resources and reducing the data access latency.

Description

Technical Field

[0001] The present invention relates to the field of supercomputing technology, and in particular, to a distributed storage system and a distributed read / write method. Background Art

[0002] In traditional intelligent computing centers and supercomputing centers, a network architecture with separated storage and computing is generally adopted. A computing cluster is composed of computing servers responsible for computing, and a storage cluster is composed of nodes responsible for storage. By mounting storage services on the computing servers, the coordination and cooperation between storage and computing are achieved.

[0003] However, in the current technical background, a certain amount of local data memory is usually integrated inside modern computing servers. The network architecture with separated storage and computing in traditional intelligent computing centers and supercomputing centers does not make full use of the storage resources inside the computing servers, resulting in a certain degree of resource waste. Summary of the Invention

[0004] In view of this, the purpose of the embodiments of the present invention is to provide a distributed storage system and a distributed read / write method to at least partially improve the above problems.

[0005] To achieve the above purpose, the technical solutions adopted in the embodiments of the present invention are as follows:

[0006] In a first aspect, an embodiment of the present invention provides a distributed storage system, including a switch and multiple computing servers; each of the computing servers includes a data processor and multiple data memories; the storage server and the first storage client are deployed on the data processor; the second storage client is deployed on the computing server; each of the data processors is communicatively connected through the switch;

[0007] The computing server is configured to send read / write instructions to the data processor based on the second storage client;

[0008] The data processor is configured to receive the read / write instructions based on the first storage client, and perform data interaction with the data memory based on the storage server according to the read / write instructions.

[0009] Optionally, when the read / write instruction is to write data, the data processor is further configured to receive the data to be written sent by the computing server based on the first storage client, and store the data to be written into each target data memory through the storage server; and generate global metadata according to the data to be written, and update the global metadata to each of the storage servers through the switch; the global metadata is used to record the data status in the data memories of each computing server, so that each computing server can access the data in the data memory of any computing server.

[0010] Optionally, the data processor is further configured to migrate the data of each data memory by using the idle network based on the storage server, including:

[0011] After storing the data to be written into each target data memory, determine whether the current computing server is using the network under high load;

[0012] If not, scan the data in each target data memory to find the first data to be migrated;

[0013] Calculate the migration target data memory of the target computing server to which the data is to be migrated according to the first data to be migrated;

[0014] Migrate the first data to be migrated to each of the migration target data memories through the switch, and update the global metadata.

[0015] Optionally, the distributed storage system further includes a hierarchical storage module; the hierarchical storage module is communicatively connected to each computing server through the switch;

[0016] The data processor is further configured to check the storage space of the computing server where it is located based on the storage server, and when the usage ratio of the storage space is greater than a preset threshold, find the second data to be migrated that meets the first preset condition, and migrate the second data to be migrated to the hierarchical storage module through the switch.

[0017] Optionally, the computing server further includes a computing network network card, and each computing network network card is communicatively connected to each data processor through the switch;

[0018] The computing server is further configured to determine whether the data to be written needs to be directly written into the hierarchical storage module, and if so, store the data to be written into the hierarchical storage module through the computing network network card.

[0019] Optionally, the computing server is further configured to calculate whether the instantaneous write rate of the data to be written exceeds the bus bandwidth between the computing server and the data storage; if so, write a preset proportion of the data to be written into the data memories of other computing servers or the hierarchical storage module through the computing network network card or / and the data processor via the switch.

[0020] Optionally, the computing server writes a preset proportion of the data to be written into the data memories of other computing servers or the hierarchical storage module through the computing network network card or / and the data processor via the switch in the following manner:

[0021] Write the data to be written into the data memory managed by the local data processor preferentially. When the local data processor reaches a preset condition, search for data that meets the second preset condition locally, and write it into the data memories of other computing servers or into the hierarchical storage module through the computing network network card or / and the data processor via the switch.

[0022] Optionally, the data processor receives the data to be written sent by the computing server based on the first storage client in the following manner, and stores the data to be written into each target data memory through the storage server:

[0023] After receiving the data to be written based on the first storage client, determine whether the data to be written needs to perform compute-intensive I / O processing;

[0024] If so, determine whether its computing resources are sufficient;

[0025] If not, call the computing server where it is located to jointly perform compute-intensive I / O processing on the data to be written;

[0026] Store the processed data into each target data memory through the storage server.

[0027] Optionally, the computing server includes multiple data processors; the data processor receives the data to be written sent by the computing server based on the first storage client in the following manner, and stores the data to be written into each target data memory through the storage server:

[0028] After receiving the data to be written based on the first storage client, determine whether its computing resources are sufficient to support the storage calculation of the data to be written;

[0029] If so, directly perform storage calculation processing on the data to be written;

[0030] Otherwise, call other data processors of the computing server where it is located to jointly perform storage calculation processing on the data to be written;

[0031] Store the processed data into each target data storage through the storage server.

[0032] Optionally, the data processor is further configured to convert the protocol of the first storage client to the storage server, so as to expose it as a PCIE device to the computing server;

[0033] The second storage client is used to convert the PCIE device of the first storage client into a file system or a block device.

[0034] Optionally, the computing server is further deployed with a third storage client; the third storage client is any one or more of an NFS client, a CIFS client, a POSIX client, and a parallel file system client.

[0035] In a second aspect, an embodiment of the present invention provides a distributed read-write method, which is applied to the distributed storage system as described above. The distributed storage system includes a switch and multiple computing servers; each computing server includes a data processor and multiple data storages; the data processor is deployed with a storage server and a first storage client; the computing server is deployed with a second storage client; each data processor is communicatively connected through the switch; the method includes:

[0036] The computing server sends read-write instructions to the data processor based on the second storage client;

[0037] The data processor receives the read-write instructions based on the first storage client, and based on the read-write instructions, performs data interaction with the data storage based on the storage server.

[0038] The distributed storage system and the distributed read-write method provided by the embodiments of the present invention ensure the isolation of storage services and computing services by placing the storage server in the data processor, save storage nodes, and overall reduce the construction cost of the entire data center; by first writing data into the data storage of this node, the writing efficiency of storage is improved, the read-write latency of storage is reduced, and then through the background migration task, the reliability of the data is ensured, and the idle network is used for migration, so that the migration task does not affect the network bandwidth and latency requirements of the computing task; by expanding the hierarchical storage module, it is ensured that the computing resources always have sufficient storage space to use; by means of the data shunt writing method, the problem that the instantaneous data writing volume of the computing task of a single computing server is greater than the insufficient internal bus bandwidth between the computing resource and the storage resource nodes is solved.

[0039] To make the above objects, features, and advantages of the present application more obvious and understandable, the following provides preferred embodiments in conjunction with the accompanying drawings, and the detailed description is as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0041] Figure 1 A schematic structural block diagram of a distributed storage system provided by an embodiment of the present invention;

[0042] Figure 2 A schematic flow diagram of data migration provided by an embodiment of the present invention;

[0043] Figure 3 Another schematic structural block diagram of a distributed storage system provided by an embodiment of the present invention;

[0044] Figure 4 Another schematic structural block diagram of a distributed storage system provided by an embodiment of the present invention;

[0045] Figure 5 A schematic flow diagram of writing to other data memories when the instantaneous write traffic is large provided by an embodiment of the present invention;

[0046] Figure 6 A schematic flow diagram of data being stored in each target data memory provided by an embodiment of the present invention;

[0047] Figure 7 Another schematic flow diagram of data being stored in each target data memory provided by an embodiment of the present invention;

[0048] Figure 8 A schematic structural block diagram of a computing server provided by an embodiment of the present invention;

[0049] Figure 9 A schematic flow diagram of a distributed read-write method provided by an embodiment of the present invention.

[0050] Reference numerals: 100 - distributed storage system; 110 - computing server; 111 - data processor; 1111 - storage server; 112 - data memory; 1112 - first storage client; 113 - second storage client; 114 - computing network network card; 115 - third storage client; 120 - switch; 130 - hierarchical storage module. Detailed implementation manners

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention described and illustrated herein can be arranged and designed in a variety of different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0053] It should be noted that: like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present invention, terms such as "first", "second", etc. are only used for differential description and cannot be understood as indicating or implying relative importance.

[0054] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0055] As described in the background art, in traditional intelligent computing centers and supercomputing centers, a network architecture with separated storage and computing is generally adopted. With the development of data processing units (DPUs), an additional part of computing power has emerged inside computing servers, and this part of resources is not currently effectively utilized. At the same time, computing servers, especially graphics processing unit (GPU) computing servers, have built-in data memories, and this part of resources is also not effectively utilized, resulting in a certain degree of resource waste.

[0056] Based on the above situation, the embodiments of the present invention provide a distributed storage system and a distributed reading and writing method. Through the storage offloading function of the DPU, the access to the data memory in the computing server is offloaded to the DPU, ensuring the isolation of the storage service and the computing service, making full use of the internal storage resources of the computing server, thereby saving computing unit resources and reducing data access latency.

[0057] Next, an exemplary description is given for the distributed storage system provided by the present invention. Specifically, Figure 1 For the structural block diagram of a distributed storage system provided by an embodiment of the present invention, refer to Figure 1 , the distributed storage system 100 includes a switch 120 and multiple computing servers 110. Each of the computing servers 110 includes a data processor 111 and multiple data memories 112. The data processor 111 is deployed with a storage server 1111 and a first storage client 1112. The computing server 110 is deployed with a second storage client 113. Each of the data processors 111 is communicatively connected through the switch 120.

[0058] Among them, the storage server 1111, the first storage client 1112, and the second storage client 113 are software components running in the distributed storage system 100.

[0059] Among them, the computing server can be a high-performance server specifically designed to execute complex computing tasks, such as scientific computing, data analysis, artificial intelligence training, etc. The data processor is a DPU, and the data memory can be an NVMe SSD. Through the storage offloading function of the DPU, the access to the NVMe SSD of the original GPU and CPU in the computing server is offloaded to the DPU.

[0060] The computing server is used to send read and write instructions to the data processor based on the second storage client.

[0061] The data processor is used to receive the read and write instructions based on the first storage client, and based on the read and write instructions, perform data interaction with the data memory based on the storage server.

[0062] The data processor is a micro server that can operate independently. It has an operating system inside, just like a computer. Through the operating system, a first storage client and a storage server are installed. The storage server is used to interact with the data memory for data, and the first storage client communicates with the second storage client in the computing server. For example, when storing data, the computing server sends the data to the data processor through communication between the second storage client and the first storage client. After the data processor obtains the data, it is handed over to the storage server for processing and then stored in the data memory. When reading data, after the data processor obtains the read instruction, the storage server retrieves the corresponding data from the data memory and then sends it to the computing server.

[0063] This distributed storage system unloads the access to the data memory in the computing server to the data processor of this node, ensuring the isolation of storage services and computing services, making use of the storage resources in the computing server, thereby saving computing resources and storage resources and reducing data access latency.

[0064] In a possible implementation manner, when the read / write instruction is to write data, the data processor is further configured to receive the data to be written sent by the computing server based on the first storage client, and store the data to be written into each target data memory through the storage server; and, generate global metadata according to the data to be written, and update the global metadata to each of the storage servers through the switch; the global metadata is used to record the data status in the data memories of each computing server, so that each computing server can access the data in the data memory of any computing server.

[0065] The data processor receives the data to be written. The received data can undergo some preprocessing operations, such as format conversion, verification, compression, etc., to ensure that the data can be smoothly received and processed by the storage server, and then the data is stored in the target data memory. Among them, the storage server can perform sharding processing on the received data and distribute these data segments to different target data memories according to a pre-set policy. Each target data memory will record its own storage status, including the information of the stored data segments, storage locations, etc.

[0066] During the process of distributing data to each data storage, the data processor generates global metadata in real time. Global metadata is a summary view of the status of all data fragments in the entire system. Global metadata includes the identifier of each data fragment, location information (i.e., stored on which device), status information (whether it has been successfully written), and other necessary meta-information. The generated global metadata is distributed to the corresponding storage servers through a network switch, ensuring that each computing server can obtain the latest global metadata according to certain rules. After receiving the global metadata, other computing servers can access the stored data of this computing server based on this information.

[0067] Writing the data to be written into the data storage of this node improves the write bandwidth of the storage and reduces the read / write latency of the storage. After the data is written, when a single computing server is offline, the data of the computing server where it is located is unavailable. Therefore, replicas or erasure codes can be saved in the data storage of this computing server to ensure basic reliability. It may also cause the global data of the system to be unbalanced, with some computing servers having a high space utilization rate and some computing servers having idle space. To improve the balance and reliability of the global data of the system, the data processor is also used to migrate the data of each data storage based on the storage server using the idle network, see Figure 2 The migration steps may include:

[0068] S11: After storing the data to be written into each target data storage, determine whether the current computing server is using the network at a high load.

[0069] S12: If not, scan the data in each data storage to find the first data to be migrated.

[0070] S13: Based on the first data to be migrated, calculate the migration target data storage of the target computing server to be migrated to.

[0071] S14: Migrate the first data to be migrated to each migration target data storage through the switch and update the global metadata.

[0072] For example, when the computing server is performing calculations, the bandwidth requirement for the network is not high. During this process, the storage server running in the DPU migrates the data using the idle network according to the data redundancy strategy to obtain higher reliability.

[0073] After data is written, check whether each computing server is using the network at a high load. If not, start scanning the relevant data storage to find the data that needs to be migrated as the first data to be migrated. It is necessary to determine which data storage on which computing server these data should be migrated to. The selection method of the data storage: First calculate the server-level reliability. For example, when the erasure code is 4+2, first select 6 independent nodes. When not enough nodes to meet the reliability can be found, first select the node set with the highest overall reliability. After determining the target data storage for migration, complete the transmission through the switch to ensure that the data can be efficiently and reliably transmitted to the new location. While the data is being migrated, it is also necessary to update the global metadata of the system to reflect the new location of the data. After the migration is completed, the global metadata needs to be updated so that each computing server can access the stored data of any node.

[0074] In a possible implementation, to prevent the situation where the internal storage space of the computing server is not enough, refer to Figure 3 The distributed storage system may further include a hierarchical storage module 130, and the hierarchical storage module is communicatively connected to each computing server through a switch.

[0075] Among them, the hierarchical storage module can be an EBOF chassis with only block functions, or an EBOF running a DPU or a standard server running a storage server side, and the standard server is a server with the same function as the storage in the computing server.

[0076] The data processor is further configured to check the storage space of the computing server where it is located based on the storage server side. When the usage ratio of the storage space is greater than a preset threshold, find the second data to be migrated that meets the first preset condition, and migrate the second data to be migrated to the hierarchical storage module through the switch.

[0077] The data processor needs to regularly or on demand check the usage of the storage space of this computing server. The content of the check may include the total storage capacity, the amount of used storage space, and the remaining available space, calculate the usage ratio of the storage space. If this ratio exceeds the preset threshold (for example, 80%), trigger the data migration process and start looking for those data items that meet the first preset condition, that is, the second data to be migrated. The first preset condition may include, but is not limited to: the access frequency of the data is relatively low, the creation date of the data is earlier, the data belongs to a certain specific type or category. Migrate the filtered second data to be migrated to the hierarchical storage module through the switch to relieve the storage pressure of the computing server.

[0078] Further, the data calculated by the computing tasks of the distributed storage system may sometimes be cold data that is not frequently used. To prevent this type of data from occupying the storage space in the computing server, this type of data can be directly written into the hierarchical storage module. Refer to Figure 4 The computing server 110 may further include computing network network cards 114. Each computing network network card 114 is communicatively connected to each data processor 111 through a switch 120.

[0079] The computing server is further configured to determine whether the data to be written needs to be directly written into the hierarchical storage module. If so, the data to be written is stored in the hierarchical storage module through the computing network network card.

[0080] Determining whether the data to be written needs to be directly written into the hierarchical storage module can be determined according to a certain policy. For example, if the data is cold data that will basically not be used in subsequent calculations, in this case, the computing server will not store the data to be written in the data memory of this computing server, but directly write the data into the hierarchical storage module through the computing network network card and the switch.

[0081] In a supercomputing center, a large amount of data is often calculated, resulting in a large amount of instantaneous write data, exceeding the bus bandwidth between the computing server and the data memory. In a possible implementation, the computing server is further configured to calculate whether the instantaneous write rate of the data to be written exceeds the bus bandwidth between the computing server and the data memory; if so, a preset proportion of the data to be written is written into the data memory or the hierarchical storage module of other computing servers through the computing network network card or / and the data processor via the switch.

[0082] When the amount of data to be written is large, the write I / O path of this computing server may reach a bottleneck due to bus bandwidth limitations, resulting in a slow write speed. In this case, data may accumulate in the computing server. In this situation, the data can be written into other data memories in the system through the DPU or the computing network network card. The data memory can be either other peer computing servers or an external hierarchical storage module.

[0083] Exemplarily, refer to Figure 5 The process of writing into other data memories may include the following steps:

[0084] S21: Determine whether the usage rate of the local data memory storage space is lower than a preset condition; if so, execute step S24, if not, execute step S22.

[0085] S22: Determine whether the current scanned data meets the second preset condition. If so, execute step S23, if not, execute step S24.

[0086] S23: Write the eligible data in the data memory managed by the local data processor to the data memory of other computing servers or the hierarchical storage module through the computing network card or / and the data processor via the switch.

[0087] S24: Exit.

[0088] Data will only be written to the data memory of other computing servers or the hierarchical storage module when the storage space utilization rate of the local data memory is higher than a certain value. When the storage space utilization rate is lower than a certain value, it will not be written to the data memory of other computing servers or the hierarchical storage module, so it will directly exit without any processing. When the storage space utilization rate of the local data memory is higher than a certain value, it will then determine whether the current scanned data meets the second preset condition. If so, the eligible data will be written to the data memory of other computing servers or the hierarchical storage module. If not, it will directly exit without any processing.

[0089] Among them, the second preset condition is that the data is cold data. If the scanned data meets the cold data condition, these data need to be written to the data memory or the hierarchical storage module of other computing servers. If it is not cold data, no processing will be done.

[0090] There can be multiple ways to store the data to be written in each target data memory. In one possible implementation, see Figure 6 , the data processor is used to receive the data to be written based on the first storage client and store the data to be written in each target data memory through the storage server:

[0091] S31: After receiving the data to be written based on the first storage client, determine whether the data to be written needs to be processed with computationally intensive I / O.

[0092] After receiving the data to be written, it is generally necessary to process the data before storing it in the data memory. At this time, first determine whether computationally intensive I / O processing is required, such as data compression, deduplication, etc.

[0093] If not, it can be directly processed using the computing resources of the DPU itself.

[0094] S32: If so, determine whether its own computing resources are sufficient.

[0095] If computationally intensive I / O processing is required, first check whether the computing resources of the DPU itself are sufficient. If sufficient, it can also be directly processed using the computing resources of the DPU itself.

[0096] S33: If not, call the computing unit of the computing server where it is located to jointly perform computationally intensive I / O processing on the data to be written.

[0097] If the computing resources of the DPU itself are not sufficient to perform compute-intensive I / O processing on the data, the CPU and / or GPU of the computing server where the DPU is located are called to jointly perform compute-intensive I / O processing on the data.

[0098] S34: The processed data is stored in each target data storage device through the storage server.

[0099] After the data is processed, it is stored in each target data storage device.

[0100] In another possible implementation, the computing server includes multiple data processors. Refer to Figure 7 , the data processor is used to receive the data to be written based on the first storage client in the following manner, and store the data to be written in each target data storage device through the storage server:

[0101] S41: After receiving the data to be written based on the first storage client, determine whether its own computing resources are sufficient to support the storage calculation of the data to be written.

[0102] Among them, the storage calculation may include but is not limited to block characteristics, file characteristics, snapshots, clone characteristics, deduplication, compression, encryption, replicas, erasure codes, etc.

[0103] S42: If so, directly perform storage calculation processing on the data to be written.

[0104] S43: If not, call other data processors of the computing server where it is located to jointly perform storage calculation processing on the data to be written.

[0105] S44: The processed data is stored in each target data storage device through the storage server.

[0106] The logic is the same as above. If the computing resources of the DPU itself are sufficient to support the storage calculation of the data to be written, directly perform storage calculation processing on the data to be written. If not, other DPU in this computing server can be called to jointly perform storage calculation processing on the data to be written. After the data is processed, it is stored in each target data storage device.

[0107] Multiple DPU inside the computing server are responsible for data storage devices with closer physical distances preferentially, and can also provide high-availability services within the node externally. When a DPU fails or the link is interrupted, the service and data link are taken over by other nodes.

[0108] Optionally, the data processor is also used to convert the protocol of the storage server based on the first storage client to expose it as a PCIE device to the computing server; the second storage client is used to convert the PCIE device of the first storage client into a file system or a block device.

[0109] The first storage client running on the data processor is responsible for access control of storage, protocol conversion, and other advanced features that require computing, and is exposed to this computing server as a PCIE device. The second storage client running on this computing server converts the PCIE device of the DPU into specific files (such as through virtio-fs conversion) and blocks (such as through virtio-blk conversion).

[0110] Optionally, refer to Figure 8 , the computing server may further include a third storage client 115; the third storage client is any one or more of an NFS client, a CIFS client, a POSIX client, and a parallel file system client. The third storage client is used to enable the computing server to flexibly access the data in the data storage. Through the traditional client, the data in each data storage can be directly accessed through the network protocol, which can improve the flexibility of the computing server to access data.

[0111] Furthermore, an embodiment of the present invention further provides a distributed read-write method, which is applied to the distributed storage system as described above. The distributed storage system includes a switch and multiple computing servers. Each computing server includes a data processor and multiple data storages; a storage server and a first storage client are deployed on the data processor, and a second storage client is deployed on the computing server. Each data processor is communicatively connected through the switch. Refer to Figure 9 , the method includes the following steps:

[0112] S51: The computing server sends read-write instructions to the data processor based on the second storage client.

[0113] S52: The data processor receives the read-write instructions based on the first storage client, and based on the read-write instructions, interacts with the data storage based on the storage server.

[0114] In summary, for the distributed storage system and the distributed reading and writing method provided by the embodiments of the present invention, by placing the storage server in the data processor, the isolation between the storage service and the computing service is ensured, storage nodes are saved, and the construction cost of the entire data center is reduced as a whole; multiple data processors inside the computing server can provide high-availability services within the node externally. When a data processor fails or the link is interrupted, the service and data link are taken over by other data processors; by first writing data into the data memory of the local node, the writing efficiency of storage is improved, the reading and writing delay of storage is reduced, and then through the background migration task, the reliability of data is ensured, and the idle network is used for migration, so that the migration task does not occupy the network bandwidth; by expanding the hierarchical storage module, it is ensured that the computing resources always have sufficient storage space for use; by means of the data shunting writing method, the problem that the instantaneous data writing volume of the computing task of a single computing server is greater than the insufficient internal node bus bandwidth between the computing resources and the storage resources is solved.

[0115] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0116] In addition, each functional module in various embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0117] If a function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only data memories (ROM, Read-Only Memory), random access data memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0118] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0119] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be encompassed within the present invention. Any reference signs in the claims should not be regarded as limiting the claims involved.

Claims

1. A distributed storage system, characterized in that: The system comprises a switch, a tiered storage module, and a plurality of computing servers; each computing server comprises a data processor, a plurality of data storage devices, and a computing network card; the data processor is deployed with a storage service client and a first storage client; the computing server is deployed with a second storage client; the data processors are communicatively connected via the switch; the computing network cards are communicatively connected to the data processors via the switch; The hierarchical storage module is communicatively connected with each of the computing servers via the switch; The computing server is configured to send a read / write instruction to the data processor based on the second storage client; The data processor is configured to receive the read / write instruction based on the first storage client, and perform data interaction between the storage service client and the data storage according to the read / write instruction; When the read / write instruction is for writing data, the data processor is further configured to receive the data to be written sent by the computing server based on the first storage client, and store the data to be written into each target data storage through the storage service end; and, generate global metadata based on the data to be written, and update the global metadata to each storage service end through the switch; the global metadata is used to record the data status in the data storage of each computing server, so that each computing server can access the data in the data storage of any computing server; The data processor is further configured to check the storage space of the computing server on which the storage service is located based on the storage service client, and when a usage ratio of the storage space is greater than a preset threshold, find second data to be migrated that meets a first preset condition, and migrate the second data to be migrated to the tiered storage module through the switch; The computing server is further configured to determine whether the data to be written needs to be directly written into the hierarchical storage module, and if so, store the data to be written into the hierarchical storage module via the computing network card; The data processor is further configured to convert a protocol of the storage service client based on the first storage client so as to expose the protocol as a PCIE device to the computing server; The second storage client is used to convert the PCIE device of the first storage client into a file system or a block device.

2. The distributed storage system according to claim 1, wherein: The data processor is further configured to migrate data of each of the data storage devices using an idle network based on the storage service client, including: After storing the data to be written into each of the target data storage devices, determining whether the current computing server is using the network at a high load; If not, scanning the data in each target data storage to find the first data to be migrated; Calculating a migration target data storage of a target computing server to be migrated based on the first data to be migrated; The first data to be migrated is migrated to each of the migration target data storages through the switch, and the global metadata is updated.

3. The distributed storage system according to claim 1, wherein: The computing server is further configured to calculate whether the instantaneous write rate of the data to be written exceeds the bus bandwidth between the computing server and the data storage device; if so, a preset proportion of the data to be written is written into the data storage devices of other computing servers or the hierarchical storage modules via the computing network card and / or the data processor via the switch.

4. The distributed storage system according to claim 3, characterized in that: The computing server is configured to write the data to be written in a preset proportion into the data storages or the hierarchical storage modules of other computing servers via the computing network card and / or the data processor via the switch in the following manner: The data to be written is preferentially written into the data storage managed by the local data processor. When the local data processor reaches the preset condition, the local data that meets the second preset condition is searched, and the data is written into the data storage of other computing servers or into the hierarchical storage module by the switch through the computing network card and / or the data processor.

5. The distributed storage system according to claim 1, wherein: The data processor is configured to receive the data to be written sent by the computing server based on the first storage client in the following manner, and store the data to be written into each target data storage through the storage service client: After receiving the data to be written based on the first storage client, determining whether the data to be written requires computationally intensive IO processing; If so, determine whether its own computing resources are sufficient; If not, call the computing server where the computing server is located to jointly perform computing-intensive IO processing on the data to be written; The processed data is stored in each target data storage device through the storage service end.

6. The distributed storage system according to claim 1, wherein: The computing server includes a plurality of data processors; the data processors are configured to receive the data to be written sent by the computing server based on the first storage client and store the data to be written into each target data storage through the storage service client in the following manner: After receiving the data to be written based on the first storage client, determining whether its own computing resources are sufficient to support storage computing of the data to be written; If so, directly perform storage and calculation processing on the data to be written; If not, calling the other data processors of the computing server where the computing server is located to jointly perform storage computing processing on the data to be written; The processed data is stored in each target data storage device through the storage service end.

7. The distributed storage system according to any one of claims 1 to 6, characterized in that: The computing server is further deployed with a third storage client; the third storage client is any one or more of an NFS client, a CIFS client, a POSIX client, and a parallel file system client.

8. A distributed reading and writing method, characterized in that: Applied to the distributed storage system according to any one of claims 1 to 7, the distributed storage system comprises a switch and a plurality of computing servers; each of the computing servers comprises a data processor and a plurality of data storage devices; the data processor is deployed with a storage service client and a first storage client; The computing server is deployed with a second storage client; Each of the data processors is communicatively connected via the switch; the method comprises: The computing server sends a read / write instruction to the data processor based on the second storage client; The data processor receives the read and write instructions based on the first storage client, and performs data interaction with the data storage based on the storage service client according to the read and write instructions.

Images