Distributed storage system and distributed read-write method

Abstract

The embodiment of the invention provides a distributed storage system and a distributed read-write method, and relates to the technical field of super computing. The distributed storage system comprises a switch and a plurality of computing servers, each computing server comprises a data processor and a plurality of data memories, each data processor is provided with a storage server and a first storage client, each computing server is provided with a second storage client, and the data processors are in communication connection through the switch. The computing server is used for sending a read-write instruction to the data processor based on the second storage client, and the data processor is used for receiving the read-write instruction based on the first storage client and performing data interaction with the data memory based on the storage server according to the read-write instruction. The access of the data memory in the computing server is unloaded to the data processor of the node, so that the isolation of the storage service and the computing service is ensured, the computing resources are saved, and the data access time delay is reduced.

Description

Technical Field

[0001] The present invention relates to the field of supercomputing technology, and in particular to a distributed storage system and a distributed reading and writing method. Background Art

[0002] In traditional intelligent computing centers and supercomputing centers, a network architecture that separates storage and computing is generally adopted. The computing cluster is composed of computing servers responsible for computing, and the storage cluster is composed of nodes responsible for storage. The coordinated cooperation of storage and computing is achieved by mounting storage services on the computing servers.

[0003] However, in the current technological context, modern computing servers usually have a certain amount of local data storage integrated into them. The traditional intelligent computing center and supercomputing center's storage and computing separation network architecture does not fully utilize the storage resources inside the computing server, resulting in a certain degree of resource waste. Summary of the invention

[0004] In view of this, an object of an embodiment of the present invention is to provide a distributed storage system and a distributed reading and writing method to at least partially improve the above-mentioned problem.

[0005] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows: In a first aspect, an embodiment of the present invention provides a distributed storage system, including a switch and a plurality of computing servers; each of the computing servers includes a data processor and a plurality of data storage devices; the data processor is deployed with a storage service client and a first storage client; the computing server is deployed with a second storage client; each of the data processors is connected in communication via the switch; The computing server is used to send a read and write instruction to the data processor based on the second storage client; The data processor is used to receive the read and write instructions based on the first storage client, and perform data interaction with the data storage based on the storage service client according to the read and write instructions.

[0006] Optionally, when the read and write instructions are for writing data, the data processor is also used to receive the data to be written sent by the computing server based on the first storage client, and store the data to be written into each target data storage device through the storage service client; and, based on the data to be written, generate global metadata, and update the global metadata to each storage service client through the switch; the global metadata is used to record the data status in the data storage device of each computing server, so that each computing server can access the data in the data storage device of any computing server.

[0007] Optionally, the data processor is further configured to migrate data of each of the data storage devices based on the storage service client using an idle network, including: After storing the data to be written into each of the target data storage devices, determining whether the current computing server is using the network at a high load; If not, scanning the data in each of the target data storage devices to find out the first data to be migrated; Calculating a migration target data storage of a migration target computing server according to the first data to be migrated; The first data to be migrated is migrated to each of the migration target data storage devices through the switch, and the global metadata is updated.

[0008] Optionally, the distributed storage system further includes a hierarchical storage module; the hierarchical storage module is communicatively connected with each of the computing servers through the switch; The data processor is also used to check the storage space of the computing server based on the storage service end, and when the usage ratio of the storage space is greater than a preset threshold, find out the second data to be migrated that meets the first preset condition, and migrate the second data to be migrated to the layered storage module through the switch.

[0009] Optionally, the computing server further includes a computing network card, and each computing network card is communicatively connected to each data processor via the switch; The computing server is further configured to determine whether the data to be written needs to be directly written into the hierarchical storage module, and if so, store the data to be written into the hierarchical storage module through the computing network card.

[0010] Optionally, the computing server is also used to calculate whether the instantaneous write rate of the data to be written exceeds the bus bandwidth between the computing server and the data storage device; if so, a preset proportion of the data to be written is written into the data storage device of each other computing server or the hierarchical storage module through the computing network card and / or the data processor via the switch.

[0011] Optionally, the computing server is used to write the to-be-written data of a preset proportion into the data storage of each other computing server or the hierarchical storage module through the computing network card and / or the data processor via the switch in the following manner: The data to be written is preferentially written into the data storage device managed by the local data processor. When the local data processor reaches a preset condition, the local data that meets the second preset condition is searched, and the data is written into the data storage device of other computing servers or into the hierarchical storage module by the switch through the computing network card and / or the data processor.

[0012] Optionally, the data processor is used to receive the data to be written sent by the computing server based on the first storage client in the following manner, and store the data to be written into each target data storage through the storage service client: After receiving the data to be written based on the first storage client, determining whether the data to be written requires computationally intensive IO processing; If so, determine whether its computing resources are sufficient; If not, calling the computing server where the computing server is located to jointly perform computing-intensive IO processing on the data to be written; The processed data is stored in each target data storage device through the storage service end.

[0013] Optionally, the computing server includes a plurality of the data processors; the data processor is used to receive the data to be written sent by the computing server based on the first storage client in the following manner, and store the data to be written into each target data storage through the storage service client: After receiving the data to be written based on the first storage client, determining whether its own computing resources are sufficient to support storage computing of the data to be written; If so, directly perform storage and calculation processing on the data to be written; If not, calling other data processors of the computing server where the computing server is located to jointly perform storage computing processing on the data to be written; The processed data is stored in each target data storage device through the storage service end.

[0014] Optionally, the data processor is further used to convert the protocol of the storage service client based on the first storage client to expose it as a PCIE device to the computing server; The second storage client is used to convert the PCIE device of the first storage client into a file system or a block device.

[0015] Optionally, the computing server is further deployed with a third storage client; the third storage client is any one or more of an NFS client, a CIFS client, a POSIX client, and a parallel file system client.

[0016] In a second aspect, an embodiment of the present invention provides a distributed reading and writing method, which is applied to the distributed storage system as described above, wherein the distributed storage system includes a switch and multiple computing servers; each of the computing servers includes a data processor and multiple data storage devices; the data processor is deployed with a storage service client and a first storage client; the computing server is deployed with a second storage client; each of the data processors is connected to each other through the switch; the method includes: The computing server sends a read and write instruction to the data processor based on the second storage client; The data processor receives the read / write instruction based on the first storage client, and performs data interaction with the data storage device based on the storage service client according to the read / write instruction.

[0017] A distributed storage system and a distributed read-write method provided by an embodiment of the present invention ensure the isolation of storage services and computing services by placing a storage service end into a data processor, thereby saving storage nodes and reducing the overall construction cost of the entire data center; by first writing data into the data storage device of the node, the storage write efficiency is improved and the storage read-write latency is reduced; and then, through a background migration task, the reliability of the data is ensured, and the migration is performed using an idle network, so that the migration task does not affect the bandwidth latency demand of the computing task for the network; by extending the hierarchical storage module, it is ensured that computing resources always have sufficient storage space to use; and through a data diversion writing method, the problem that the instantaneous write data volume of a computing task of a single computing server is greater than the insufficient internal bus bandwidth of the node between the computing resource and the storage resource is solved.

[0018] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic structural block diagram of a distributed storage system provided by an embodiment of the present invention; Figure 2 A schematic diagram of a data migration process provided by an embodiment of the present invention; Figure 3 Another schematic structural block diagram of a distributed storage system provided by an embodiment of the present invention; Figure 4 Another schematic structural block diagram of a distributed storage system provided by an embodiment of the present invention; Figure 5 A schematic diagram of a process of writing into other data storage devices when the instantaneous write flow is large provided by an embodiment of the present invention; Figure 6 A schematic diagram of a process of storing data into target data storage devices provided by an embodiment of the present invention; Figure 7 Another schematic diagram of a process for storing data into target data storage devices provided by an embodiment of the present invention; Figure 8 A schematic structural block diagram of a computing server provided in an embodiment of the present invention; Fig. 9 A schematic flow chart of a distributed reading and writing method provided in an embodiment of the present invention.

[0021] Icons: 100 - distributed storage system; 110 - computing server; 111 - data processor; 1111 - storage service client; 112 - data storage; 1112 - first storage client; 113 - second storage client; 114 - computing network card; 115 - third storage client; 120 - switch; 130 - layered storage module. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0025] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0026] As described in the background technology, in traditional intelligent computing centers and supercomputing centers, a network architecture with separated storage and computing is generally adopted. With the development of data processors (Data Processing Unit, DPU), a part of computing power is added inside the computing server. This part of resources is not currently effectively utilized. At the same time, computing servers, especially graphics processing unit (Graphics Processing Unit, GPU) computing servers, have their own data storage devices, and this part of resources is not effectively utilized, resulting in a certain degree of resource waste.

[0027] Based on the above situation, an embodiment of the present invention provides a distributed storage system and a distributed reading and writing method. Through the storage offloading function of the DPU, the access to the data storage in the computing server is offloaded to the DPU, ensuring the isolation of the storage service and the computing service, and making full use of the storage resources inside the computing server, thereby saving computing unit resources and reducing data access latency.

[0028] The following is an exemplary description of the distributed storage system provided by the present invention. Specifically, Figure 1 A schematic diagram of a distributed storage system according to an embodiment of the present invention is shown in FIG. Figure 1 The distributed storage system 100 includes a switch 120 and multiple computing servers 110. Each computing server 110 includes a data processor 111 and multiple data storage devices 112. The data processor 111 is deployed with a storage service client 1111 and a first storage client 1112. The computing server 110 is deployed with a second storage client 113. Each data processor 111 is communicatively connected via the switch 120.

[0029] Among them, the storage service client 1111 , the first storage client 1112 , and the second storage client 113 are software components running in the distributed storage system 100 .

[0030] The computing server can be a high-performance server specially designed to perform complex computing tasks, such as scientific computing, data analysis, artificial intelligence training, etc. The data processor is a DPU, and the data storage can be an NVMe SSD. Through the storage offloading function of the DPU, the NVMe SSD access originally from the GPU and CPU in the computing server is offloaded to the DPU.

[0031] The computing server is used to send read and write instructions to the data processor based on the second storage client.

[0032] The data processor is used to receive the read and write instructions based on the first storage client, and perform data interaction with the data storage based on the storage service client according to the read and write instructions.

[0033] The data processor is a micro server that can run independently. It has an operating system like a computer. Through the operating system, a first storage client and a storage service client are installed. The storage service client is used to interact with the data storage device, and the first storage client communicates and interacts with the second storage client in the computing server. For example, when storing data, the computing server communicates with the first storage client through the second storage client and sends the data to the data processor. After the data processor obtains the data, it is processed by the storage service client and stored in the data storage device. When reading data, after the data processor obtains the read instruction, the storage service client obtains the corresponding data from the data storage device and sends it to the computing server.

[0034] This distributed storage system ensures the isolation of storage services and computing services by offloading access to the data storage in the computing server to the data processor of the local node, utilizing the storage resources in the computing server, thereby saving computing resources and storage resources and reducing data access latency.

[0035] In one possible implementation, when the read and write instructions are for writing data, the data processor is also used to receive the data to be written sent by the computing server based on the first storage client, and store the data to be written into each target data storage device through the storage service client; and, based on the data to be written, generate global metadata, and update the global metadata to each storage service client through the switch; the global metadata is used to record the data status in the data storage device of each computing server, so that each computing server can access the data in the data storage device of any computing server.

[0036] The data processor receives the data to be written, and the received data can undergo some preprocessing, such as format conversion, verification, compression, etc., to ensure that the data can be smoothly received and processed by the storage service end, and then stored in the target data storage. Among them, the storage service end can process the received data in segments and distribute these data segments to different target data storages according to pre-set strategies. Each target data storage will record its own storage status, including the stored data segment information, storage location, etc.

[0037] As data is distributed to various data storage devices, the data processor generates global metadata in real time. Global metadata is a summary view of the status of all data fragments in the entire system. Global metadata includes the identifier of each data fragment, location information (i.e., on which device it is stored), status information (whether it has been successfully written), and other necessary meta-information. The generated global metadata is distributed to the corresponding storage service client through the network switch to ensure that each computing server can obtain the latest global metadata according to certain rules. After receiving the global metadata, other computing servers can access the storage data of this computing server based on this information.

[0038] The data to be written is written into the data storage of this node, which improves the storage write bandwidth and reduces the storage read and write latency. After the data is written, when a single computing server is offline, the data on the computing server is unavailable. Therefore, a copy or erasure code can be made in the data storage of this computing server to ensure basic reliability. It may also cause the global data of the system to be unbalanced, with some computing servers having high space utilization and some computing servers having idle space. In order to improve the balance and reliability of the global data of the system, the data processor is also used to migrate the data of each of the data storage devices based on the storage service end using the idle network, see Figure 2 , the migration steps may include: S11: After storing the data to be written into each target data storage device, determine whether the current computing server is using the network at a high load.

[0039] S12: If not, scan the data in each data storage device to find out the first data to be migrated.

[0040] S13: Calculate a migration target data storage of a target computing server to be migrated according to the first data to be migrated.

[0041] S14: Migrate the first data to be migrated to each migration target data storage through the switch, and update the global metadata.

[0042] For example, when computing, the computing server does not have a high demand for network bandwidth. During this process, the storage service client running in the DPU uses the idle network to migrate data based on the data redundancy strategy to achieve higher reliability.

[0043] After the data is written, check whether each computing server is using the network at a high load. If not, start scanning the relevant data storage to find the data that needs to be migrated as the first data to be migrated. It is necessary to decide which data storage on which computing server these data should be migrated to. Selection method of data storage: Prioritize computing server-level reliability. For example, when the erasure code is 4+2, give priority to 6 independent nodes. When nodes that meet the reliability requirements cannot be found, give priority to the node set with the highest total reliability. After determining the migration target data storage, complete the transmission through the switch to ensure that the data can be efficiently and reliably transmitted to the new location. At the same time as the data migration, the global metadata of the system needs to be updated to reflect the new location of the data. After the migration is completed, the global metadata needs to be updated so that each computing server can access the storage data of any node.

[0044] In a possible implementation, in order to prevent the situation where the internal storage space of the computing server is insufficient, see Figure 3 The distributed storage system may further include a hierarchical storage module 130 , which is communicatively connected to each computing server via a switch.

[0045] Among them, the tiered storage module can be an EBOF chassis with only block functions, or it can be an EBOF running a DPU or a standard server running a storage service client. A standard server is a server with the same functions as the storage in a computing server.

[0046] The data processor is also used to check the storage space of the computing server based on the storage service end. When the usage ratio of the storage space is greater than a preset threshold, it finds the second data to be migrated that meets the first preset condition and migrates the second data to be migrated to the tiered storage module through the switch.

[0047] The data processor needs to check the storage space usage of the computing server regularly or on demand. The content of the check may include the total storage capacity, the amount of storage space used, and the remaining available space. The usage ratio of the storage space is calculated. If this ratio exceeds the preset threshold (for example, 80%), the data migration process is triggered to start looking for data items that meet the first preset condition, that is, the second data to be migrated. The first preset condition may include but is not limited to: the data has a low access frequency, the data is created on an earlier date, and the data belongs to a specific type or category. The selected second data to be migrated is migrated to the tiered storage module through the switch to reduce the pressure on the computing server storage.

[0048] Furthermore, the data calculated by the computing tasks of the distributed storage system may be cold data, which is not often used. In order to prevent such data from occupying the storage space in the computing server, such data can be directly written into the hierarchical storage module. Figure 4 The computing server 110 may further include a computing network card 114 , and each computing network card 114 is communicatively connected to each data processor 111 via a switch 120 .

[0049] The computing server is also used to determine whether the data to be written needs to be directly written into the tiered storage module. If so, the data to be written is stored into the tiered storage module through the computing network card.

[0050] Whether the data to be written needs to be written directly into the hierarchical storage module can be determined based on certain strategies. For example, if the data is cold data that will not be used in subsequent calculations, in this case, the computing server will not store the data to be written into the data storage device of the computing server, but will directly write the data into the hierarchical storage module through the computing network card and the switch.

[0051] In supercomputing centers, a large amount of data is often calculated, resulting in a large amount of instantaneous data write, which exceeds the bus bandwidth between the computing server and the data storage. In one possible implementation, the computing server is also used to calculate whether the instantaneous write rate of the data to be written exceeds the bus bandwidth between the computing server and the data storage; if so, a preset proportion of the data to be written is written to the data storage or hierarchical storage module of other computing servers through the computing network card and / or the data processor via the switch.

[0052] When a large amount of data needs to be written, the write IO path of this computing server may reach a bottleneck due to bus bandwidth limitations, resulting in slow writing speed. At this time, data on the computing server may accumulate. In this case, the data can be written to other data storage devices in the system through the DPU or computing network card. The data storage device can be either other peer computing servers or external hierarchical storage modules.

[0053] For example, see Figure 5 , the process of writing to other data storage may include the following steps: S21: Determine whether the storage space usage rate of the local data memory is lower than a preset condition; if so, execute step S24, if not, execute step S22.

[0054] S22: Determine whether the current scan data meets the second preset condition. If yes, execute step S23; if no, execute step S24.

[0055] S23: Writing the qualified data in the data storage managed by the local data processor into the data storage or hierarchical storage module of other computing servers through the computing network card and / or the data processor by the switch.

[0056] S24: Exit.

[0057] When the storage space usage rate of the local data storage device is higher than a certain value, the data will be written into the data storage device or hierarchical storage module of other computing servers. When the storage space usage rate is lower than a certain value, the data will not be written into the data storage device or hierarchical storage module of other computing servers, so it will exit directly without processing. When the storage space usage rate of the local data storage device is higher than a certain value, it will be determined whether the current scanned data meets the second preset condition. If so, the data that meets the condition will be written into the data storage device or hierarchical storage module of other computing servers. If not, it will exit directly without processing.

[0058] Among them, the second preset condition is that the data is cold data. If the scanned data meets the cold data condition, the data needs to be written into the data storage or hierarchical storage module of other computing servers. If it is not cold data, no processing is performed.

[0059] There are many ways to store the data to be written into each target data storage. In one possible implementation, see Figure 6 The data processor is used to receive the data to be written based on the first storage client in the following manner, and store the data to be written into each target data storage through the storage service client: S31: after receiving the data to be written based on the first storage client, determining whether the data to be written requires computationally intensive IO processing.

[0060] After receiving the data to be written, it is generally necessary to process the data before storing it in the data storage device. At this time, it is necessary to first determine whether the data needs to be processed by computationally intensive IO processing, such as data compression and deduplication.

[0061] If not, the DPU's own computing resources can be used directly for processing.

[0062] S32: If yes, determine whether the computing resources are sufficient.

[0063] If computationally intensive IO processing is required, first check whether the DPU's own computing resources are sufficient. If sufficient, the DPU's own computing resources can be used directly for processing.

[0064] S33: If not, call the computing unit of the computing server where the computing unit is located to jointly perform computing-intensive IO processing on the data to be written.

[0065] If the DPU's own computing resources are insufficient to perform compute-intensive IO processing on the data, the DPU calls the CPU and / or GPU of the computing server where it is located to jointly perform compute-intensive IO processing on the data.

[0066] S34: The processed data is stored in each target data storage device through the storage service end.

[0067] After the data processing is completed, it is stored in each target data storage device.

[0068] In another possible implementation, the computing server includes multiple data processors. Figure 7 The data processor is used to receive the data to be written based on the first storage client in the following manner, and store the data to be written into each target data storage through the storage service client: S41: After receiving the data to be written based on the first storage client, determine whether its own computing resources are sufficient to support storage computing of the data to be written.

[0069] Among them, storage computing may include but is not limited to block characteristics, file characteristics, snapshots, clone characteristics, deduplication, compression, encryption, replicas, erasure codes, etc.

[0070] S42: If yes, directly perform storage and calculation processing on the data to be written.

[0071] S43: If not, call other data processors of the computing server where the computing server is located to jointly perform storage computing and processing on the data to be written.

[0072] S44: The processed data is stored in each target data storage device through the storage service end.

[0073] The logic is the same as above. If the computing resources of the DPU itself are sufficient to support the storage and calculation of the data to be written, the storage and calculation processing of the data to be written is directly performed. If it is insufficient to support the storage and calculation of the data to be written, other DPUs in the computing server can be called to jointly perform storage and calculation processing on the data to be written. After the data processing is completed, it is stored in each target data storage device.

[0074] Multiple DPUs inside the computing server give priority to data storage devices that are physically closer, and can also provide high-availability services within the node. When a DPU fails or the link is interrupted, the business and data link are taken over by other nodes.

[0075] Optionally, the data processor is further used to convert the protocol of the storage service client based on the first storage client to expose it as a PCIE device to the computing server; the second storage client is used to convert the PCIE device of the first storage client into a file system or block device.

[0076] The first storage client running on the data processor is responsible for storage access control, protocol conversion, and other advanced features that require computing, and is exposed to the computing server as a PCIE device. The second storage client running on the computing server converts the DPU's PCIE device into specific files (such as through virtio-fs conversion) and blocks (such as through virtio-blk conversion).

[0077] Alternatively, see Figure 8 The computing server may further include a third storage client 115; the third storage client is any one or more of an NFS client, a CIFS client, a POSIX client, and a parallel file system client. The third storage client is used to enable the computing server to flexibly access the data of the data storage. Through a traditional client, the data of each data storage can be directly accessed through a network protocol, which can improve the flexibility of the computing server in accessing data.

[0078] Furthermore, an embodiment of the present invention also provides a distributed reading and writing method, which is applied to the above distributed storage system, wherein the distributed storage system includes a switch and multiple computing servers, each computing server includes a data processor and multiple data storage devices; the data processor is deployed with a storage service client and a first storage client, the computing server is deployed with a second storage client, and each data processor is connected to the switch through communication, see Fig. 9 , the method comprises the following steps: S51: The computing server sends a read / write instruction to the data processor based on the second storage client.

[0079] S52: The data processor receives the read / write instruction based on the first storage client, and performs data interaction with the data storage based on the storage service client according to the read / write instruction.

[0080] In summary, the distributed storage system and distributed reading and writing method provided by the embodiments of the present invention ensure the isolation of storage services and computing services by placing the storage service end into the data processor, saving storage nodes and reducing the overall construction cost of the entire data center. Multiple data processors inside the computing server can provide high-availability services within the node to the outside world. When a data processor fails or the link is interrupted, the business and data link are taken over by other data processors. By first writing data to the data storage device of this node, the storage writing efficiency is improved and the storage reading and writing delay is reduced. Then, through the background migration task, the reliability of the data is guaranteed, and the migration is performed using the idle network, so that the migration task does not occupy the network bandwidth. By extending the hierarchical storage module, it is ensured that computing resources always have sufficient storage space to use. Through the data diversion writing method, the problem that the instantaneous write data amount of the computing task of a single computing server is greater than the insufficient internal bus bandwidth of the node between the computing resources and the storage resources is solved.

[0081] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0083] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only data memory (ROM, Read-Only Memory), random access data memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0084] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0085] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A distributed storage system, characterized in that: It includes a switch and a plurality of computing servers; each of the computing servers includes a data processor and a plurality of data storage devices; the data processor is deployed with a storage service client and a first storage client; the computing server is deployed with a second storage client; each of the data processors is connected in communication via the switch; The computing server is used to send a read and write instruction to the data processor based on the second storage client; The data processor is used to receive the read and write instructions based on the first storage client, and perform data interaction with the data storage based on the storage service client according to the read and write instructions.

2. The distributed storage system according to claim 1, characterized in that: When the read / write instruction is to write data, the data processor is also used to receive the data to be written sent by the computing server based on the first storage client, and store the data to be written into each target data storage device through the storage service client; and, based on the data to be written, generate global metadata, and update the global metadata to each storage service client through the switch; the global metadata is used to record the data status in the data storage device of each computing server, so that each computing server can access the data in the data storage device of any computing server.

3. The distributed storage system according to claim 2, characterized in that: The data processor is further used to migrate the data of each data storage device based on the storage service end by using the idle network, including: After storing the data to be written into each of the target data storage devices, determining whether the current computing server is using the network at a high load; If not, scanning the data in each of the target data storage devices to find out the first data to be migrated; Calculating a migration target data storage of a migration target computing server according to the first data to be migrated; The first data to be migrated is migrated to each of the migration target data storage devices through the switch, and the global metadata is updated.

4. The distributed storage system according to claim 2, characterized in that: The distributed storage system further comprises a hierarchical storage module; the hierarchical storage module is communicatively connected with each of the computing servers via the switch; The data processor is also used to check the storage space of the computing server based on the storage service end, and when the usage ratio of the storage space is greater than a preset threshold, find out the second data to be migrated that meets the first preset condition, and migrate the second data to be migrated to the layered storage module through the switch.

5. The distributed storage system according to claim 4, characterized in that: The computing server further includes a computing network card, each of which is communicatively connected to each of the data processors via the switch; The computing server is further configured to determine whether the data to be written needs to be directly written into the hierarchical storage module, and if so, store the data to be written into the hierarchical storage module through the computing network card.

6. The distributed storage system according to claim 5, characterized in that: The computing server is also used to calculate whether the instantaneous write rate of the data to be written exceeds the bus bandwidth between the computing server and the data storage device; if so, a preset proportion of the data to be written is written into the data storage devices of other computing servers or the hierarchical storage modules through the computing network card and / or the data processor via the switch.

7. The distributed storage system according to claim 6, characterized in that: The computing server is used to write the preset proportion of the to-be-written data into the data storage of each of the other computing servers or the hierarchical storage module through the computing network card and / or the data processor via the switch in the following manner: The data to be written is preferentially written into the data storage device managed by the local data processor. When the local data processor reaches a preset condition, the local data that meets the second preset condition is searched, and the data is written into the data storage device of other computing servers or into the hierarchical storage module by the switch through the computing network card and / or the data processor.

8. The distributed storage system according to claim 2, characterized in that: The data processor is used to receive the data to be written sent by the computing server based on the first storage client in the following manner, and store the data to be written into each target data storage through the storage service client: After receiving the data to be written based on the first storage client, determining whether the data to be written requires computationally intensive IO processing; If so, determine whether its computing resources are sufficient; If not, calling the computing server where the computing server is located to jointly perform computing-intensive IO processing on the data to be written; The processed data is stored in each target data storage device through the storage service end.

9. The distributed storage system according to claim 2, characterized in that: The computing server includes a plurality of data processors; the data processors are used to receive the data to be written sent by the computing server based on the first storage client in the following manner, and store the data to be written into each target data storage through the storage service client: After receiving the data to be written based on the first storage client, determining whether its own computing resources are sufficient to support storage computing of the data to be written; If so, directly perform storage and calculation processing on the data to be written; If not, calling other data processors of the computing server where the computing server is located to jointly perform storage computing processing on the data to be written; The processed data is stored in each target data storage device through the storage service end.

10. The distributed storage system according to any one of claims 1 to 9, characterized in that: The data processor is further used to convert the protocol of the storage service client based on the first storage client to expose it as a PCIE device to the computing server; The second storage client is used to convert the PCIE device of the first storage client into a file system or a block device.

11. The distributed storage system according to any one of claims 1 to 9, characterized in that: The computing server is also deployed with a third storage client; the third storage client is any one or more of an NFS client, a CIFS client, a POSIX client, and a parallel file system client.

12. A distributed reading and writing method, characterized in that: Applied to the distributed storage system according to any one of claims 1 to 11, the distributed storage system comprises a switch and a plurality of computing servers; each of the computing servers comprises a data processor and a plurality of data storage devices; the data processor is deployed with a storage service client and a first storage client; The computing server is deployed with a second storage client; Each of the data processors is communicatively connected via the switch; the method comprises: The computing server sends a read and write instruction to the data processor based on the second storage client; The data processor receives the read / write instruction based on the first storage client, and performs data interaction with the data storage device based on the storage service client according to the read / write instruction.

Images