A distributed storage system, data processing method, device and medium

Abstract

The application relates to the technical field of distributed storage, and discloses a distributed storage system, a data processing method, equipment and a medium. A storage server stores acquired original data in host memory and issues a write request notification to an intelligent network card; the intelligent network card reads out the original data according to the host memory address corresponding to the write request notification; the original data is subjected to erasure operation according to the number of erasure blocks, the obtained erasure blocks are stored in the network card memory, and a write response notification is fed back to the storage server, so that the storage server issues a read command carrying the network card memory address of the corresponding erasure block to each storage node, and each storage node pulls the corresponding erasure block from the network card memory according to the network card memory address carried in the read command. The intelligent network card releases the corresponding network card memory according to the release notification. The intelligent network card is arranged on the storage node, the erasure operation of data is completed by the intelligent network card, and the performance of the distributed storage system is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage, in particular to a distributed storage system, a data processing method, equipment and a medium. BACKGROUND

[0002] Current data centers are applying distributed storage systems on a large scale, and the distributed storage system contains dozens, hundreds or even thousands of nodes. In such a large-scale distributed system, node downtime and failure become the norm, and this design model requires data centers to provide higher performance, higher efficiency and lower total cost of ownership. Therefore, efficient fault-tolerant mechanisms have emerged to enable data lost due to machine and device failure to be recovered. The common practice at present is the multi-copy backup mechanism and the erasure mechanism.

[0003] Distributed storage often uses a multi-copy backup mechanism and an erasure mechanism to ensure the reliability of stored data. Since the erasure mechanism involves a large amount of matrix operations and cross-node data transmission, it will cause a large computational and transmission overhead to the system. However, the current mainstream usage is to use a central processing unit (CPU) to run software to perform the encoding and decoding calculations of the erasure. When the I / O bandwidth of the system is large, the erasure will occupy a lot of CPU resources, and the CPU also has to handle the business of other software modules. The CPU and memory resource preemption caused by erasure operations will inevitably affect the overall performance.

[0004] It can be seen that how to improve the performance of distributed storage is a problem to be solved by those skilled in the art. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a distributed storage system, a data processing method, equipment and a medium, which can solve the problem of low performance of the distributed storage system.

[0006] To solve the above technical problems, the embodiments of the present application provide a distributed storage system, which comprises a plurality of storage nodes, each storage node comprising a storage server and an intelligent network card; when a storage server of any target storage node in the plurality of storage nodes stores original data, the target storage node is taken as a master storage node;

[0007] The storage server of the master storage node is configured to store the obtained original data in the host memory and issue a write request notification to the intelligent network card of the master storage node; the write request notification corresponds to a host memory address and a number of erasure blocks; the storage server receives a write response notification fed back by the intelligent network card of the master storage node, issues a read command carrying a network card memory address of a corresponding erasure block to each storage node, so that each storage node pulls the corresponding erasure block from the network card memory according to the network card memory address carried in the read command; and the storage server issues a release notification to the intelligent network card of the master storage node; the release notification corresponds to a network card memory address to be released.

[0008] The intelligent network card of the master storage node is configured to read the original data from the host memory according to the host memory address corresponding to the write request notification when the write request notification is obtained; perform erasure operation on the original data according to the number of erasure blocks to obtain the same number of erasure blocks as the number of erasure blocks; store each erasure block in the network card memory, and feed back a write response notification to the storage server of the master storage node; the write response notification corresponds to a network card memory address of each erasure block; and release the corresponding network card memory when the release notification issued by the storage server of the master storage node is received.

[0009] In one aspect, the remaining storage nodes except the master storage node in the plurality of storage nodes are used as slave storage nodes.

[0010] The intelligent network card of the slave storage node is configured to determine whether the network card memory address carried in the read command is the network card memory address of the slave storage node when the read command issued by the storage server of the master storage node is received; and pull the corresponding target erasure block from the network card memory corresponding to the network card memory address by using a remote direct data access mode and store the target erasure block in the network card memory of the slave storage node when the network card memory address carried in the read command is not the network card memory address of the slave storage node.

[0011] In one aspect, the storage server of the master storage node is configured to, after storing the obtained raw data in the host memory, put a data write request carrying the host memory address and the number of erasure blocks into a request queue, issue a write request notification to the intelligent network card of the master storage node, read the network card memory address of each erasure block from the response queue and delete the data write request recorded in the request queue in the case of receiving the write response notification issued by the intelligent network card of the master storage node, issue a write address extraction notification to the intelligent network card of the master storage node, issue a read command carrying the network card memory address of the corresponding erasure block to each slave storage node, put a release memory request into a management queue, wherein the release memory request carries the network card memory address to be released, issue a release notification to the intelligent network card of the master storage node, and delete the release memory request recorded in the management queue in the case of receiving the memory release response notification sent by the intelligent network card of the master storage node; and the intelligent network card of the master storage node is configured to, in the case of receiving the write request notification, read the data write request from the request queue and read the raw data from the host memory according to the host memory address carried in the data write request, put the network card memory address of each erasure block into a response queue and issue a write response notification to the storage server of the master storage node after storing each erasure block in the network card memory, read the release memory request from the management queue in the case of receiving the release notification issued by the storage server of the master storage node, release the corresponding network card memory according to the network card memory address to be released carried in the release memory request, delete the network card memory address of each erasure block recorded in the response queue, and send a memory release response notification to the storage server of the master storage node.

[0012] The intelligent network card of the master storage node is configured to, in the case of receiving the write request notification, read the data write request from the request queue and read the raw data from the host memory according to the host memory address carried in the data write request, put the network card memory address of each erasure block into a response queue and issue a write response notification to the storage server of the master storage node after storing each erasure block in the network card memory, read the release memory request from the management queue in the case of receiving the release notification issued by the storage server of the master storage node, release the corresponding network card memory according to the network card memory address to be released carried in the release memory request, delete the network card memory address of each erasure block recorded in the response queue, and send a memory release response notification to the storage server of the master storage node.

[0013] In one aspect, the storage server of the master storage node is configured to issue a read request notification to the intelligent network card of the master storage node, wherein the read request notification corresponds to a data amount and a number of erasure blocks, receive a read response notification fed back by the intelligent network card of the master storage node, issue a write command carrying a target network card memory address to each storage node, so that each storage node writes an erasure block into the network card memory corresponding to the target network card memory address, and allocate a host memory according to the data amount and issue an erasure operation notification to the intelligent network card of the master storage node, wherein the erasure operation notification corresponds to a host memory address of the host memory.

[0014] The intelligent network card of the master storage node is configured to, in a case where the read request notification is acquired, allocate target network card memory according to the data amount and the number of erasure blocks corresponding to the read request notification, and feed back a write response notification to the storage server of the master storage node; wherein the write response notification corresponds to a target network card memory address of the target network card memory; receive the erasure blocks written into the target network card memory by each slave storage node; in a case where an erasure operation notification issued by the storage server of the master storage node is received, perform erasure processing on all the erasure blocks in the target network card memory to obtain reorganized data; and write the reorganized data into the host memory.

[0015] In one aspect, the storage server of the master storage node is configured to put a data read request carrying a data amount and a number of erasure blocks into a request queue; issue a read request notification to the intelligent network card of the master storage node; in a case where a read response notification issued by the intelligent network card of the master storage node is received, read the target network card memory address from a response queue and delete the data read request recorded in the request queue; issue a read address extraction notification to the intelligent network card of the master storage node; issue a write command carrying the target network card memory address to each slave storage node; after allocating host memory according to the data amount, put an erasure operation request into a management queue; wherein the erasure operation request carries a host memory address; issue an erasure operation notification to the intelligent network card of the master storage node; in a case where a reorganization response notification sent by the intelligent network card of the master storage node is received, delete the erasure operation request recorded in the management queue; issue a data extraction notification to the intelligent network card of the master storage node.

[0016] The intelligent network card of the master storage node is configured to, in a case where the read request notification is received, read the data read request from the request queue and allocate target network card memory according to the data amount and the number of erasure blocks carried in the data read request; put a target network card memory address of the target network card memory into a response queue and issue a read response notification to the storage server of the master storage node; in a case where a read address extraction notification issued by the storage server of the master storage node is received, read an erasure operation request from the management queue and write reorganized data into the host memory according to a host memory address carried in the erasure operation request; delete the target network card memory address recorded in the response queue; and send a reorganization response notification to the storage server of the master storage node.

[0017] In one aspect, the intelligent network card comprises an acceleration engine management module, an acceleration engine cache module, and at least one erasure engine.

[0018] The acceleration engine management module is configured to, in the case that the erasure notification issued by the storage server is received, configure a PCIE DMA to turn on a data channel with the storage server, acquire a corresponding erasure request according to the data channel, copy the erasure request to a local buffer of the acceleration engine cache module, and parse a control command issued by the storage server to send the erasure request to a matched erasure engine.

[0019] The erasure engine is configured to acquire the erasure request issued by the acceleration engine management module, parse the erasure request to acquire a to-be-processed data address and a matrix address, acquire to-be-processed data and an erasure matrix according to the to-be-processed data address and the matrix address in a direct memory access mode, copy the to-be-processed data and the erasure matrix to a local buffer of the acceleration engine cache module, perform erasure processing on the to-be-processed data by using the erasure matrix to obtain erasure data, wherein the erasure data is a plurality of erasure blocks obtained by erasure coding or reorganized data obtained by erasure decoding, store the erasure data to an output buffer of the acceleration engine cache module, and feed back the erasure data and response information to a memory of the storage server.

[0020] The acceleration engine cache module is configured to cache the erasure request, the to-be-processed data, the erasure matrix, and the erasure data.

[0021] In one aspect, the intelligent network card further comprises a cyclic redundancy check module and a memory control module.

[0022] The cyclic redundancy check module is configured to perform cyclic redundancy check on the erasure data.

[0023] The memory control module is configured to, in the case that the erasure data passes the cyclic redundancy check, issue the erasure data to a double data rate synchronous dynamic random access memory connected with the intelligent network card.

[0024] In one aspect, the intelligent network card is inserted into a corresponding storage server through a high-speed serial communication interconnection standard.

[0025] In one aspect, storage nodes are interconnected through a remote direct data access network provided by a 2x100G converged Ethernet of the intelligent network card.

[0026] The embodiment of the application further provides a data processing method based on a distributed storage system, which is suitable for an intelligent network card, and the method comprises the following steps:

[0027] In the case of obtaining the write request notification, the original data is read from the host memory according to the host memory address corresponding to the write request notification; wherein the write request notification corresponds to a host memory address and a number of erasure blocks;

[0028] According to the number of erasure blocks, the original data is subjected to erasure operation to obtain the same number of erasure blocks as the number of erasure blocks;

[0029] Each of the erasure blocks is stored in the network card memory, and a write response notification is fed back to the storage server of the master storage node, so that the storage server of the master storage node issues a read command carrying the network card memory address of the corresponding erasure block to each storage node, so that each storage node pulls the corresponding erasure block from the network card memory according to the network card memory address carried in the read command; wherein the write response notification corresponds to the network card memory address of each of the erasure blocks;

[0030] In the case of receiving the release notification issued by the storage server of the master storage node, the corresponding network card memory is released; wherein the release notification corresponds to the network card memory address to be released.

[0031] On the one hand, according to the host memory address corresponding to the write request notification, the original data is read from the host memory, which includes:

[0032] The data write request is read from the request queue, and the original data is read from the host memory according to the host memory address carried in the data write request; wherein the request queue records the data write request carrying the host memory address and the number of erasure blocks put into the request queue by the storage server of the master storage node after storing the obtained original data in the host memory;

[0033] The write response notification is fed back to the storage server of the master storage node, which includes:

[0034] The network card memory address of each of the erasure blocks is put into the response queue, and the write response notification is issued to the storage server of the master storage node, so that the storage server of the master storage node reads the network card memory address of each of the erasure blocks from the response queue and deletes the data write request recorded in the request queue; the write address extraction notification is issued to the intelligent network card of the master storage node; the read command carrying the network card memory address of the corresponding erasure block is issued to each of the slave storage nodes; the release memory request is put into the management queue; wherein the release memory request carries the network card memory address to be released;

[0035] The corresponding network card memory is released, which includes:

[0036] read a release memory request from the management queue, and release corresponding network card memory according to a network card memory address to be released carried in the release memory request.

[0037] In one aspect, the method further comprises:

[0038] In a case where the read request notification is acquired, target network card memory is allocated according to a data amount and a number of erasure blocks corresponding to the read request notification, and a write response notification is fed back to a storage server of the master storage node; the write response notification corresponds to a target network card memory address of the target network card memory.

[0039] The erasure blocks written into the target network card memory by the slave storage nodes are received.

[0040] In a case where an erasure operation notification issued by the storage server of the master storage node is received, all erasure blocks in the target network card memory are subjected to erasure processing to obtain reorganized data, and the reorganized data is written into the host memory.

[0041] In one aspect, allocating target network card memory according to a data amount and a number of erasure blocks corresponding to the read request notification, and feeding back a write response notification to a storage server of the master storage node comprise:

[0042] The data read request is read from the request queue, and target network card memory is allocated according to a data amount and a number of erasure blocks carried in the data read request.

[0043] A target network card memory address of the target network card memory is put into a response queue, and a read response notification is issued to a storage server of the master storage node, so that the storage server of the master storage node reads the target network card memory address from the response queue and deletes the data read request recorded in the request queue in a case where the read response notification is received.

[0044] Embodiments of the present application also provide a data processing device based on a distributed storage system, comprising:

[0045] A memory is configured to store a computer program.

[0046] A processor is configured to execute the computer program to implement steps of the data processing method based on the distributed storage system.

[0047] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program; the computer program is executed by a processor to implement steps of the data processing method based on the distributed storage system.

[0048] It can be seen from the above technical solution that the distributed storage system includes a plurality of storage nodes, each storage node including a storage server and an intelligent network card; when the storage server of any one target storage node in the plurality of storage nodes stores original data, the target storage node is taken as a master storage node; the storage server of the master storage node stores the obtained original data in the host memory and issues a write request notification to the intelligent network card of the master storage node; wherein the write request notification corresponds to a host memory address and an erasure block number; the intelligent network card of the master storage node reads out the original data from the host memory according to the host memory address corresponding to the write request notification when the write request notification is obtained; the original data is subjected to erasure operation according to the erasure block number to obtain the same number of erasure blocks as the erasure block number; each erasure block is stored in the network card memory, and a write response notification is fed back to the storage server of the master storage node; wherein the write response notification corresponds to the network card memory address of each erasure block. The storage server of the master storage node receives the write response notification fed back by the intelligent network card of the master storage node, issues a read command carrying the network card memory address of the corresponding erasure block to each storage node, so that each storage node pulls the corresponding erasure block from the network card memory according to the network card memory address carried in the read command; after each storage node completes the pulling of the erasure block, a completion notification is fed back to the storage server of the master storage node, at which time the storage server can issue a release notification to the intelligent network card of the master storage node; wherein the release notification corresponds to the network card memory address to be released. The intelligent network card of the master storage node releases the corresponding network card memory when receiving the release notification issued by the storage server of the master storage node.

[0049] The beneficial effects of the present application are that for each storage node, by setting an intelligent network card on the storage node, the intelligent network card completes the erasure operation of the data, greatly reducing the workload of the storage server, reducing the impact of CPU and memory resource preemption caused by erasure operation on the overall performance, and improving the performance of the distributed storage system. And the storage server performs the preliminary preparation work of erasure processing including storage of original data and determination of the number of erasure blocks, without the intelligent network card completing all related processing procedures of erasure operation. Therefore, data erasure operation can be completed by interacting between the storage server and the intelligent network card on one storage node. The present application uses an intelligent network card to perform erasure, compared with the traditional scheme, by optimizing the erasure data distribution algorithm, the storage server erasure operation is unloaded while the data copying within the storage cluster is reduced. From the data transmission volume point of view, using the intelligent network card erasure unloading reduces the data transmission volume within the distributed storage cluster, realizes efficient fault tolerance mechanism and calculation unloading of multiple storage nodes on data within the distributed storage cluster, high-speed data transmission and reception, and improves the performance and reliability of the distributed storage system. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0051] Figure 1 A structural schematic diagram of a distributed storage system provided by the embodiment of the present application is shown in the figure.

[0052] Figure 2 A fault-tolerant processing architecture of a new distributed storage system provided by the embodiment of the present application is shown in the figure.

[0053] Figure 3 A schematic diagram of an erasure correction processing flow based on multiple storage nodes provided by the embodiment of the present application is shown in the figure.

[0054] Figure 4 A schematic diagram of I / O communication between a storage server and an intelligent network card provided by the embodiment of the present application is shown in the figure.

[0055] Figure 5 A schematic diagram of a data writing flow provided by the embodiment of the present application is shown in the figure.

[0056] Figure 6 A software deployment schematic diagram of a storage node provided by the embodiment of the present application is shown in the figure.

[0057] Figure 7 A structural schematic diagram of erasure correction firmware on an intelligent network card provided by the embodiment of the present application is shown in the figure.

[0058] Figure 8 A flowchart of a data processing method based on distributed storage provided by the embodiment of the present application is shown in the figure.

[0059] Figure 9 A structural diagram of a data processing device based on distributed storage provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0061] The terms "include", "includes" and "including", as well as any forms of these terms used in the specification and in the accompanying claims, are intended to be inclusive in a manner similar to the term "comprising" as an open transition term without precluding any additional or omissions "comprising" recited elements, steps, methods, objects, or the like.

[0062] For those skilled in the art to better understand the technical scheme of the present application, the present application will be further described in detail below in conjunction with the drawings and specific embodiments.

[0063] Next, a distributed storage system provided by an embodiment of the present application is described in detail. Figure 1 The structure diagram of a distributed storage system provided by an embodiment of the present application is shown in the figure, which includes a plurality of storage nodes, each of which includes a storage server 11 and an intelligent network card 12; when the storage server 11 of any target storage node in the plurality of storage nodes stores original data, the target storage node is taken as a master storage node. Figure 1 The n storage nodes are taken as an example, and the storage node 1 is taken as a master storage node.

[0064] The storage server 11 of the master storage node stores the obtained original data in the host memory and issues a write request notification to the intelligent network card 12 of the master storage node; wherein the write request notification corresponds to a host memory address and a number of erasure blocks.

[0065] The intelligent network card 12 of the master storage node reads the original data from the host memory according to the host memory address corresponding to the write request notification when the write request notification is obtained; and performs erasure operation on the original data according to the number of erasure blocks to obtain the same number of erasure blocks as the number of erasure blocks.

[0066] The erasure code (Erasure Code, EC) technology can include erasure coding operation and erasure decoding operation, wherein the erasure coding operation can perform erasure operation on the original data to obtain a plurality of data blocks and a plurality of check blocks. The erasure decoding operation can perform erasure operation on all data blocks and check blocks to obtain reorganized data.

[0067] In the embodiment of the present application, in order to facilitate description, the data blocks and the check blocks can be collectively referred to as erasure blocks.

[0068] After the intelligent network card 12 of the master storage node performs erasure operation on the original data, it can store each erasure block in the network card memory and feed back a write response notification to the storage server 11 of the master storage node; wherein the write response notification corresponds to the network card memory address of each erasure block.

[0069] The storage server 11 of the master storage node receives the write response notification fed back by the intelligent network card 12 of the master storage node, and issues a read command carrying the network card memory address of the corresponding erasure block to each storage node, so that each storage node pulls the corresponding erasure block from the network card memory according to the network card memory address carried in the read command.

[0070] In the embodiment of the application, the remaining storage nodes except the master storage node in the plurality of storage nodes can be slave storage nodes.

[0071] The storage server 11 of the master storage node can issue a read command to the master storage node and the slave storage nodes. In actual application, the read command issued to different storage nodes can carry the network card memory address corresponding to different erasure blocks, so that each storage node can pull the erasure block corresponding to itself.

[0072] After receiving the read command, the intelligent network card 12 of the master storage node can obtain the corresponding erasure block by using the direct memory access (DMA) mode.

[0073] When the intelligent network card 12 of the slave storage node receives the read command issued by the storage server 11 of the master storage node, it is judged whether the network card memory address carried in the read command is the network card memory address of itself. If the network card memory address carried in the read command is not the network card memory address of itself, the corresponding target erasure block is pulled from the network card memory corresponding to the network card memory address by using the remote direct memory access (RDMA) mode, and the target erasure block is stored in the network card memory of itself.

[0074] In actual application, after each storage node pulls the corresponding erasure block from the network card memory, it can feed back a completion notification to the storage server 11 of the master storage node, so that the storage server 11 of the master storage node can know the reading situation of each erasure block.

[0075] After all the storage nodes pull the erasure block corresponding to themselves, the storage server 11 of the master storage node can issue a release notification to the intelligent network card 12 of the master storage node; the release notification corresponds to the network card memory address to be released. When the intelligent network card 12 of the master storage node receives the release notification issued by the storage server 11 of the master storage node, it releases the corresponding network card memory.

[0076] It can be seen from the above technical solution that the distributed storage system includes a plurality of storage nodes, each storage node including a storage server and an intelligent network card; when the storage server of any target storage node in the plurality of storage nodes stores original data, the target storage node is taken as a master storage node; the storage server of the master storage node stores the obtained original data in the host memory and issues a write request notification to the intelligent network card of the master storage node; wherein the write request notification corresponds to a host memory address and an erasure block number; the intelligent network card of the master storage node reads out the original data from the host memory according to the host memory address corresponding to the write request notification when the write request notification is obtained; the original data is subjected to erasure operation according to the erasure block number to obtain the same number of erasure blocks as the erasure block number; each erasure block is stored in the network card memory, and a write response notification is fed back to the storage server of the master storage node; wherein the write response notification corresponds to the network card memory address of each erasure block. The storage server of the master storage node receives the write response notification fed back by the intelligent network card of the master storage node, issues a read command carrying the network card memory address of the corresponding erasure block to each storage node, so that each storage node pulls the corresponding erasure block from the network card memory according to the network card memory address carried in the read command; after each storage node completes the pulling of the erasure block, a completion notification is fed back to the storage server of the master storage node, at which time the storage server can issue a release notification to the intelligent network card of the master storage node; wherein the release notification corresponds to the network card memory address to be released. The intelligent network card of the master storage node releases the corresponding network card memory when receiving the release notification issued by the storage server of the master storage node.

[0077] The beneficial effects of the present application are that for each storage node, by setting an intelligent network card on the storage node, the intelligent network card completes the erasure operation of the data, greatly reducing the workload of the storage server, reducing the impact of CPU and memory resource preemption caused by erasure operation on the overall performance, and improving the performance of the distributed storage system. And the storage server performs the preliminary preparation work of erasure processing including storage of original data and determination of the number of erasure blocks, without the intelligent network card completing all related processing procedures of erasure operation, so that data erasure operation can be completed by interacting with the storage server through an intelligent network card on a storage node. The present application uses an intelligent network card to perform erasure, compared with the traditional scheme, by optimizing the erasure data distribution algorithm, the storage server erasure operation is unloaded while the data copying within the storage cluster is reduced, from the data transmission amount, the use of intelligent network card erasure unloading reduces the data transmission amount within the distributed storage cluster, and a high-efficiency fault-tolerant mechanism and calculation unloading, high-speed data transmission are realized for data in the distributed storage cluster, and the performance and reliability of the distributed storage system are improved.

[0078] The application adopts an intelligent network card to perform erasure correction, compared with a traditional scheme, by optimizing an erasure correction data distribution algorithm, data copying in a storage cluster is reduced while offloading storage server erasure correction operation.

[0079] Figure 2 A novel distributed storage system fault-tolerant processing architecture is provided for the embodiments of the application, Figure 2 The interaction between two storage nodes is taken as an example, each storage node can communicate with a computing server through a computing-storage service network. Each storage node includes a storage server and a smart network interface card (SmartNIC), and the storage server includes a CPU and a network interface card (NIC) provided by the storage server. In order to realize the interaction between the storage server and the smart network interface card, storage software can be deployed on the storage server, and the storage software can include an object storage device client (OSDC), an object storage device (OSD) controller, a baseboard manager controller (BMC) software adaptation, a basic input output system (BIOS) software adaptation, management software, and bottom software. Various types of firmware are deployed on the Smart NIC, including EC firmware, RDMA firmware, and cyclic redundancy check (CRC) firmware. The bottom software on the storage server is used to trigger the operation of each firmware on the smart network interface card, so as to realize the interaction between the storage server and the smart network interface card.

[0080] The Smart NIC can be inserted into the storage server through a peripheral component interconnect express (PCIe) standard. In the internal network of the distributed storage cluster, the storage nodes can be interconnected through the remote direct data access (RDMA over Converged Ethernet, RoCE) network provided by the 2x100G converged Ethernet on the Smart NIC.

[0081] Figure 3A schematic diagram of a multi-storage node-based erasure processing flow is provided for the embodiments of the present application, each storage node comprising a storage server and an intelligent network card, the storage server can comprise a CPU, a NIC, a host memory connected with the CPU, and a plurality of OSDs. Taking one storage node as an example, the host memory is connected with the CPU on the left and right, and the CPU is connected with the plurality of OSDs through a switch. Figure 3 The component connected with the CPU at the upper left corner is a NIC, and the component connected with the CPU at the upper right corner is a Smart NIC.

[0082] From the perspective of data transmission volume, using the intelligent network card erasure unloading can reduce the data transmission volume in the distributed storage cluster. Taking an erasure ratio (2:1) as an example, the data transmission volume can be reduced by 25%; taking an erasure ratio (8:2) as an example, the data transmission volume can be reduced by 41%.

[0083] Taking an I / O write flow as an example, the storage server can be regarded as a host end, and the system running flow involved in the Smart NIC erasure is as follows: the host end receives a write request, processes according to a block, file, object, etc. protocol, and parses the original data; the host end calls the EC firmware on the intelligent network card through the OSDC to perform erasure coding on the original data. In actual application, the host end can call an RDMA interface, register the host memory, and return the host memory address at the same time; copy the coded data to the host memory; encapsulate the write request, package the host memory address into the request; send a message to the master storage node; the master storage node parses the message, sends a sub-message to all slave storage nodes; all master and slave storage nodes can call a unilateral reading to pull the data; the request returns a response message (reply) from the master storage node.

[0084] When the storage server is connected with the intelligent network card, the PCIe interface communication mode is followed, and the management and information query are directly queried and set by using the intelligent network card PCIe device configuration space register. For the user space query and setting of the storage server, the intelligent network card driver provides a character device interface, and the user space program finally operates the intelligent network card PCIe device configuration space register through the character device.

[0085] Figure 4 A schematic diagram of I / O communication between a storage server and an intelligent network card is provided for the embodiments of the present application, the storage server serving as a host end, and the large block data transmission between the host end and the intelligent network card being completed by the intelligent network card DMA chip operating the host memory. A request queue and a response queue are established between the host and the intelligent network card, and the request queue and the response queue form an I / O queue pair. In order to distinguish between management operations and I / O operations, a management queue is established for each intelligent network card, and the I / O queue pair can be defined to support one or more according to the number of concurrent supports of the intelligent network card.

[0086] Taking a data write flow as an example, after storing the obtained original data in the host memory, the storage server 11 of the master storage node can place a write request, i.e., a data write request carrying the host memory address and the number of erasure blocks, into a request queue, and send a write request notification to the intelligent network card 12 of the master storage node. The intelligent network card 12 of the master storage node reads the write request from the request queue upon receiving the request notification. Then, the intelligent network card 12 performs erasure processing, which includes reading the original data from the host memory according to the host memory address carried in the data write request, performing erasure operation on the original data according to the number of erasure blocks to obtain the same number of erasure blocks as the number of erasure blocks, and storing the erasure blocks in the network card memory. After storing the erasure blocks in the network card memory, the intelligent network card can respond, i.e., place the network card memory addresses of the erasure blocks into a response queue, and send a write response notification to the storage server 11 of the master storage node.

[0087] Upon receiving the write response notification sent by the intelligent network card 12 of the master storage node, the storage server 11 of the master storage node processes the response. The process of processing the response includes reading the network card memory addresses of the erasure blocks from the response queue, and deleting the data write request recorded in the request queue; sending a write address extraction notification to the intelligent network card 12 of the master storage node; sending a read command carrying the network card memory addresses of the corresponding erasure blocks to each slave storage node; placing a memory release request into a management queue, wherein the memory release request carries the network card memory address to be released; and sending a release notification to the intelligent network card 12 of the master storage node.

[0088] Upon receiving the release notification sent by the storage server 11 of the master storage node, the intelligent network card 12 of the master storage node reads the memory release request from the management queue, and releases the corresponding network card memory according to the network card memory address to be released carried in the memory release request; deletes the network card memory addresses of the erasure blocks recorded in the response queue; and sends a memory release response notification to the storage server 11 of the master storage node.

[0089] Upon receiving the memory release response notification sent by the intelligent network card 12 of the master storage node, the storage server 11 of the master storage node deletes the memory release request recorded in the management queue; and sends a response extraction notification to the intelligent network card 12 of the master storage node, indicating that the data write flow has been completed.

[0090] In the embodiment of the application, the I / O queue is used to process the file cache related to the erasure calculation, which ensures the orderly execution of the erasure operation flow, improves the response speed when the storage server and the intelligent network card interact with data, realizes high-speed data transmission and reception, and improves the performance and reliability of the distributed storage system.

[0091] Figure 5A schematic diagram of a data writing process provided in an embodiment of the present application, a host end can store original data in a host memory, Figure 5 In the embodiment, the original data is represented by numbers 1 to 16. The host end issues a write request notification to the intelligent network card. The intelligent network card can DMA the original data to the network card memory, and the intelligent network card performs erasure operation on the original data. When the data needs multiple strips to be put down, multiple rounds of calculation can be performed. The erasure blocks recombined by the erasure operation can be taken away by each storage node.

[0092] Figure 5 In the embodiment, the original data is split into four groups, the first group of data "1 2 3 4" performs erasure operation, and the erasure data "1 2 3 4 A a" can be obtained; the second group of data "5 6 7 8" performs erasure operation, and the erasure data "5 6 7 8 B b" can be obtained; the third group of data "9 10 11 12" performs erasure operation, and the erasure data "9 10 11 12 C Y" can be obtained; the fourth group of data "13 14 15 16" performs erasure operation, and the erasure data "13 14 15 16 D d" can be obtained. After recombination, six erasure blocks can be obtained, four data blocks and two check blocks. The first data block is composed of "1 5 9 13", the second data block is composed of "2 6 10 14", the third data block is composed of "3 7 11 15", and the third data block is composed of "4 8 12 16". The first check block is composed of "A B C D", and the second check block is composed of "a b Y d". In the embodiment of the present application, the data blocks and the check blocks are collectively referred to as erasure blocks. Different erasure blocks can be stored on different OSDs of different storage nodes. Figure 5 In the embodiment, six erasure blocks are stored on six OSDs, namely OSD1 to OSD6.

[0093] In the embodiment of the present application, for large data, the data can be split into multiple strips, and multiple rounds of erasure operation are performed to ensure the smooth execution of data erasure operation. After obtaining multiple erasure blocks, the multiple erasure blocks are stored on different OSDs to ensure the security of the erasure blocks. Even if the erasure blocks on a certain OSD are damaged, the original data can be restored by reading the erasure blocks on other OSDs.

[0094] The above description is based on the data writing process. Next, the data reading process will be introduced.

[0095] The storage server 11 of the master storage node can issue a read request notification to the intelligent network card 12 of the master storage node, wherein the read request notification corresponds to a data amount and an erasure block number. Based on the data amount, the size of the memory space required by the data can be determined, and based on the erasure block number, the specific form of the erasure matrix required for performing erasure operation can be determined.

[0096] In the embodiment of the application, the memory corresponding to the storage server can be referred to as host memory, and the memory corresponding to the intelligent network card can be referred to as network card memory.

[0097] The intelligent network card 12 of the master storage node allocates target network card memory according to the data amount and the erasure block number corresponding to the read request notification when the read request notification is obtained, and feeds back a write response notification to the storage server 11 of the master storage node, wherein the write response notification corresponds to a target network card memory address of the target network card memory.

[0098] The storage server 11 of the master storage node receives the read response notification fed back by the intelligent network card 12 of the master storage node, and issues a write command carrying the target network card memory address to each storage node, so as to write the erasure block into the network card memory corresponding to the target network card memory address by each storage node.

[0099] The storage server 11 of the master storage node can issue a write command to the master storage node and the slave storage node, and the slave storage node can read the erasure block recorded in the target network card memory in an RDMA mode. The intelligent network card 12 of the master storage node can receive the erasure block written into the target network card memory by each slave storage node, and the master storage node can read the erasure block recorded in the target network card memory in a DMA mode.

[0100] The storage server 11 of the master storage node allocates host memory according to the data amount, and issues an erasure operation notification to the intelligent network card 12 of the master storage node, wherein the erasure operation notification corresponds to a host memory address of the host memory.

[0101] The intelligent network card 12 of the master storage node performs erasure processing on all erasure blocks in the target network card memory to obtain reorganized data when the erasure operation notification issued by the storage server 11 of the master storage node is received, and writes the reorganized data into the host memory.

[0102] In the embodiment of the application, for the data reading process, a message queue and a response queue can also be used to store the relevant requests and notifications in the reading process. In order to distinguish between management operations and I / O operations, a management queue can be established on each intelligent network card.

[0103] The storage server 11 of the master storage node can put the data read request carrying the data amount and the number of erasure blocks into a request queue; and send a read request notification to the intelligent network card 12 of the master storage node.

[0104] The intelligent network card 12 of the master storage node reads the data read request from the request queue in the case of receiving the read request notification, and allocates target network card memory according to the data amount and the number of erasure blocks carried in the data read request; puts the target network card memory address of the target network card memory into a response queue, and sends a read response notification to the storage server 11 of the master storage node.

[0105] The storage server 11 of the master storage node reads the target network card memory address from the response queue and deletes the data read request recorded in the request queue in the case of receiving the read response notification sent by the intelligent network card 12 of the master storage node. The storage server 11 of the master storage node sends a read address extraction notification to the intelligent network card 12 of the master storage node; and sends a write command carrying the target network card memory address to each slave storage node, so that each slave storage node can write the erasure block into the network card memory corresponding to the target network card memory address after receiving the write command.

[0106] After the storage server 11 of the master storage node allocates the host memory according to the data amount, the storage server 11 can put the erasure operation request into a management queue; the erasure operation request carries the host memory address; and sends an erasure operation notification to the intelligent network card 12 of the master storage node, so that the intelligent network card 12 performs erasure processing on all erasure blocks in the target network card memory to obtain reorganized data; and writes the reorganized data into the host memory.

[0107] The intelligent network card 12 of the master storage node reads the erasure operation request from the management queue and writes the reorganized data into the host memory according to the host memory address carried in the erasure operation request in the case of receiving the read address extraction notification sent by the storage server 11 of the master storage node; deletes the target network card memory address recorded in the response queue; and sends a reorganization response notification to the storage server 11 of the master storage node.

[0108] The storage server 11 of the master storage node deletes the erasure operation request recorded in the management queue in the case of receiving the reorganization response notification sent by the intelligent network card 12 of the master storage node; and sends a data extraction notification to the intelligent network card 12 of the master storage node.

[0109] In the embodiment of the present application, by the interaction of the storage server and the intelligent network card, the erasure correction operation is offloaded to the intelligent network card for execution, the pressure of the storage server is relieved, and the CPU and memory resources are solved in the traditional way. In order to realize the communication between the storage server and the intelligent network card, the software and hardware design on the storage server and the intelligent network card is needed to ensure that the intelligent network card completes the erasure correction operation of data.

[0110] The hardware of the intelligent network card can use a Field-Programmable Gate Array (FPGA) chip. The erasure correction module, RDMA module and CRC module are written by RTL language. The ARM core can be used for firmware management in the FPGA.

[0111] For software design, it can be divided into an acceleration card driver running in the storage server, an EC firmware running in the intelligent network card, an RDMA firmware, a CRC firmware and a management module running on the ARM core.

[0112] Figure 6 A software deployment schematic diagram of a storage node provided by the embodiment of the present application includes an acceleration card driver, an object storage device host (OSDC), management software adaptation, BMC software adaptation and BIOS software adaptation deployed on the storage server. In actual application, the acceleration card driver can be referred to as bottom software, the management software adaptation can be referred to as management software, and the object storage device host can be referred to as core software. The EC firmware, RDMA firmware and CRC firmware can be deployed on the intelligent network card. In order to realize the control of each firmware, the driver on the storage server can include the corresponding drivers of EC, RDMA and CRC.

[0113] The acceleration card driver is divided into two parts of kernel state and user state. The kernel state driver runs in the Linux kernel space, including a device management driver (Insp_accl_driver), a UIO device driver (insp_uio), a memory management driver (Insp_dma_buf) and a process management driver (Insp_process_mngr). The user state driver runs in the user space, including a user state memory management driver (insp_dma_buf_user) and a user state acceleration card driver (insp_accl_user).

[0114] The device management driver contains functions such as firmware download, heartbeat management, accelerator hardware management, and UIO device management. A user state program indirectly accesses the PCIe Misc BAR space of the acceleration card by accessing the / dev / insp_accl_ctrl device node provided by the device management driver, to complete the control of each subsystem of the acceleration card, including initialization, enabling, state configuration and query, etc.

[0115] The UIO device driver provides a UIO device node to the user space under the management of the device management driver, and provides address mapping functions of the UIO device to the PCIe ETR BAR. The user state driver can directly access the PCIe BAR space of the acceleration card through the physical address mapping of the UIO device, to complete specific acceleration task processing.

[0116] The memory management driver provides a continuous physical memory allocation function to the user state. The user state driver program can apply for continuous physical memory by accessing the memory management driver device node / dev / insp_dma_buf.

[0117] The process management driver is mainly for the management of kernel state resources. The user state multi-process accesses the hardware accelerator, and the corresponding hardware accelerator resources need to be reasonably allocated according to the configuration mode.

[0118] The user state memory management driver provides a fixed 2MB and 4KB memory block allocation interface to the application program. The allocated memory is used for input and output buffer and IO queue buffer of the acceleration card hardware. The 2MB memory allocation is provided by the Linux operating system hugepage driver, and the user state memory management driver only records the memory application. The 4KB memory is allocated by the kernel state memory management driver.

[0119] The user state acceleration card driver is responsible for the management of sending and receiving rings, and is responsible for assembling business requests and analyzing business responses.

[0120] Taking a common acceleration request as an example, the business process is as follows: a user application program enables all instances through a device management driver. The user application program obtains corresponding instance resources through a process management driver. The user application program completes allocation of physical memory of a request / response ring and allocation of memory of an input / output buffer through a user memory management driver. The user application program writes the physical memory address and size of the ring into the corresponding register of a PCIe ETR bar through a UIO device driver. The user application program initiates a business request through a user acceleration card driver, and request information is filled into the request ring. The user application program accesses the PCIe ETR bar space of the acceleration card through the UIO device driver, updates the TAIL register, and notifies the hardware to start processing the request. The user application program starts polling the response ring to check whether the request is completed. After the hardware processes the request, the corresponding node in the response ring is updated. The user application program obtains the response through the user acceleration card driver, and the business request is completed.

[0121] In the embodiment of the application, by deploying OSDC on the storage server, the OSDC can realize data stripe alignment, memory registration, sending of control messages to the intelligent network card of each storage node, etc. By deploying the acceleration card driver on the storage server, the storage server can call the intelligent network card to perform data erasure operation, data transfer, cyclic redundancy check, etc.

[0122] In the embodiment of the application, the erasure correction firmware can be deployed on the intelligent network card to realize data erasure operation. For the functions required to be realized by the erasure correction firmware, the erasure correction firmware can be divided into different functional modules.

[0123] Figure 7 A structure diagram of the erasure correction firmware on the intelligent network card is provided in the embodiment of the application, Figure 7 The erasure correction firmware of the intelligent network card can include an acceleration engine management module (AEM), an acceleration engine buffer pool (AEBP), and at least one erasure engine (Erasure), taking the interaction between the erasure correction firmware and the storage server as an example. The erasure correction firmware can communicate with the PCIe4 RC of the storage server through the PCIe4 EP, realizing communication between the erasure correction firmware and the storage server. The intelligent network card can communicate with the baseboard management controller through I2C. The intelligent network card can also deploy a memory control module (DDR Ctrl) to transmit data to or read data from a double data rate (DDR). The management module can also run on the ARM core (ARM Core) on the intelligent network card.

[0124] The storage server can prepare a corresponding erasure request according to a predetermined command format after the data is ready, and then issue an erasure notification to the acceleration engine management module of the intelligent network card. In the case that the acceleration engine management module receives the erasure notification issued by the storage server 11, the PCIE DMA is configured to turn on the data path with the storage server 11. According to the data path, the corresponding erasure request is obtained; the erasure request is copied to the local buffer of the acceleration engine cache module. The control command issued by the storage server 11 is parsed to send the erasure request to the matching erasure engine. The acceleration engine cache module can be used to cache erasure requests, to-be-processed data, erasure matrices, and erasure data, etc.

[0125] After the erasure engine obtains the erasure request issued by the acceleration engine management module, the erasure request can be parsed to obtain the to-be-processed data address and the matrix address. According to the to-be-processed data address and the matrix address, the to-be-processed data and the erasure matrix are obtained in a direct memory access mode, and the to-be-processed data and the erasure matrix are copied to the local buffer of the acceleration engine cache module. The to-be-processed data is processed by using the erasure matrix to obtain erasure data; wherein the erasure data is a plurality of erasure blocks obtained by erasure coding or reorganized data obtained by erasure decoding.

[0126] The erasure engine can store the erasure data to the output buffer of the acceleration engine cache module, and feed back the erasure data and response information (response) to the memory of the storage server 11. The memory of the storage server 11 can use DDR. The interface function provided by the intelligent network card can be a callback function, which provides data erasure calculation results, including erasure calculation result correct or erasure calculation result error, etc.

[0127] In the embodiment of the application, a cyclic redundancy check module can also be deployed on the intelligent network card 12; the cyclic redundancy check module is used for cyclic redundancy check on the erasure data. After the intelligent network card 12 obtains the data, the network card CRC interface of the local storage node can be called to complete the CRC calculation.

[0128] In the case that the erasure data passes the cyclic redundancy check, the intelligent network card can issue the erasure data to the double data rate synchronous dynamic random access memory connected with the intelligent network card 12 through the memory control module.

[0129] In the embodiment of the present application, after the storage server receives the I / O request sent by the computing server and completes the processing of the storage protocol, the erasure correction engine of the intelligent network card can be used to perform erasure correction operation to obtain erasure correction data, thereby reducing the workload of the storage server and improving the system performance. By deploying different firmware on the intelligent network card, the intelligent network card can support data transfer, data erasure correction and data verification functions.

[0130] Figure 8 A flowchart of a data processing method based on distributed storage is provided in the embodiment of the present application, which is suitable for an intelligent network card, and the method comprises:

[0131] S801: In the case of obtaining a write request notification, the original data is read from the host memory according to the host memory address corresponding to the write request notification.

[0132] The write request notification corresponds to a host memory address and a number of erasure correction blocks.

[0133] S802: Perform erasure correction operation on the original data according to the number of erasure correction blocks to obtain the same number of erasure correction blocks as the number of erasure correction blocks.

[0134] S803: Store each erasure correction block in the network card memory, and feed back a write response notification to the storage server of the master storage node, so that the storage server of the master storage node issues a read command carrying the network card memory address of the corresponding erasure correction block to each storage node, so that each storage node pulls the corresponding erasure correction block from the network card memory according to the network card memory address carried in the read command.

[0135] The write response notification corresponds to the network card memory address of each erasure correction block.

[0136] S804: In the case of receiving the release notification issued by the storage server of the master storage node, release the corresponding network card memory.

[0137] The release notification corresponds to the network card memory address to be released.

[0138] Figure 8 The description of the features in the corresponding embodiment can be referred to Figure 1 The related description of the corresponding embodiment will not be repeated here.

[0139] It can be seen from the above technical solution that the intelligent network card reads out the original data from the host memory according to the host memory address corresponding to the write request notification in the case of obtaining the write request notification. The write request notification corresponds to the host memory address and the number of erasure blocks. The original data is subjected to erasure operation according to the number of erasure blocks to obtain the same number of erasure blocks as the number of erasure blocks. Each erasure block is stored in the network card memory, and a write response notification is fed back to the storage server of the master storage node, so that the storage server of the master storage node issues a read command carrying the network card memory address of the corresponding erasure block to each storage node, so that each storage node pulls the corresponding erasure block from the network card memory according to the network card memory address carried in the read command. The write response notification corresponds to the network card memory address of each erasure block. In the case of receiving the release notification issued by the storage server of the master storage node, the corresponding network card memory is released; the release notification corresponds to the network card memory address to be released. In this technical solution, the intelligent network card completes the erasure operation of the data, greatly reducing the workload of the storage server, reducing the influence of CPU and memory resource occupation caused by erasure operation on the overall performance, and improving the performance of the distributed storage system.

[0140] Figure 9 The structure diagram of the data processing device based on distributed storage provided by the embodiment of the application is shown in Figure 9 The data processing device based on distributed storage includes a memory 90 for storing a computer program.

[0141] The processor 91 is used to execute the computer program to realize the steps of the data processing method based on distributed storage of the above-mentioned embodiment.

[0142] The data processing device based on distributed storage provided by the embodiment of the application can include but is not limited to a smart phone, a tablet computer, a notebook computer or a desktop computer, etc.

[0143] The processor 91 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 91 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 91 may also include a main processor and a coprocessor. The main processor, also known as the CPU, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 91 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 91 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0144] The memory 90 may include one or more computer-readable storage media, which may be non-transitory. The memory 90 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 90 is used to store at least the following computer program 901, which, after being loaded and executed by the processor 91, is capable of implementing the relevant steps of the data processing method based on distributed storage disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 90 may also include an operating system 902 and data 903, and the storage method may be temporary or permanent storage. The operating system 902 may include Windows, Unix, Linux, etc. The data 903 may include, but is not limited to, raw data, erasure blocks, etc.

[0145] In some embodiments, the data processing device based on distributed storage may further include a display screen 92, an input / output interface 93, a communication interface 94, a power supply 95, and a communication bus 96.

[0146] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on data processing devices based on distributed storage and may include more or fewer components than illustrated.

[0147] It can be understood that if the data processing method based on distributed storage in the above embodiments is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and performs all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), an electrically erasable programmable ROM, a register, a hard disk, a removable magnetic disk, a CD-ROM, a magnetic disk or an optical disk, and various media that can store program codes.

[0148] Based on this, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the data processing method based on distributed storage are implemented.

[0149] The above describes a distributed storage system, a data processing method, an apparatus and a computer readable storage medium provided by the embodiment of the present application in detail. The embodiments in the specification are described in a progressive manner, and each embodiment mainly describes the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the apparatus disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0150] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0151] The above describes in detail the distributed storage system, the data processing method, the equipment and the computer readable storage medium provided by the application. The principles and implementation manners of the application are described by applying specific examples in this paper, and the above description of the examples is only used to help understand the method of the application and its core idea. It should be pointed out that, for ordinary skilled persons in the technical field, some improvements and modifications can be made to the application without departing from the principles of the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

Claims

1. A distributed storage system, characterized by, The method comprises the following steps: including a plurality of storage nodes, each of which comprises a storage server and an intelligent network card; when a storage server of any target storage node in the plurality of storage nodes stores original data, the target storage node is taken as a master storage node; The storage server of the master storage node is configured to store the obtained original data in a host memory and issue a write request notification to the intelligent network card of the master storage node; the write request notification corresponds to a host memory address and a number of erasure blocks; a write response notification fed back by the intelligent network card of the master storage node is received, and a read command carrying a network card memory address of the corresponding erasure block is issued to each storage node, so that each storage node pulls the corresponding erasure block from the network card memory according to the network card memory address carried in the read command; a release notification is issued to the intelligent network card of the master storage node; the release notification corresponds to a network card memory address to be released; The intelligent network card of the master storage node is configured to, when the write request notification is obtained, read the original data from the host memory according to the host memory address corresponding to the write request notification; perform erasure operation on the original data according to the number of erasure blocks to obtain the same number of erasure blocks as the number of erasure blocks; store each erasure block in the network card memory and feed back a write response notification to the storage server of the master storage node; the write response notification corresponds to the network card memory address of each erasure block; and release the corresponding network card memory when the release notification issued by the storage server of the master storage node is received; The intelligent network card comprises an acceleration engine management module, an acceleration engine cache module and at least one erasure engine; The acceleration engine management module is configured to, when the erasure notification issued by the storage server is received, configure a PCIE DMA to turn on a data path with the storage server; obtain a corresponding erasure request according to the data path; copy the erasure request to a local buffer of the acceleration engine cache module; and analyze the control command issued by the storage server to send the erasure request to a matched erasure engine; The erasure engine is configured to obtain the erasure request issued by the acceleration engine management module; analyze the erasure request to obtain a to-be-processed data address and a matrix address; obtain to-be-processed data and an erasure matrix according to the to-be-processed data address and the matrix address in a direct memory access mode, and copy the to-be-processed data and the erasure matrix to a local buffer of the acceleration engine cache module; perform erasure processing on the to-be-processed data by using the erasure matrix to obtain erasure data; the erasure data is a plurality of erasure blocks obtained by erasure encoding or reorganized data obtained by erasure decoding; store the erasure data to an output buffer of the acceleration engine cache module; and feed back the erasure data and response information to the storage server; an interface function provided by the intelligent network card is a callback function, which provides data erasure calculation results, including correct erasure calculation results or incorrect erasure calculation results. The acceleration engine cache module is configured to cache the erasure correction request, the to-be-processed data, the erasure correction matrix, and the erasure correction data. The intelligent network card further comprises a cyclic redundancy check module and a memory control module. The cyclic redundancy check module is configured to perform cyclic redundancy check on the erasure correction data. The memory control module is configured to, in a case where the erasure correction data passes the cyclic redundancy check, issue the erasure correction data to a double data rate synchronous dynamic random access memory connected to the intelligent network card.

2. The distributed storage system of claim 1, wherein, The remaining storage nodes except the master storage node in the plurality of storage nodes are used as slave storage nodes. The intelligent network card of the slave storage node is configured to, in a case where a read command issued by a storage server of the master storage node is received, determine whether a network card memory address carried in the read command is a network card memory address of the slave storage node. In a case where the network card memory address carried in the read command is not the network card memory address of the slave storage node, the slave storage node is configured to pull a target erasure block corresponding to the network card memory address from a network card memory corresponding to the network card memory address by using a remote direct memory access mode, and store the target erasure block in a network card memory of the slave storage node.

3. The distributed storage system of claim 1, wherein, The storage server of the master storage node is configured to, after storing the obtained original data in a host memory, put a data write request carrying the host memory address and the number of erasure blocks into a request queue. The storage server of the master storage node is configured to, in a case where a write response notification issued by the intelligent network card of the master storage node is received, read network card memory addresses of the erasure blocks from the response queue, and delete the data write request recorded in the request queue. The storage server of the master storage node is configured to issue a write address extraction notification to the intelligent network card of the master storage node, and issue a read command carrying a network card memory address of a corresponding erasure block to each slave storage node. The storage server of the master storage node is configured to put a release memory request into a management queue, wherein the release memory request carries a network card memory address to be released, issue a release notification to the intelligent network card of the master storage node, in a case where a memory release response notification sent by the intelligent network card of the master storage node is received, delete the release memory request recorded in the management queue, and issue a response extraction notification to the intelligent network card of the master storage node. The intelligent network card of the master storage node is configured to read the data write request from the request queue and read the original data from the host memory according to the host memory address carried in the data write request when the write request notification is received; after storing each erasure block in the network card memory, put the network card memory address of each erasure block into the response queue and send a write response notification to the storage server of the master storage node; read the release memory request from the management queue and release the corresponding network card memory according to the network card memory address to be released carried in the release memory request when the release notification sent by the storage server of the master storage node is received; delete the network card memory address of each erasure block recorded in the response queue; and send a memory release response notification to the storage server of the master storage node.

4. The distributed storage system of claim 1, wherein, The storage server of the master storage node is configured to send a read request notification to the intelligent network card of the master storage node; the read request notification corresponds to a data amount and an erasure block number; receive the read response notification fed back by the intelligent network card of the master storage node, send a write command carrying a target network card memory address to each storage node, so that each storage node writes an erasure block into the network card memory corresponding to the target network card memory address; allocate host memory according to the data amount and send an erasure operation notification to the intelligent network card of the master storage node; the erasure operation notification corresponds to a host memory address of the host memory; The intelligent network card of the master storage node is configured to allocate a target network card memory according to the data amount and the erasure block number corresponding to the read request notification when the read request notification is obtained, and feed back a write response notification to the storage server of the master storage node; the write response notification corresponds to a target network card memory address of the target network card memory; receive the erasure blocks written into the target network card memory by each slave storage node; perform erasure processing on all erasure blocks in the target network card memory to obtain reorganized data when the erasure operation notification sent by the storage server of the master storage node is received; and write the reorganized data into the host memory.

5. The distributed storage system of claim 4, wherein, The storage server of the master storage node is configured to put a data read request carrying a data amount and an erasure block number into a request queue; send a read request notification to the intelligent network card of the master storage node; Read the target network card memory address from the response queue and delete the data read request recorded in the request queue when the read response notification sent by the intelligent network card of the master storage node is received. The intelligent network card of the master storage node is configured to, in a case of receiving the read request notification, read the data read request from the request queue, and allocate target network card memory according to the data amount and the number of erasure blocks carried in the data read request; put the target network card memory address of the target network card memory into a response queue, and send a read response notification to the storage server of the master storage node; in a case of receiving the read address extraction notification sent by the storage server of the master storage node, read an erasure operation request from the management queue, and write reorganized data into the host memory according to the host memory address carried in the erasure operation request; and delete the target network card memory address recorded in the response queue, and send a reorganization response notification to the storage server of the master storage node. The intelligent network card of the master storage node is configured to, in a case of receiving the read request notification, read the data read request from the request queue, and allocate target network card memory according to the data amount and the number of erasure blocks carried in the data read request; put the target network card memory address of the target network card memory into a response queue, and send a read response notification to the storage server of the master storage node; in a case of receiving the read address extraction notification sent by the storage server of the master storage node, read an erasure operation request from the management queue, and write reorganized data into the host memory according to the host memory address carried in the erasure operation request; and delete the target network card memory address recorded in the response queue, and send a reorganization response notification to the storage server of the master storage node.

6. The distributed storage system of claim 1, wherein, The intelligent network card is inserted into the corresponding storage server through a high-speed serial communication interconnection standard.

7. The distributed storage system of claim 1, wherein, The storage nodes are interconnected through a remote direct data access network provided by a 2*100G converged Ethernet of the intelligent network card.

8. A data processing method based on a distributed storage system, characterized by, The method is suitable for an intelligent network card, and the method comprises the following steps: In a case of obtaining a write request notification, original data is read from the host memory according to the host memory address corresponding to the write request notification; the write request notification corresponds to a host memory address and a number of erasure blocks; An erasure operation is performed on the original data according to the number of erasure blocks, so as to obtain the same number of erasure blocks as the number of erasure blocks; Each of the erasure blocks is stored in a network card memory, and a write response notification corresponding to the network card memory address of each of the erasure blocks is fed back to the storage server of the master storage node, so that the storage server of the master storage node sends a read command carrying the network card memory address of the corresponding erasure block to each of the storage nodes, and each of the storage nodes pulls the corresponding erasure block from the network card memory according to the network card memory address carried in the read command; In a case of receiving a release notification sent by the storage server of the master storage node, the corresponding network card memory is released; the release notification corresponds to a network card memory address to be released. The intelligent network card comprises an acceleration engine management module, an acceleration engine cache module and at least one erasure correction engine; the acceleration engine management module is configured to, in the case of receiving the erasure correction notification issued by the storage server, configure the PCI DMA to turn on the data path with the storage server; obtain the corresponding erasure correction request according to the data path; copy the erasure correction request to the local buffer of the acceleration engine cache module; analyze the control command issued by the storage server to send the erasure correction request to the matched erasure correction engine; the erasure correction engine is configured to obtain the erasure correction request issued by the acceleration engine management module; analyze the erasure correction request to obtain the data address to be processed and the matrix address; obtain the data to be processed and the erasure correction matrix according to the data address to be processed and the matrix address in the direct memory access mode, and copy the data to be processed and the erasure correction matrix to the local buffer of the acceleration engine cache module; perform erasure correction processing on the data to be processed by using the erasure correction matrix to obtain erasure correction data; wherein the erasure correction data is a plurality of erasure correction blocks obtained by erasure correction coding or reorganized data obtained by erasure correction decoding; store the erasure correction data to the output buffer of the acceleration engine cache module; feed back the erasure correction data and response information to the memory of the storage server; the interface function provided by the intelligent network card is a callback function, which provides data erasure correction calculation results, including correct erasure correction calculation results or incorrect erasure correction calculation results; the acceleration engine cache module is configured to cache the erasure correction request, the data to be processed, the erasure correction matrix and the erasure correction data; The intelligent network card further comprises a cyclic redundancy check module and a memory control module; the cyclic redundancy check module is configured to perform cyclic redundancy check on the erasure correction data; the intelligent network card calls the network card CRC interface of the local storage node to complete CRC calculation; the memory control module is configured to, in the case that the erasure correction data passes the cyclic redundancy check, issue the erasure correction data to the double data rate synchronous dynamic random access memory connected with the intelligent network card.

9. The data processing method based on a distributed storage system according to claim 8, characterized in that, According to the host memory address corresponding to the write request, the original data is read from the host memory, which comprises: reading the data write request from the request queue and reading the original data from the host memory according to the host memory address carried in the data write request; wherein the request queue records the data write request carrying the host memory address and the number of erasure correction blocks, which is put into the request queue by the storage server of the host storage node after storing the obtained original data in the host memory; feeding back the write response notification to the storage server of the host storage node comprises: The NIC memory addresses of each of the erasure blocks are put into a response queue, and a write response notification is issued to the storage server of the master storage node, so that the storage server of the master storage node reads the NIC memory addresses of each of the erasure blocks from the response queue and deletes the data write request recorded in the request queue; a write address extraction notification is issued to the intelligent NIC of the master storage node; a read command carrying the NIC memory address of the corresponding erasure block is issued to each slave storage node; a release memory request is put into a management queue; wherein the release memory request carries the NIC memory address to be released; The corresponding NIC memory is released, including: The release memory request is read from the management queue, and the corresponding NIC memory is released according to the NIC memory address to be released carried in the release memory request.

10. The data processing method based on a distributed storage system according to claim 8, characterized in that, Further comprising: In the case of obtaining the read request notification, the target NIC memory is allocated according to the data amount and the number of erasure blocks corresponding to the read request notification, and a write response notification is fed back to the storage server of the master storage node; wherein the write response notification corresponds to the target NIC memory address of the target NIC memory; Receiving the erasure blocks written by each slave storage node to the target NIC memory; In the case of receiving the erasure operation notification issued by the storage server of the master storage node, performing erasure processing on all erasure blocks in the target NIC memory to obtain reorganized data; and writing the reorganized data into the host memory.

11. The data processing method based on a distributed storage system according to claim 10, characterized in that, Allocating the target NIC memory according to the data amount and the number of erasure blocks corresponding to the read request notification, and feeding back a write response notification to the storage server of the master storage node, includes: The data read request is read from the request queue, and the target NIC memory is allocated according to the data amount and the number of erasure blocks carried in the data read request; The target NIC memory address of the target NIC memory is put into a response queue, and a read response notification is issued to the storage server of the master storage node, so that the storage server of the master storage node reads the target NIC memory address from the response queue in the case of receiving the read response notification, and deletes the data read request recorded in the request queue.

12. A data processing apparatus based on a distributed storage system, characterized by, Including: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the data processing method based on the distributed storage system according to any one of claims 8 to 11.

13. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the data processing method based on the distributed storage system according to any one of claims 8 to 11.

Images