Distributed storage system data processing method, program product, device and medium

Abstract

The present invention discloses a distributed storage system data processing method, program product, device and medium, which relates to the technical field of distributed storage systems, and is used to realize data processing of distributed storage systems. Aiming at the performance bottleneck caused by the upper limit of network card performance in the current system, a distributed storage system data processing method is provided, which realizes the erasure code calculation and distribution calculation process of business data from the main node to the client by pre-deploying the erasure code calculation module in the client and subscribing to the data distribution map update in the storage pool. The client directly sends the corresponding transmission subtask to the storage node to omit the process in which the client sends business data to the main node, which can save the network card from one data transmission of business data, thereby reducing the burden on the network card and alleviating the performance bottleneck of the distributed storage system caused by the upper limit of the network card.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage systems, and in particular to a distributed storage system data processing method, program product, device and medium. Background Art

[0002] At present, in the actual application of distributed storage systems, in order to ensure the reliability of data, redundancy strategies are often adopted. Commonly used redundancy technologies include erasure codes and replicas. The use of redundancy strategies will cause the backend storage system to forward and write data multiple times when the customer writes data once to ensure redundant storage of data. Data forwarding and writing in the storage system rely on the network card, so the redundancy strategy will undoubtedly increase the burden of the network card exponentially. This burden was not obvious in the previous systems that mostly used mechanical hard disks (HDDs) as the underlying storage devices. The underlying storage devices with low bandwidth are still the performance bottleneck of the storage system.

[0003] However, with the continuous development of storage technology, the cost of solid-state storage devices has continued to decrease, and more and more storage manufacturers have begun to use solid-state drives as underlying storage devices. In particular, the use of protocols such as the Non-Volatile Memory Host Controller Interface Specification (NVMExpress, NVMe) has greatly improved the bandwidth and number of operations per second (ops) of underlying devices and reduced latency consumption. Currently, the bandwidth of ordinary solid-state drives can reach several hundred MB / s, while devices using NVMe and the high-speed serial computer expansion bus standard (PCI-Express, PCI-e) 4.0 interface can reach 8GB / s. This shows that the performance bottleneck of the storage system has gradually shifted from the underlying hard disk to the network layer.

[0004] Therefore, technicians in this field are in urgent need of a distributed storage system data processing method to solve the system performance bottleneck problem caused by the upper limit of network card performance in the current distributed storage system. Summary of the invention

[0005] The purpose of the present invention is to provide a distributed storage system data processing method, program product, device and medium, which are used to solve the system performance bottleneck problem caused by the upper limit of network card performance in the current distributed storage system.

[0006] In order to solve the above technical problems, the present invention provides a distributed storage system data processing method, which is applied to the client side of the distributed storage system; wherein the client is pre-deployed with an erasure code calculation module, and the erasure code calculation module is a module integrating an erasure code algorithm and an interface used by the erasure code algorithm; the client is pre-loaded with a library file required for the erasure code algorithm during initialization; the client also subscribes to data distribution mapping updates in the storage pool;

[0007] Methods include:

[0008] Receive and analyze the data transmission request sent by the client node to obtain the data transmission task;

[0009] Calling the erasure code calculation module to process the data transmission task to obtain a plurality of transmission subtasks;

[0010] Update the data distribution map based on subscriptions and obtain the storage service information corresponding to each storage node;

[0011] Performing distributed computing on the data transmission task according to the storage service information corresponding to each of the storage nodes to obtain transmission subtasks corresponding to different storage nodes;

[0012] Sending each of the transmission subtasks to the corresponding storage node so that the storage node completes the corresponding transmission subtask;

[0013] After each of the storage nodes completes the corresponding transmission subtask, it returns data transmission success information to the client node.

[0014] In a possible embodiment, the data transmission request is a data writing request;

[0015] Then the data transmission task is: writing the data to be written into the storage cluster;

[0016] The transmission subtask is: writing a data block or a check block into the corresponding storage node; wherein the data block and the check block are obtained by the erasure code calculation module according to the data to be written.

[0017] In a possible embodiment, calling the erasure code calculation module to process the data transmission task to obtain a plurality of transmission subtasks includes:

[0018] Dividing the data to be written into a plurality of data blocks of the same size;

[0019] Calling the erasure code calculation module to perform erasure code calculation on the data block to generate the check block;

[0020] Performing distributed computing on the data transmission task according to the storage service information corresponding to each storage node to obtain transmission subtasks corresponding to different storage nodes includes:

[0021] Determine the storage node corresponding to each of the data blocks and the check blocks according to the storage service information corresponding to each of the storage nodes and through a distributed algorithm;

[0022] Sending each of the transmission subtasks to the corresponding storage node includes:

[0023] According to the storage service information corresponding to each storage node, the write request of each data block and the check block is sent to the storage service corresponding to the storage node.

[0024] In a possible embodiment, the storage node includes: a master node and a slave node;

[0025] After each storage node completes the corresponding transmission subtask, returning data transmission success information to the client node includes:

[0026] After receiving the write success information returned by the master node in the storage node, returning the data write success information to the client node;

[0027] After receiving the write success information returned by each slave node, the master node returns the write success information to the client;

[0028] After completing the writing of the corresponding data block or the check block, each slave node returns a write success message to the master node.

[0029] In a possible embodiment, it further includes:

[0030] When any of the storage nodes fails to write the corresponding data block or the check block, obtaining a version log of each of the storage nodes;

[0031] Selecting a version log from the version logs of each storage node as an authoritative log;

[0032] Compare the differences between the version log of each storage node and the authoritative log, and determine the data rollback task corresponding to each storage node;

[0033] The data rollback task is sent to each corresponding storage node, so that each storage node rolls back the data to the version corresponding to the authoritative log.

[0034] In a possible embodiment, selecting a version log as an authoritative log from the version logs of the storage nodes includes:

[0035] Parsing the version log of each storage node to obtain the version number;

[0036] The obtained version numbers are compared, and the version log with the largest number of identical version numbers is selected as the authoritative log.

[0037] In a possible embodiment, the data transmission request is a data reading request;

[0038] Then the data transmission task is: reading target data from the storage cluster;

[0039] The transmission subtask is: reading a corresponding data block or a check block from the storage node; wherein the data block and the check block are obtained by the erasure code calculation module according to the target data.

[0040] In a possible embodiment, calling the erasure code calculation module to process the data transmission task to obtain a plurality of transmission subtasks includes:

[0041] Calling the erasure code calculation module to perform erasure code calculation on the target data to determine the corresponding data block and the check block;

[0042] Performing distributed computing on the data transmission task according to the storage service information corresponding to each storage node to obtain transmission subtasks corresponding to different storage nodes includes:

[0043] Determine the storage node corresponding to each data block and each check block according to the storage service information corresponding to each storage node and through a distributed algorithm;

[0044] Sending each of the transmission subtasks to the corresponding storage node includes:

[0045] According to the storage service information corresponding to each storage node, the read request of each data block and each check block is sent to the storage service corresponding to the storage node.

[0046] In a possible embodiment, after each storage node completes the corresponding transmission subtask, returning data transmission success information to the client node includes:

[0047] Decoding and verifying each of the data blocks and the check blocks to obtain the target data;

[0048] The data read success information including the target data is sent to the client node.

[0049] In a possible embodiment, before sending the read request of each data block and each check block to the storage service corresponding to the storage node according to the storage service information corresponding to each storage node, the method further includes:

[0050] Applying for memory space corresponding to the length of the target data;

[0051] After sending the read requests of the data blocks and the check blocks to the storage service corresponding to the storage node according to the storage service information corresponding to each storage node, the method further includes:

[0052] Whenever the data block or the check block returned by the storage node is received, the data block or the check block is stored in the memory space, and corresponding metadata is maintained.

[0053] In a possible embodiment, it further includes:

[0054] If any of the storage nodes fails to respond to the corresponding read request for more than a preset period of time, or any of the storage nodes returns a read error message, the memory space and metadata are cleaned up, and the process returns to the step of sending the read request for each of the data blocks and each of the check blocks to the storage service corresponding to the storage node according to the storage service information corresponding to each of the storage nodes.

[0055] In a possible embodiment, sending each of the transmission subtasks to the corresponding storage node so that the storage node completes the corresponding transmission subtask includes:

[0056] Each transmission subtask is sent to the corresponding storage node based on the transmission control protocol, so that when the storage node receives the corresponding transmission subtask, it independently adds the business corresponding to the transmission subtask to the business queue inside the storage service.

[0057] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the distributed storage system data processing method as described above when executed by a processor.

[0058] In order to solve the above technical problems, the present invention further provides a distributed storage system data processing device, comprising:

[0059] Memory for storing computer programs;

[0060] The processor is used to implement the steps of the distributed storage system data processing method as described above when executing the computer program.

[0061] To solve the above technical problem, the present invention further provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the distributed storage system data processing method as described above are implemented.

[0062] The present invention provides a distributed storage system data processing method, which pre-deploys an erasure code calculation module integrated with an erasure code algorithm and an interface required for implementing the erasure code algorithm in a client located in the distributed storage system protocol layer, and loads the library file required for the erasure code algorithm during the initialization process of the client, so that the client can support the erasure code calculation function. In addition, the data distribution map (map) update is subscribed in the client, so that every time the data distribution changes in the storage cluster, it is not only notified to the storage service, but also to the client synchronously, and the client can synchronously grasp the storage service corresponding to the detailed data distribution, so that the client can directly implement data distribution calculation.

[0063] Based on the above settings, in this method, the erasure code calculation and distributed calculation originally implemented in the master node in the storage cluster are pulled up to the client in the distributed storage system. Then, when the client receives the data transmission task sent by the client node, it can directly perform the erasure code calculation and data distribution calculation locally; when the client node needs to write data, the k+m-1 blocks of data after the erasure code calculation are directly sent to the corresponding k+m-1 storage nodes for storage, and no longer forwarded through the master node; when the client node needs to read data, the same is true. The client directly reads the corresponding data block from the corresponding storage node based on the supported erasure code calculation and distributed computing functions, and then returns to the client node, without the need for forwarding by the master node. That is, based on the data processing process implemented by this method, the client's data read and write requests are no longer forwarded to each child node through the master node, but the client directly interacts with the child node. In this way, the consumption of the master node network card forwarding is avoided, thereby reducing the burden on the network card, that is, reducing the performance bottleneck caused by the network card.

[0064] The computer program product, distributed storage system data processing device, and non-volatile storage medium provided by the present invention correspond to the above method and have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1An application architecture diagram of a distributed storage system data processing method provided by the present invention;

[0067] Figure 2 A flow chart of a distributed storage system data processing method provided by the present invention;

[0068] Figure 3 A response flow chart of a data writing scenario provided by the present invention;

[0069] Figure 4 A response flow chart of a data reading scenario provided by the present invention;

[0070] Figure 5 A structural diagram of a distributed storage system data processing device provided by the present invention;

[0071] Figure 6 A structural diagram of another distributed storage system data processing device provided by the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] The core of the present invention is to provide a distributed storage system data processing method, program product, device and medium.

[0074] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0075] At present, in the application of storage technologies such as distributed storage systems, bandwidth-based services often use erasure code redundancy solutions to save hard disk space. In a possible erasure code scenario, assume that the storage system uses a k+m erasure code algorithm (that is, divide the data into k data blocks, and generate m check blocks based on these k data blocks).

[0076] At this time, if the customer's business node (hereinafter referred to as the customer node) sends a data read and write service to the client (assuming that the data volume of this service is S); the traditional solution is that the client determines the master node and master storage device to which the service data belongs through a distributed algorithm such as a hash algorithm; and sends the service data to the corresponding master node through the network; at this time, the network card undertakes the data transmission task of data volume S.

[0077] Afterwards, after the master node receives the entire block of business data through network transmission, it calculates k+m blocks of data corresponding to the business data through the internal distributed algorithm and erasure code algorithm; among them, 1 block of data belongs to the master node itself, and the remaining k+m-1 blocks of data belong to the other k+m-1 child nodes and their corresponding storage devices; based on this, the master node sends the k+m-1 blocks of data to the corresponding child nodes through the network card; at this time, the network card undertakes the data transmission task of (k+m-1) / k*S.

[0078] The architecture of the above business process is as follows Figure 1 As shown in the figure, when data redundancy is implemented based on erasure coding technology, for a service with a data volume of S, the network card needs to forward data with a data volume of (k+m-1) / k*S+S. As a result, with the evolution of storage technology and the reduction of the cost of solid-state storage devices, network cards have gradually become a shackle that restricts the performance of distributed storage systems.

[0079] To solve the above problems, the present invention provides a distributed storage system data processing method, which is applied to the client side of the distributed storage system. The client is pre-deployed with an erasure code calculation module, which is a module that integrates an erasure code algorithm and an interface used by the erasure code algorithm; the client is pre-loaded with a library file required for the erasure code algorithm during initialization; and the client also subscribes to data distribution mapping updates in the storage pool.

[0080] If Figure 2 As shown, the method includes:

[0081] S11: Receive and parse the data transmission request sent by the client node to obtain the data transmission task.

[0082] S12: Calling the erasure code calculation module to process the data transmission task to obtain multiple transmission subtasks.

[0083] S13: Update the data distribution map based on the subscription and obtain the storage service information corresponding to each storage node.

[0084] S14: Distributed computing is performed on the data transmission task according to the storage service information corresponding to each storage node to obtain transmission subtasks corresponding to different storage nodes.

[0085] S15: Send each transmission subtask to a corresponding storage node, so that the storage node can complete the corresponding transmission subtask.

[0086] S16: After each storage node completes the corresponding transmission subtask, it returns data transmission success information to the client node.

[0087] Before describing the specific process of this method, the principle of this method is described first:

[0088] As can be seen from the above, in the business processing flow of the traditional distributed storage system, since the client node is deployed on the node in the distributed storage system, the data sent to the client in the distributed storage system does not belong to the network transmission in the distributed storage system, that is, it will not become a network shackle inside the distributed storage system. Therefore, in a business process, there are mainly two data transmissions that require the use of the network card in the distributed storage system: one is that after the client receives the business data with a data volume of S, it sends the business data to the master node, and the network card transmission data volume of this process is S; the second is that after the master node receives the business data with a data volume of S, it calculates the erasure code and distributes the k+m-1 blocks of data that need to be distributed to other nodes (one of the k+m blocks of data is processed by the master node itself), and the data volume of each block of data is S / k. The master node needs to send the k+m-1 blocks of data to the corresponding k+m-1 child nodes, and the network card transmission data volume of this process is (k+m-1) / k*S.

[0089] It can be seen that in the traditional storage system data processing business that adopts the erasure code redundancy technology, for a business data with a data volume S, the network card needs to transmit data with a data volume of (k+m-1) / k*S, which is obviously larger than the data volume S of the business itself, resulting in a waste of resources and room for optimization.

[0090] Based on this, the method provided by the present invention provides a solution: that is, to implement erasure code calculation and distributed calculation of business data on the client side, so that the client can directly obtain k+m-block data corresponding to the business data and k+m storage nodes corresponding to these data. Afterwards, the client directly interacts with the corresponding k+m storage nodes for the k+m block data, avoiding the re-forwarding of the master node, thereby eliminating the network transmission process with a data volume of S between the client and the master node (but there is an additional process of sending data size S / k to the master node, that is, the final reduced data transmission volume is SS / k), achieving the purpose of reducing the burden on the network card, thereby alleviating the performance bottleneck problem of the distributed storage system caused by the upper limit of the network card.

[0091] However, for the erasure code calculation and distributed calculation of business data, the traditional scheme is implemented by the master node, and the client does not have the function of implementing erasure code and distributed calculation. Based on this, a distributed storage system data processing method provided by the present invention pre-deploys an erasure code calculation module in the client. Among them, the erasure code calculation module is a module that integrates the erasure code algorithm and the interface used by the erasure code algorithm. And the client is pre-loaded with the library files required for the erasure code algorithm during initialization, so that the client can support the erasure code calculation function. In addition, the client in this method also subscribes to the data distribution map (map) update in the storage pool; whenever the data distribution changes, not only the storage service will be notified, but also the client that has subscribed to the data distribution map update will be synchronously notified; the client can obtain the storage service corresponding to the detailed data distribution by parsing the information in the map, combined with the distributed algorithm that the client can already implement, so that the client can support the distributed calculation of business data.

[0092] Based on the above, the service request sent by the client node can be processed through the process of steps S11 to S16.

[0093] Among them, corresponding to step S12, that is, the process of calculating the complete business data based on the erasure code to obtain k+m blocks of data (k data blocks and m check blocks). Corresponding to step S14, that is, the process of corresponding the k+m blocks of data obtained after the above erasure code calculation to k+m storage nodes. It should be particularly noted that since the data distribution calculation in this method does not need to be implemented by the master node, and the client itself cannot be used for data storage. Therefore, it is necessary to distribute all k+m blocks of data to k+m storage nodes, and these k+m storage nodes include the master node.

[0094] It can be seen that the distributed storage system data processing method provided by the present invention pre-deploys the erasure code calculation module on the client and loads the library files required for implementing the erasure code calculation in the initialization phase, so that the client can support the erasure code calculation function. Furthermore, the client also subscribes to the data distribution map update in the storage pool, so that the client can synchronously know the changes in data distribution, and combined with the originally achievable distributed algorithm, the distributed calculation function of business data can be realized.

[0095] Based on the above settings, this method pulls the erasure code calculation and distribution calculation process of business data from the master node to the client. The client directly sends the corresponding transmission subtask to k+m storage nodes to omit the process of the client sending business data to the master node, which can save the network card from transmitting the business data once, thereby reducing the burden on the network card and alleviating the performance bottleneck of the distributed storage system caused by the upper limit of the network card.

[0096] On the other hand, in the above method, there is no restriction on the type of data transmission request sent by the client node. Common data transmission requests mainly include data write request and data read request. And these two requests have the problem that the network card needs to forward much more data than the original service based on the erasure code technology. Therefore, in the subsequent embodiments of the present invention, these two services are mainly used for explanation.

[0097] First, for the data writing scenario, this embodiment provides a corresponding implementation scheme:

[0098] The data transmission request in the above step S11 is a data write request; the data transmission task is: writing the data to be written into the storage cluster; the transmission subtask is: writing the data block or the check block into the corresponding storage node.

[0099] The data block and the check block are obtained by the erasure code calculation module according to the data to be written.

[0100] It is easy to know that after the client can support the erasure code calculation function, the erasure code calculation for the data to be written (i.e., the business data in the data writing scenario) implemented by the client is no different in substance from the common erasure code calculation implemented by the master node, and step S12 in this method can be implemented based on the existing erasure code calculation scheme.

[0101] For step S12, this embodiment also provides a possible implementation scheme, and step S12 specifically includes:

[0102] S121-A: Divide the data to be written into multiple data blocks of the same size.

[0103] S122-A: Calling the erasure code calculation module to perform erasure code calculation on the data block to generate a check block.

[0104] Correspondingly, step S14 is:

[0105] S14-A: Determine the storage node corresponding to each data block and check block according to the storage service information corresponding to each storage node and through a distributed algorithm;

[0106] Correspondingly, step S15 is:

[0107] S15-A: According to the storage service information corresponding to each storage node, the write request of each data block and check block is sent to the storage service of the corresponding storage node.

[0108] Taking the above-mentioned k+m erasure code algorithm as an example, assume that the amount of data to be written is S; then in step S121-A, the data to be written with the amount of S is divided into k data blocks of equal size; and then in step S122-A, based on the k data blocks, the corresponding m check blocks are obtained by calling the erasure code calculation module.

[0109] Furthermore, since the distributed storage system data processing method provided in the embodiment of the present invention changes the original process of the master node sending data processing to each child node, and changes it to a method in which the client directly interacts with each storage node. Therefore, for the operation response problem in the data writing scenario, this embodiment also provides a possible implementation plan:

[0110] Storage nodes include: master nodes and slave nodes;

[0111] Then step S16 includes:

[0112] S16-A: After receiving the write success information returned by the master node in the storage node, the data write success information is returned to the client node.

[0113] After receiving the write success information returned by each slave node, the master node returns the write success information to the client; after completing the writing of the corresponding data block or check block, each slave node returns the write success information to the master node.

[0114] That is, for data writing scenarios, since the write success message does not contain business data, it will not cause additional burden on the network card; therefore, the original master node can be used to receive the response information of all other child nodes and then return the write success message to the client; after receiving the write success message sent by the master node, the client returns the write success message to the client node. The response process is as follows Figure 3 As shown, it can be implemented based on the original response solution in the distributed storage system without additional deployment.

[0115] Furthermore, this embodiment provides a possible implementation scheme based on the above embodiment, and the above method further includes:

[0116] S21: When a storage node fails to write a corresponding data block or a check block, the version log of each storage node is obtained.

[0117] S22: Select one version log from the version logs of each storage node as the authoritative log.

[0118] S23: Compare the differences between the version log of each storage node and the authoritative log, and determine the data rollback task corresponding to each storage node.

[0119] S24: Send the data rollback task to each corresponding storage node, so that each storage node rolls back the data to the version corresponding to the authoritative log.

[0120] That is, this embodiment provides a data rollback solution for handling write failure or failure scenarios. In the actual application of distributed storage systems, write failures may occur due to hard disk failures or service downtime. To address this problem, this embodiment obtains the version logs (logs) of each storage node, selects one as the authoritative log, and the data of other storage nodes are all based on the authoritative log for data rollback.

[0121] It is easy to understand that the erasure coding technology itself is a solution for achieving data redundancy. When data is lost, as long as the lost data blocks do not exceed m, all the original business data blocks can be restored through the remaining data blocks and the check blocks. Considering that the erasure coding technology and its data recovery solution are well known to those skilled in the art, this embodiment will not be described in detail here.

[0122] Furthermore, this embodiment also provides a solution for selecting an authoritative log, and step S22 specifically includes:

[0123] S221: Parse the version log of each storage node to obtain the version number.

[0124] S222: Compare the obtained version numbers and select the version log with the largest number of identical version numbers as the authoritative log.

[0125] It should be noted that this embodiment uses the version log with the largest number of identical version numbers as the authoritative log. Since data rollback only requires data rollback for storage nodes where the version log is different from the authoritative log, this embodiment can roll back data on the least number of nodes when performing data rollback, thereby bringing faster data rollback efficiency.

[0126] It is easy to know that the authoritative log selection method provided in this embodiment is only a possible implementation plan, which is a plan that prioritizes data rollback efficiency. In addition, the authoritative log can also select a version log with a higher priority, or directly select the version log of the master node as the authoritative log. In actual applications, the appropriate authoritative log selection strategy can be freely selected according to actual needs, and this embodiment does not limit this.

[0127] Exemplarily, this embodiment also provides another possible solution for selecting an authoritative log, and the method further includes:

[0128] When a storage node is created, a priority is assigned to each storage node.

[0129] Accordingly, step S22 specifically includes:

[0130] According to the priority of each storage node, the version log corresponding to the storage node with the highest priority is selected as the authoritative log.

[0131] This embodiment provides a priority-based custom authoritative log selection solution, that is, when a distributed storage cluster is deployed, each storage node is prioritized in advance according to certain needs. Then, when data is rolled back, the storage node with the highest priority is used as the data version based on which the data is rolled back, that is, the version log corresponding to the storage node is selected as the authoritative log, thereby realizing the selection of a custom authoritative log.

[0132] On the other hand, for the data reading scenario, this embodiment also provides a corresponding implementation scheme:

[0133] The data transmission request in the above step S11 is a data read request; the data transmission task is: reading target data from the storage cluster; the transmission subtask is: reading corresponding data blocks or check blocks from the storage node.

[0134] The data block and the check block are obtained by the erasure code calculation module according to the target data.

[0135] It is easy to understand that the target data is the business data in the data reading scenario. Each storage node needs to read the corresponding data block from the corresponding storage device and send it to the client. The client then decodes and verifies it to obtain the final target data and returns it to the client node.

[0136] Similar to the embodiment corresponding to the above-mentioned data writing scenario, after the client can support the erasure code calculation function, there is no substantial difference between the erasure code calculation of the target data implemented by the client and the common erasure code calculation implemented by the master node, and step S12 in this method can be implemented based on the existing erasure code calculation scheme.

[0137] For step S12, this embodiment also provides a possible implementation scheme, and step S12 specifically includes:

[0138] S12-B: Calling the erasure code calculation module to perform erasure code calculation on the target data to determine the corresponding data block and check block.

[0139] Correspondingly, step S14 is:

[0140] S14-B: Determine the storage node corresponding to each data block and each check block according to the storage service information corresponding to each storage node and through a distributed algorithm.

[0141] Correspondingly, step S15 is:

[0142] S15-B: According to the storage service information corresponding to each storage node, the read request of each data block and each check block is sent to the storage service of the corresponding storage node.

[0143] For this embodiment, it is easy to know that in the data encoding process of the erasure code, each data block and check block is encoded (such as calculating a hash value) by a distributed algorithm such as a hash algorithm, and then the storage location of the data block is characterized according to the encoding. For example, data positioning is achieved by using a hash table, and the hash values ​​of all data blocks and check blocks are stored in the hash table. The hash value can be used to locate the position of the data block or check block, that is, the corresponding storage node. That is, this embodiment is also the reverse process of the encoding process, and the storage location corresponding to each data block of the target data is determined by decoding, and then the required data is read from the corresponding storage node.

[0144] It should be noted that, in this embodiment, the client directly sends the corresponding data read request to each storage node without passing through the master node. Accordingly, this embodiment also provides a data read response solution, and the above step S16 is specifically as follows:

[0145] S161-B: Decode and verify each data block and check block to obtain target data.

[0146] S162-B: Send the data read success information including the target data to the client node.

[0147] That is, in this embodiment, each storage node also directly returns the data reading result to the client without passing through the master node. It should be noted that this embodiment is different from the above-mentioned data writing scenario. It changes the original method in which the sub-nodes first uniformly return the data reading results to the master node, and the master node returns the data reading results to the client after receiving the data reading results of all the sub-nodes. This is because the data reading results contain business data, that is, the target data that needs to be read. If the original data response scheme is adopted, taking the above-mentioned business data example with a data volume of S as an example, there is still a data transmission with a data volume of (k+m-1) / k*S between the sub-node and the master node, and a data transmission with a data volume of S between the master node and the client, which does not achieve the purpose of reducing the amount of network transmission data provided by the method of the present invention. Based on this, in this embodiment, for the data reading scenario, each storage node directly sends the corresponding data to the client after reading it, and the client performs subsequent erasure code decoding and verification process to obtain the final target data to return to the client node, completing a data reading service.

[0148] Based on the above embodiments, it can be known that the data processing method for a distributed storage system provided by the present invention reduces the amount of data transmitted over the network in different processes in data writing scenarios and data reading scenarios. Specifically, in the data writing scenario, the reduction in the amount of data transmitted over the network is mainly reflected in the request sending process; that is, the data writing request carries the business data, and is sent directly to each storage node through the client, bypassing the main node, thereby achieving the purpose of reducing the amount of data transmitted over the network. As for the data reading scenario, the reduction in the amount of data transmitted over the network is mainly reflected in the request response process; that is, the response result of the data writing and reading request carries the business data, and the data reading result is returned directly to the client through each storage node, bypassing the main node, thereby achieving the purpose of reducing the amount of data transmitted over the network.

[0149] Furthermore, this embodiment also provides a possible implementation scheme for how to directly obtain the data read by each storage node through the client and obtain the target data that is ultimately used to be returned to the client node. Before step S15-B, the method further includes:

[0150] S31: Apply for memory space corresponding to the length of the target data.

[0151] After step S15-B, the method further includes:

[0152] S32: Whenever a data block or a check block is received from a storage node, the data block or the check block is stored in the memory space, and the corresponding metadata is maintained.

[0153] That is, for the client in the original distributed storage system that is not responsible for receiving the data reading results returned by each storage node and obtaining the target data, this embodiment provides a possible implementation plan: before issuing a data reading request, the client first applies for a memory space corresponding to the data length of the target data in the memory to store the target data and complete the preparation for data reading.

[0154] Furthermore, after the data read request is issued and each storage node returns the data read result one after another, the client can store the data blocks or check blocks returned by each storage node in the memory space and maintain their corresponding metadata until all storage nodes have finished reading the data. The original business data, that is, the target data, can be restored based on the acquired data blocks and check blocks and returned to the client node to complete the data reading business.

[0155] Furthermore, in addition to abnormal problems such as write failure that may occur during the data writing process, read failure or no response may also occur during the data reading process. In view of this scenario, this embodiment provides a possible implementation scheme, and the above method also includes:

[0156] S33: If there is a storage node that fails to respond to the corresponding read request for more than a preset period of time, or if there is a storage node that returns a read error message, then the memory space and metadata are cleared, and the process returns to step S15-B.

[0157] That is, in this embodiment, a solution for resending a data read request is provided to solve the problem of data read failure or non-response due to occasional problems, and further improve the reliability of the data processing method.

[0158] On the other hand, since the distributed storage system data processing method provided in the present invention utilizes erasure codes and distributed computing originally implemented by the master node to be pulled up to the client for execution, and then directly interacts with each storage node (mainly child nodes) through the client to achieve the purpose of reducing the amount of network transmission data. Based on this, in order to ensure the normal processing of data services, higher requirements are placed on the service processing order between each storage node.

[0159] To solve this problem, this embodiment provides a corresponding implementation scheme, and the above step S15 specifically includes:

[0160] Each transmission subtask is sent to a corresponding storage node based on the transmission control protocol, so that when the storage node receives the corresponding transmission subtask, it independently adds the service corresponding to the transmission subtask to the service queue inside the storage service.

[0161] That is, in this embodiment, the sequence of messages sent by the client (i.e., the above-mentioned transmission subtasks) to each storage node is guaranteed through the mechanism of the Transmission Control Protocol (TCP) itself. When each storage node receives the message, it independently adds the service to the service queue inside the storage service. Since the TCP protocol guarantees the sequence of messages received by each storage node, each storage node independently reports the service when receiving the message. Through the sequential dequeue and locking mechanism of the internal service queue, it can be ensured that the storage service executes the service in sequence and returns the result. Furthermore, it can be ensured that the response result received by the client is consistent with the order when the service is issued, thereby achieving the effect of reducing the amount of network transmission data described in the above embodiment while ensuring the reliability of service processing.

[0162] In the above embodiment, a distributed storage system data processing method is described in detail, and the present invention also provides a corresponding embodiment of a distributed storage system data processing device. It should be noted that the present invention describes the embodiment of the device part from two perspectives, one is based on the perspective of functional modules, and the other is based on the perspective of hardware.

[0163] From the perspective of functional modules, this embodiment provides a distributed storage system data processing device, which is also applied to the client side of the distributed storage system; wherein the client is pre-deployed with an erasure code calculation module, which is a module integrating an erasure code algorithm and an interface used by the erasure code algorithm; the client is pre-loaded with a library file required for the erasure code algorithm during initialization; the client also subscribes to data distribution mapping updates in the storage pool.

[0164] Among them, the device Figure 5 As shown, including:

[0165] The service receiving module 11 is used to receive and analyze the data transmission request sent by the client node to obtain the data transmission task.

[0166] The erasure code calculation module 12 is used to call the erasure code calculation module to process the data transmission task to obtain multiple transmission subtasks;

[0167] A distribution acquisition module 13 is used to update the subscribed data distribution map and acquire storage service information corresponding to each storage node;

[0168] The distributed computing module 14 is used to perform distributed computing on the data transmission task according to the storage service information corresponding to each storage node, so as to obtain transmission subtasks corresponding to different storage nodes;

[0169] The service delivery module 15 is used to send each transmission subtask to the corresponding storage node so that the storage node can complete the corresponding transmission subtask;

[0170] The service response module 16 is used to return data transmission success information to the client node after each storage node completes the corresponding transmission subtask.

[0171] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0172] Figure 6 A structural diagram of a distributed storage system data processing device provided by another embodiment of the present invention, such as Figure 6 As shown, a distributed storage system data processing device includes: a memory 20 for storing computer programs;

[0173] The processor 21 is used to implement the steps of a distributed storage system data processing method as described in the above embodiment when executing a computer program.

[0174] The distributed storage system data processing device provided in this embodiment may include but is not limited to a mobile terminal, a personal computer, a workstation, etc.

[0175] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0176] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of a distributed storage system data processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include, but is not limited to, a distributed storage system data processing method, etc.

[0177] In some embodiments, a distributed storage system data processing device may also include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0178] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on a distributed storage system data processing device, and may include more or fewer components than those shown in the figure.

[0179] An embodiment of the present invention provides a distributed storage system data processing device, including a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a distributed storage system data processing method.

[0180] In addition to the embodiments of a distributed storage system data processing method and device provided in the above embodiments, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the distributed storage system data processing method described in any of the above embodiments can be implemented.

[0181] Since the embodiments of the computer program product part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the computer program product part, which will not be repeated here.

[0182] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0183] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0184] The above is a detailed introduction to a distributed storage system data processing method, program product, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0185] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A distributed storage system data processing method, characterized in that: Applied to the client side of a distributed storage system; wherein the client is pre-deployed with an erasure code calculation module, which is a module integrating an erasure code algorithm and an interface used by the erasure code algorithm; the client is pre-loaded with a library file required for the erasure code algorithm during initialization; the client also subscribes to data distribution mapping updates in a storage pool; Methods include: Receive and analyze the data transmission request sent by the client node to obtain the data transmission task; Calling the erasure code calculation module to process the data transmission task to obtain a plurality of transmission subtasks; Update the data distribution map based on subscriptions and obtain the storage service information corresponding to each storage node; Performing distributed computing on the data transmission task according to the storage service information corresponding to each of the storage nodes to obtain transmission subtasks corresponding to different storage nodes; Sending each of the transmission subtasks to the corresponding storage node so that the storage node completes the corresponding transmission subtask; After each of the storage nodes completes the corresponding transmission subtask, returning data transmission success information to the client node; The sending of each transmission subtask to the corresponding storage node so that the storage node completes the corresponding transmission subtask includes: Each transmission subtask is sent to the corresponding storage node based on the transmission control protocol, so that when the storage node receives the corresponding transmission subtask, it independently adds the business corresponding to the transmission subtask to the business queue inside the storage service.

2. The distributed storage system data processing method according to claim 1, characterized in that: The data transmission request is a data writing request; Then the data transmission task is: writing the data to be written into the storage cluster; The transmission subtask is: writing a data block or a check block into the corresponding storage node; wherein the data block and the check block are obtained by the erasure code calculation module according to the data to be written.

3. The distributed storage system data processing method according to claim 2, characterized in that: Calling the erasure code calculation module to process the data transmission task to obtain a plurality of transmission subtasks includes: Dividing the data to be written into a plurality of data blocks of the same size; Calling the erasure code calculation module to perform erasure code calculation on the data block to generate the check block; Performing distributed computing on the data transmission task according to the storage service information corresponding to each storage node to obtain transmission subtasks corresponding to different storage nodes includes: Determine the storage node corresponding to each of the data blocks and the check blocks according to the storage service information corresponding to each of the storage nodes and through a distributed algorithm; Sending each of the transmission subtasks to the corresponding storage node includes: According to the storage service information corresponding to each storage node, the write request of each data block and the check block is sent to the storage service corresponding to the storage node.

4. The distributed storage system data processing method according to claim 3, characterized in that: The storage nodes include: a master node and a slave node; After each storage node completes the corresponding transmission subtask, returning data transmission success information to the client node includes: After receiving the write success information returned by the master node in the storage node, returning the data write success information to the client node; After receiving the write success information returned by each slave node, the master node returns the write success information to the client; After completing the writing of the corresponding data block or the check block, each slave node returns a write success message to the master node.

5. The distributed storage system data processing method according to claim 4, characterized in that: Also includes: When any of the storage nodes fails to write the corresponding data block or the check block, obtaining a version log of each of the storage nodes; Selecting a version log from the version logs of each storage node as an authoritative log; Compare the differences between the version log of each storage node and the authoritative log, and determine the data rollback task corresponding to each storage node; The data rollback task is sent to each corresponding storage node, so that each storage node rolls back the data to the version corresponding to the authoritative log.

6. The distributed storage system data processing method according to claim 5, characterized in that: Selecting a version log from the version logs of the storage nodes as the authoritative log includes: Parsing the version log of each storage node to obtain the version number; The obtained version numbers are compared, and the version log with the largest number of identical version numbers is selected as the authoritative log.

7. The distributed storage system data processing method according to claim 1, characterized in that: The data transmission request is a data reading request; Then the data transmission task is: reading target data from the storage cluster; The transmission subtask is: reading a corresponding data block or a check block from the storage node; wherein the data block and the check block are obtained by the erasure code calculation module according to the target data.

8. The distributed storage system data processing method according to claim 7, characterized in that: Calling the erasure code calculation module to process the data transmission task to obtain a plurality of transmission subtasks includes: Calling the erasure code calculation module to perform erasure code calculation on the target data to determine the corresponding data block and the check block; Performing distributed computing on the data transmission task according to the storage service information corresponding to each storage node to obtain transmission subtasks corresponding to different storage nodes includes: Determine the storage node corresponding to each data block and each check block according to the storage service information corresponding to each storage node and through a distributed algorithm; Sending each of the transmission subtasks to the corresponding storage node includes: According to the storage service information corresponding to each storage node, the read request of each data block and each check block is sent to the storage service corresponding to the storage node.

9. The distributed storage system data processing method according to claim 8, characterized in that: After each storage node completes the corresponding transmission subtask, returning data transmission success information to the client node includes: Decoding and verifying each of the data blocks and the check blocks to obtain the target data; The data read success information including the target data is sent to the client node.

10. The distributed storage system data processing method according to claim 8, characterized in that: Before sending the read requests of the data blocks and the check blocks to the storage service corresponding to the storage node according to the storage service information corresponding to the storage node, the method further includes: Applying for memory space corresponding to the length of the target data; Then, after sending the read requests of each data block and each check block to the storage service corresponding to the storage node according to the storage service information corresponding to each storage node, the method further includes: Whenever the data block or the check block returned by the storage node is received, the data block or the check block is stored in the memory space, and corresponding metadata is maintained.

11. The distributed storage system data processing method according to claim 10, characterized in that: Also includes: If any of the storage nodes fails to respond to the corresponding read request for more than a preset period of time, or any of the storage nodes returns a read error message, the memory space and metadata are cleaned up, and the process returns to the step of sending the read request for each of the data blocks and each of the check blocks to the storage service corresponding to the storage node according to the storage service information corresponding to each of the storage nodes.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the distributed storage system data processing method as described in any one of claims 1 to 11 are implemented.

13. A distributed storage system data processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the distributed storage system data processing method as described in any one of claims 1 to 11 when executing the computer program.

14. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the distributed storage system data processing method according to any one of claims 1 to 11 are implemented.

Images