Data Processing Method, Device, Data Access End, and Storage Medium

Abstract

A data processing method, apparatus, data access end, and storage medium provided by an embodiment of the present invention relate to the field of distributed storage. First, a data write request is received, and the data write request includes data to be stored. Then, the data to be stored is processed to obtain a plurality of data blocks, and each data block is assigned a serial number. Next, for each data block, according to the serial number of the data block, a target node is determined from a plurality of storage nodes, and the data block and the data version number of the data to be stored are sent to the target node, so that the target node stores the data block and the data version number in the first space and the second space of the target node respectively, wherein the serial number of the data block and the number of the target node satisfy a preset mapping relationship, and the first space is located before the second space, avoiding the situation where the data version numbers in each storage node are the same while the data block storage fails, enabling the data access end to promptly recover the data blocks that failed to be stored, and ensuring data consistency.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage, and in particular, to a data processing method, apparatus, data access end, and storage medium. Background Art

[0002] In the era of big data, with the explosive growth of massive data, distributed storage is increasingly being applied. Distributed storage means that a data access end divides the data to be stored into multiple data stripes, encodes the multiple data stripes through an erasure code algorithm to obtain redundant check data, and then stores each data stripe and the check data on multiple storage nodes. When there is data that fails to be stored, the data that has been successfully stored can be used to recover the data that has failed to be stored.

[0003] In existing distributed storage technologies, a bitmap is generally used to record the storage locations of data stripes, and data stripe read and write operations are performed by finding the storage locations. The bitmap is usually recorded by metadata and stored on each storage node together with the data stripes. When any one of the storage nodes loses power abnormally, it is possible that the metadata is successfully stored at this node while the data stripe fails to be stored. At this time, since the data version numbers in the metadata between this node and other nodes are the same, the data access end will not recover the data stripes that have failed to be stored at this node, and the data consistency cannot be guaranteed. Summary of the Invention

[0004] To overcome the deficiencies of the prior art, embodiments of the present invention provide a data processing method, apparatus, data access end, and storage medium, which can avoid the situation where the data version numbers in each storage node are the same while there are data stripes that have failed to be stored, enabling the data access end to promptly recover the data stripes that have failed to be stored, thereby ensuring data consistency.

[0005] Embodiments of the present invention may be implemented as follows:

[0006] In a first aspect, the present invention provides a data processing method, which is applied to a data access end in a distributed storage system. The distributed storage system further includes multiple storage nodes, each of the storage nodes is provided with a number, and each of the storage nodes is communicatively connected to the data access end. The method includes:

[0007] Receiving a data write request, where the data write request includes data to be stored;

[0008] Processing the data to be stored to obtain multiple data blocks, and each of the data blocks is assigned a serial number;

[0009] For each of the data blocks, determine a target node from the multiple storage nodes according to the serial number of the data block, and send the data block and the data version number of the data to be stored to the target node, so that the target node stores the data block and the data version number in the first space and the second space of the target node respectively, where the serial number of the data block and the number of the target node satisfy a preset mapping relationship, and the first space is located before the second space.

[0010] In an alternative embodiment, before the step of processing the data to be stored to obtain multiple data blocks, the method further includes:

[0011] If the data write request is one, use the current timestamp as the data version number of the data to be stored;

[0012] If the data write requests are multiple, perform multiple increment operations on the current timestamp, and use the result of each increment operation as the data version number of the data to be stored in one of the data write requests according to the order of reception time of each data write request.

[0013] In an alternative embodiment, the step of processing the data to be stored to obtain multiple data blocks includes:

[0014] Slice the data to be stored into multiple data stripes according to a preset length;

[0015] Combine every preset number of the data stripes into an original data block to obtain multiple first data blocks;

[0016] Perform erasure coding on the multiple original data blocks to obtain multiple check data blocks, and the multiple data blocks include the multiple original data blocks and the multiple check data blocks.

[0017] In an alternative embodiment, the method further includes:

[0018] Receive a data read request, where the data read request includes the write node order of the data to be read;

[0019] Read the target data version number from the second space of each storage node according to the write node order;

[0020] If all the target data version numbers are the same, read the target data blocks from the first space of each storage node according to the write node order;

[0021] Generate the data to be read according to the preset mapping relationship and all the target data blocks to respond to the data read request.

[0022] In an alternative embodiment, the method further includes:

[0023] If there are inconsistent target data version numbers, divide the multiple storage nodes into normal nodes and abnormal nodes according to each target data version number, where the target data version numbers corresponding to all the normal nodes are the same, and the target data version number corresponding to each abnormal node is different from the target data version numbers corresponding to all the normal nodes;

[0024] Read target data blocks from the first space of each normal node according to the write node order;

[0025] Restore the target data blocks corresponding to each abnormal node according to the target data blocks corresponding to each normal node;

[0026] Generate the data to be read according to the preset mapping relationship and all the target data blocks to respond to the data read request.

[0027] In an alternative embodiment, the step of generating the data to be read according to the preset mapping relationship and all the target data blocks includes:

[0028] Assign serial numbers to each target data block according to the preset mapping relationship and the number of each storage node;

[0029] Sort all the target data blocks according to the serial numbers of each target data block to obtain the data to be read.

[0030] In an alternative embodiment, the method further includes:

[0031] Determine the target area in the first space of each abnormal node according to the write node order and a preset size;

[0032] For each abnormal node, use the target data block corresponding to the abnormal node to overwrite the content of the target area of the abnormal node.

[0033] In a second aspect, the present invention provides a data processing device, which is applied to a data access end in a distributed storage system. The distributed storage system further includes multiple storage nodes, each storage node is provided with a number, and each storage node is communicatively connected to the data access end. The method includes:

[0034] A receiving module, configured to receive a data write request, where the data write request includes data to be stored;

[0035] A processing module, configured to process the data to be stored to obtain multiple data blocks, and each data block is assigned a serial number;

[0036] A sending module, configured to, for each of the data blocks, determine a target node from the multiple storage nodes according to the serial number of the data block, and send the data block and the data version number of the data to be stored to the target node, so that the target node stores the data block and the data version number in a first space and a second space of the target node respectively, where the serial number of the data block and the number of the target node satisfy a preset mapping relationship, and the first space is located before the second space.

[0037] In a third aspect, the present invention provides a data access end, including a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the data processing method according to any one of the foregoing embodiments is implemented.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data processing method according to any one of the foregoing embodiments is implemented.

[0039] Compared with the prior art, for a data processing method, device, data access end and storage medium provided by an embodiment of the present invention, first, a data write request is received, and the data write request includes data to be stored; then, the data to be stored is processed to obtain multiple data blocks, and each data block is assigned a serial number; then, for each data block, a target node is determined from multiple storage nodes according to the serial number of the data block, and the data block and the data version number of the data to be stored are sent to the target node, so that the target node stores the data block and the data version number in a first space and a second space of the target node respectively, where the serial number of the data block and the number of the target node satisfy a preset mapping relationship, and the first space is located before the second space. Since the embodiment of the present invention sends each data block and the data version number of the data to be stored to the target node together, so that the target node stores the data block and the data version number in its first space and second space respectively, and the first space in the target node is located before the second space, thus avoiding the situation that the data version numbers in each storage node are the same while the data block storage fails, enabling the data access end to timely recover the data blocks with storage failures and ensuring data consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0041] Figure 1A structural schematic diagram of a distributed storage system provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a distributed storage process provided by an embodiment of the present invention;

[0043] Figure 3 A flowchart of a data processing method provided by an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of a data write request response process provided by an embodiment of the present invention;

[0045] Figure 5 A flowchart of an implementation manner of step S102 provided by an embodiment of the present invention;

[0046] Figure 6 Another flowchart of a data processing method provided by an embodiment of the present invention;

[0047] Figure 7 A schematic diagram of a data read request response process provided by an embodiment of the present invention;

[0048] Figure 8 Another schematic diagram of a data read request response process provided by an embodiment of the present invention;

[0049] Figure 9 An example of a data write request response process provided by an embodiment of the present invention;

[0050] Figure 10 An example of a data read request response process provided by an embodiment of the present invention;

[0051] Figure 11 Another example of a data read request response process provided by an embodiment of the present invention;

[0052] Figure 12 A structural schematic block diagram of a data access end provided by an embodiment of the present invention;

[0053] Figure 13 A functional unit block diagram of a data processing device provided by an embodiment of the present invention.

[0054] Icons: 300 - data access end; 310 - memory; 320 - processor; 400 - data processing device; 401 - receiving module; 402 - processing module; 403 - sending module. Detailed implementation manners

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations.

[0056] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0057] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0058] In addition, terms such as "first" and "second" are used only for distinguishing descriptions and cannot be construed as indicating or implying relative importance.

[0059] It should be noted that, without conflict, the features in the embodiments of the present invention can be combined with each other.

[0060] In the era of big data, with the explosive growth of massive data, distributed storage is increasingly being applied. Distributed storage requires the storage system to have characteristics such as high availability and high performance. However, when traditional file systems are applied to distributed storage, there are mostly problems such as poor read and write performance and low reliability. At the same time, when the number of nodes is relatively large, it also poses a great challenge to ensuring data consistency.

[0061] Current industry solutions include allocating location information for data to perform data reading and writing, but this method will cause data fragmentation, and the random reading and writing of data will result in poor performance of the storage system; or storing the metadata of data in multiple storage nodes and copying replicas, and at the same time performing checksum and repair on the metadata to ensure data consistency, but the drawback of this method is that replica copying will occupy extra space, and checksum and repair are also time-consuming and consume system performance; or when reading and writing data, judging whether the data version numbers of each replica of the data are consistent. If not, select the most complete replica for replacement, but this method cannot accurately judge whether the data version numbers of each replica are consistent when a node is offline or data anomalies occur, which will lead to data inconsistency; or optimizing erasure coding technology and multi-replica technology to improve the distributed data storage performance, but when the data volume is large, algorithm optimization cannot save much performance.

[0062] That is to say, the existing distributed storage technology has not yet solved the problem of how to ensure data consistency while improving the read / write performance of the distributed storage system.

[0063] In view of this, embodiments of the present invention provide a data processing method, which will be introduced in detail below.

[0064] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the result of the distributed storage system provided by the embodiments of the present invention. The distributed storage system includes a data access end and multiple storage nodes. The data access end is communicatively connected to each storage node.

[0065] The data access end can interact with the upper-layer application or an external host, receive a data write request sent by the upper-layer application or the external host. As Figure 2 shown, the data access end divides the data into n original data blocks and m parity data blocks according to the erasure coding algorithm, and at the same time generates a data version number for each original data block or parity data block. Finally, each original data block or parity data block and the corresponding data version number are written into each storage node at the same time. When a data read request is made, the data access end reads the data version number of the data block from the storage node for comparison. If the data version numbers of the data blocks are the same, the data block is read to respond to the data read request. If the data version numbers of the read data blocks are different, the data access end will perform data recovery on the original data blocks and parity data blocks through erasure calculation and then read. The data access end can be a server, a personal computer (hereinafter referred to as PC), a laptop computer, etc. The data access end can also be one or more program modules on a device, or a virtual machine or container running on a device. The client can also be a cluster composed of multiple devices. For example, it can be a collective name for multiple program modules distributed on multiple devices.

[0066] The storage node can store the original data blocks and / or parity data blocks from the data access end. The storage node can be a server, a PC, a laptop computer, etc. The storage node can be a physical storage node or a logical storage node obtained by dividing a physical storage node.

[0067] Please refer to Figure 3 , Figure 3 which shows a process of the data processing method provided by the embodiments of the present invention. The data processing method includes steps S101 to S103, and its execution subject is Figure 1 the data access end in

[0068] S101, receive a data write request.

[0069] Among them, the data write request includes the data to be stored, and the data write request can be sent by the upper-layer application or the external host to the data access end.

[0070] S102. Process the data to be stored to obtain multiple data blocks.

[0071] Among them, after receiving the data write request, the data access end divides the data to be stored with a length of L into n original data blocks, then generates m parity data blocks through the erasure ratio, and assigns serial numbers to all n original data blocks. The value range of the serial number is [1, n], and serial numbers are also assigned to m parity data blocks. The value range of the serial number is [1, m] (see Figure 4 ).

[0072] S103. For each data block, determine the target node from multiple storage nodes according to the serial number of the data block, and send the data block and the data version number of the data to be stored to the target node, so that the target node stores the data block and the data version number in the first space and the second space of the target node respectively.

[0073] Among them, the serial number of the data block and the number of the target node satisfy a preset mapping relationship, and the first space is before the second space. As Figure 4 shown, the storage nodes are divided into data nodes and parity nodes, which are used to store the original data blocks and parity data blocks respectively. The total number of data nodes is n, and the total number of parity nodes is m. And each data node is set with a number, and the value range of the number is [1, n], and each parity node is also set with a number, and the value range of the number is [1, m].

[0074] The preset mapping relationship includes the correspondence between the serial numbers of n original data blocks and the numbers of n data nodes, and the correspondence between the serial numbers of m parity data blocks and the numbers of m parity nodes. For each original data block or parity data block, according to its serial number and the preset mapping relationship, determine the target node from n data nodes or m parity nodes, and then send the original data block or parity data block and the data version number of the data to be stored generated based on the current timestamp to the target node. For example, the target node of the original data block 1 is the data node 1, and the target node of the parity data block m is the parity node m (see Figure 4 ).

[0075] For each data node or parity node, its disk space is divided into a first space and a second space, and the first space is before the second space. When writing the data block and the data version number, the data block should be written to disk before the data version number, thus avoiding the situation where the data version number is stored successfully but the data block is stored failed, enabling the data access end to recover the data block with storage failure in time to ensure data consistency.

[0076] The above method provided by the embodiment of the present invention has the beneficial effect that by sending each data block together with the data version number of the data to be stored to the target node, the target node stores the data block and the data version number in its first space and second space respectively, and the first space in the target node is located before the second space, thereby avoiding the situation where the data version numbers in each storage node are the same but the data block storage fails, enabling the data access end to promptly recover the data blocks with storage failures and ensuring data consistency.

[0077] In the existing method, since the data version number is generated by each storage node, when a certain storage node has an abnormal power failure, in addition to the situation where the data version number is stored successfully but the data block storage fails, there may also be a situation where the data version number is stored incorrectly but the data block is stored successfully, which will cause the data access end to recover the data blocks stored successfully in each storage node, consuming system performance. In this regard, before performing step S102, the embodiment of the present invention also provides an implementation manner in which the data access end generates the data version number, which will be introduced in detail below.

[0078] In the embodiment of the present invention, when the data access end generates the data version number of the data to be stored, there may be the following two situations.

[0079] Situation 1: If there is one data write request, the current time stamp is used as the data version number of the data to be stored.

[0080] Among them, when the data access end receives one data write request, the system time stamp of the current distributed storage system can be directly used as the data version number of the data to be stored in this data write request.

[0081] Situation 2: If there are multiple data write requests, the current time stamp is incremented multiple times, and according to the order of receipt time of each data write request, the result of each increment operation is used as the data version number of the data to be stored in one data write request.

[0082] Among them, when the data access end receives multiple data write requests, the system time stamp of the current distributed storage system is used as the initial value for multiple increment operations. The number of increment operations is the total number of data write requests, and according to the order of receipt time of each data write request, the result of each increment operation is used as the data version number of the data to be stored in one data write request, so as to obtain the data version number of the data to be stored in each data write request.

[0083] The above generation method of the data version number provided by the embodiment of the present invention can realize unified management of the data version number and ensure data consistency through this data version number.

[0084] The following provides a detailed introduction to step S102.

[0085] Please refer to Figure 5 , Figure 5 which shows a process of the implementation manner of step S102 provided by an embodiment of the present invention. Step S102 includes sub-steps S102-1 to S102-3.

[0086] S102-1: The data to be stored is segmented into multiple data stripes according to a preset length.

[0087] Among them, the length of each data stripe is the preset length. As Figure 4 shown, the preset length is x, and the data to be stored with a length of L is segmented into L / x data stripes.

[0088] S102-2: Every preset number of data stripes are combined into an original data block, obtaining multiple original data blocks.

[0089] Among them, the preset number is determined by the preset length, the number of data nodes, and the length of the data to be stored. As Figure 4 shown, the length of the data to be stored is L, the preset length is x, and the number of data nodes is n, then the preset number is L / nx. It can be understood that, in accordance with the segmentation order of the data stripes, every L / nx data stripes form an original data block, and a total of n original data blocks are obtained.

[0090] S102-3: Erasure coding is performed on the multiple original data blocks to obtain multiple parity data blocks.

[0091] Among them, as Figure 4 shown, erasure operation is performed on n original data blocks to obtain m parity data blocks. It can be understood that each parity data block includes L / nx data stripes, and the length of each data stripe is x.

[0092] As Figure 4 shown, a fixed and continuous first space and a second space are allocated for the data version numbers of the data blocks and the data to be stored in each storage node, changing the data reading and writing method within a data block from random reading and writing to sequential reading and writing, improving the reading and writing performance of the distributed storage system.

[0093] After introducing the process of the data access end processing the data write request, the following will provide a detailed introduction to the process of the data access end processing the data read request.

[0094] Please refer to Figure 6 , Figure 6 which shows another process of the data processing method provided by an embodiment of the present invention. The data processing method includes steps S201 to S207.

[0095] S201, Receive a data read request.

[0096] Among them, the data read request includes the writing node order of the data to be read. The data to be read includes the original data blocks stored in multiple data nodes by the data access end through processing data write requests. The writing node order refers to the position order of the original data blocks in the first space of the storage nodes.

[0097] As Figure 4 shown, the data access end processes data write request 1, data write request 2,..., data write request k in sequence according to the reception time. Among them, the original data block 1 corresponding to data write request 1, the original data block 1 corresponding to data write request 2,..., the original data block 1 corresponding to data write request k are sequentially written into the first space of data node 1. Similarly, the data version numbers of the data to be stored in data write request 1, the data version numbers of the data to be stored in data write request 2,..., the data version numbers of the data to be stored in data write request k are sequentially written into the second space of data node 1. It can be understood that the original data block n corresponding to data write request 1, the original data block n corresponding to data write request 2,..., the original data block n corresponding to data write request k are sequentially written into the first space of data node n, and the data version numbers of the data to be stored in data write request 1, the data version numbers of the data to be stored in data write request 2,..., the data version numbers of the data to be stored in data write request k are sequentially written into the second space of data node n. If the writing node order of the data to be read is 2, the data to be read is composed of the second original data blocks from data node 1 to data node n.

[0098] S201, According to the writing node order, read the target data version numbers from the second spaces of each storage node.

[0099] Among them, multiple data version numbers are stored in the second space of each storage node, and the space sizes occupied by each data version number are the same. For each storage node, determine the target area in the second space according to the writing node order and the space size occupied by the data version number, and take the content read from the target area as the target data version number.

[0100] For example, if the writing node order is 4 and the space size occupied by each data version number is 8B, then calculate that the target area in the second space of each storage node is from the 24th B to the 32nd B, and take the content read from the target area in the second space of each storage node as the target data version number corresponding to that storage node.

[0101] S203, If all the target data version numbers are the same, then according to the writing node order, read the target data blocks from the first spaces of each storage node.

[0102] Among them, when the target data version numbers read from the second spaces of each storage node are all the same, it means that the data blocks used to form the data to be read are all successfully stored at each storage node. Similarly, multiple data blocks are stored in the first space of each storage node, and the space size occupied by each data block is the same. For each storage node, according to the writing node order and the space size occupied by the data block, the target area in the first space is determined, the content read from the target area is used as the target data block, and then all the target data blocks are combined to obtain the data to be read.

[0103] For example, if the writing node order is 4 and the space size occupied by each data block is 128K, then the target area in the first space of each storage node is calculated to be from 384K to 512K, and the content read from the target area in the first space of each storage node is used as the target data block corresponding to that storage node.

[0104] S204, if there are inconsistent target data version numbers, divide the multiple storage nodes into normal nodes and abnormal nodes according to each target data version number.

[0105] Among them, when there are inconsistent target data version numbers among the target data version numbers read from the second spaces of each storage node, it means that the data blocks used to form the data to be read at some storage nodes are stored unsuccessfully. At this time, according to each target data version number, the multiple storage nodes are divided into two categories: normal nodes and abnormal nodes. Since the data blocks used to form the data to be read at the normal nodes are stored successfully and the data blocks used to form the data to be read at the abnormal nodes are stored unsuccessfully, it can be understood that the target data version numbers corresponding to all normal nodes are the same, and the target data version number corresponding to each abnormal node is inconsistent with the target data version numbers corresponding to all normal nodes.

[0106] S205, according to the writing node order, read the target data blocks from the first space of each normal node.

[0107] Among them, similar to the above step 203, multiple data blocks are stored in the first space of each normal node, and the space size occupied by each data block is the same. For each normal node, according to the writing node order and the space size occupied by the data block, the target area in the first space is determined, and the content read from the target area is used as the target data block.

[0108] S206, restore the target data blocks corresponding to each abnormal node according to the target data blocks corresponding to each normal node.

[0109] Among them, by performing erasure correction calculation on the target data blocks corresponding to all normal nodes, the target data blocks corresponding to each abnormal node are restored.

[0110] S207. Generate the data to be read according to the preset mapping relationship and all target data blocks in response to the data read request.

[0111] Among them, as Figure 7 shown, when the target data version numbers read from the second space of each storage node are all the same, all the target data blocks read from the first space of the slave data nodes can be used to generate the data to be read. As Figure 8 shown, when there are inconsistent target data version numbers among the target data version numbers read from the second space of each storage node, the target data blocks corresponding to all data nodes (which may be normal nodes or abnormal nodes) can be used to generate the data to be read.

[0112] The implementation process of step S207 is as follows:

[0113] First, assign serial numbers to each target data block according to the preset mapping relationship and the number of each storage node;

[0114] Then, sort all the target data blocks according to the serial numbers of each target data block to obtain the data to be read.

[0115] It can be understood that since the data blocks stored at the abnormal nodes for composing the data to be read are storage failures, in order to facilitate subsequent data reading, the target data blocks corresponding to the abnormal nodes obtained by the recovery process also need to be rewritten. The detailed implementation process is as follows:

[0116] First, determine the target area in the first space of each abnormal node according to the write node order and the preset size;

[0117] Then, for each abnormal node, use the target data block corresponding to the abnormal node to overwrite the content of the target area of the abnormal node.

[0118] To introduce the foregoing data processing method more clearly, the embodiment of the present invention assumes that the number of storage nodes in the distributed storage system is 3 (2 data nodes and 1 parity node), and the erasure ratio is 2:1 for illustration.

[0119] As Figure 9 shown, the data access end receives a data write request sent by the upper-layer application or an external host, splits the to-be-stored data with a total length of 256K in the data write request into 2 original data blocks and 1 parity data block according to the erasure ratio. Each original data block or parity data block includes 32 data stripes, and the length of each data stripe is 4K. According to the current timestamp of the distributed storage system, the data version number of the to-be-stored data is obtained as 164961834.

[0120] In response to a data write request, write the original data block numbered 1 and the data version number into the first space and the second space of the data node numbered 1 respectively according to the preset mapping relationship, write the original data block numbered 2 and the data version number into the first space and the second space of the data node numbered 2, and write the check data block numbered 1 and the data version number into the first space and the second space of the check node numbered 1.

[0121] Each data node consists of multiple first spaces of 64MB in size and second spaces of 128KB in size. Multiple data write requests store multiple original data blocks and corresponding data version numbers continuously on the disk of the data node. Similarly, each check node also consists of multiple first spaces of 64MB in size and second spaces of 128KB in size. Multiple data write requests store multiple check blocks and corresponding data version numbers continuously on the disk of the data node.

[0122] The data access end receives a data read request sent by an upper-layer application or an external host, reads the target data version numbers from the second spaces of 2 data nodes and 1 check node according to the writing node order of the data to be read in the data read request, and makes a comparison.

[0123] As Figure 10 shown, if the target data version numbers read from the second spaces of 2 data nodes and 1 check node are consistent and the erasure ratio is satisfied, read the target data blocks from the first spaces of 2 data nodes according to the writing node order of the data to be read, and then combine all the target data blocks according to the preset mapping relationship and the numbers of the data nodes to obtain the data to be read, so as to respond to the data read request.

[0124] As Figure 11 shown, if the target data version number read from the second space of data node 1 is inconsistent with the target data version number read from the second space of data node 1 and is consistent with the target data version number read from the second space of check node 1, and the erasure ratio is satisfied, the data access end first reads the target data blocks from the first spaces of data node 1 and check node 1 according to the writing node order of the data to be read, and restores the target data block corresponding to data node 2 through erasure calculation. After the restoration process is completed, the data access end combines the target data blocks corresponding to data node 1 and data node 2 into the data to be read according to the preset mapping relationship and the numbers of the data nodes, so as to respond to the data read request.

[0125] Furthermore, an embodiment of the present invention also provides a structural schematic diagram of the data access end 300. Please refer to Figure 12 , the data access end 300 may include a memory 310 and a processor 320.

[0126] Among them, the processor 320 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the data processing method provided in the foregoing method embodiments.

[0127] The memory 310 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 310 may exist independently and be connected to the processor 320 through a communication bus. The memory 310 may also be integrated with the processor 320. Among them, the memory 310 is used to store machine-executable instructions for executing the solution of this application. The processor 320 is used to execute the machine-executable instructions stored in the memory 310 to implement the foregoing method embodiments.

[0128] An embodiment of the present invention further provides a computer-readable storage medium including a computer program, and the computer program can be used to perform related operations in the data processing method provided in the foregoing method embodiments when executed.

[0129] Please refer to Figure 13 , Figure 13 which is a functional unit block diagram of a data processing device 400 provided by an embodiment of the present invention. The data processing device 400 is applied to the data access end 300 and may include a receiving module 401, a processing module 402, and a sending module 403. Among them, the receiving module 401, the processing module 402, and the sending module 403 can all be stored in the memory or computer-readable storage medium in software form. It should be noted that for the data processing device 400 provided by the embodiment of the present invention, its basic principle and the technical effects generated are the same as those of the foregoing embodiments. For the sake of brief description, the parts not mentioned in the embodiments of the present invention are not pointed out.

[0130] The receiving module 401 is used to receive a data write request, and the data write request includes data to be stored.

[0131] A processing module 402 is configured to process the data to be stored to obtain a plurality of data blocks, and each data block is assigned a serial number.

[0132] A sending module 403 is configured to, for each data block, determine a target node from a plurality of storage nodes according to the serial number of the data block, and send the data block and the data version number of the data to be stored to the target node, so that the target node stores the data block and the data version number in a first space and a second space of the target node respectively, wherein the serial number of the data block and the number of the target node satisfy a preset mapping relationship, and the first space is located before the second space.

[0133] In one implementation, the processing module 402 is further configured to use the current timestamp as the data version number of the data to be stored if the data write request is one; if the data write requests are multiple, perform multiple self-increment operations on the current timestamp, and use the result of each self-increment operation as the data version number of the data to be stored in a data write request according to the sequence of reception times of each data write request.

[0134] In one implementation, the processing module 402 is specifically configured to divide the data to be stored into a plurality of data stripes according to a preset length; form each preset number of data stripes into an original data block to obtain a plurality of check data blocks; perform erasure coding on the plurality of original data blocks to obtain a plurality of check data blocks, and the plurality of data blocks include a plurality of original data blocks and a plurality of check data blocks.

[0135] In one implementation, the receiving module 401 is further configured to receive a data read request, and the data read request includes the write node sequence of the data to be read; the processing module 402 is further configured to read a target data version number from the second space of each storage node according to the write node sequence; if all the target data version numbers are the same, read the target data block from the first space of each storage node according to the write node sequence; generate the data to be read according to the preset mapping relationship and all the target data blocks to respond to the data read request.

[0136] In one implementation, the processing module 402 is further configured to, if there are inconsistent target data version numbers, divide the plurality of storage nodes into normal nodes and abnormal nodes according to each target data version number, wherein the target data version numbers corresponding to all the normal nodes are the same, and the target data version number corresponding to each abnormal node is different from the target data version numbers corresponding to all the normal nodes; read the target data block from the first space of each normal node according to the write node sequence; recover the target data block corresponding to each abnormal node according to the target data block corresponding to each normal node; generate the data to be read according to the preset mapping relationship and all the target data blocks to respond to the data read request.

[0137] In one implementation, the processing module 402 is specifically configured to assign serial numbers to each target data block according to a preset mapping relationship and the number of each storage node; sort all the target data blocks according to the serial numbers of each target data block to obtain the data to be read.

[0138] In one implementation, the processing module 402 is further configured to determine a target area in the first space of each abnormal node according to the write node order and a preset size; for each abnormal node, use the target data block corresponding to the abnormal node to overwrite the content of the target area of the abnormal node.

[0139] A data processing method, apparatus, data access end and storage medium provided by an embodiment of the present invention. First, a data write request is received, and the data write request includes data to be stored; then, the data to be stored is processed to obtain a plurality of data blocks, and each data block is assigned a serial number; then, for each data block, according to the serial number of the data block, a target node is determined from a plurality of storage nodes, and the data block and the data version number of the data to be stored are sent to the target node, so that the target node stores the data block and the data version number in the first space and the second space of the target node respectively, where the serial number of the data block and the number of the target node satisfy a preset mapping relationship, and the first space is located before the second space. Since the embodiment of the present invention sends each data block and the data version number of the data to be stored to the target node together, the target node stores the data block and the data version number in its first space and second space respectively, and the first space in the target node is located before the second space, thereby avoiding the situation where the data version numbers in each storage node are the same while the data block storage fails, enabling the data access end to timely recover the data blocks that fail to be stored and ensuring data consistency.

[0140] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, Applied to the data access end in a distributed storage system, the distributed storage system further includes a plurality of storage nodes, each of the storage nodes is set with a number, and each of the storage nodes is communicatively connected to the data access end. The method includes: Receiving a data write request, the data write request including data to be stored; Processing the data to be stored to obtain a plurality of data blocks, and each of the data blocks is assigned a serial number; For each of the data blocks, determining a target node from the plurality of storage nodes according to the serial number of the data block, and sending the data block and the data version number of the data to be stored to the target node, so that the target node stores the data block and the data version number in a first space and a second space of the target node respectively, wherein the serial number of the data block and the number of the target node satisfy a preset mapping relationship, the first space and the second space are consecutive, and the first space is located before the second space; Receiving a data read request, the data read request including the write node order of the data to be read; Reading the target data version number from the second space of each storage node according to the write node order; If there are inconsistent target data version numbers, dividing the plurality of storage nodes into normal nodes and abnormal nodes according to each of the target data version numbers, wherein the target data version numbers corresponding to all the normal nodes are the same, and the target data version number corresponding to each abnormal node is different from the target data version numbers corresponding to all the normal nodes; Reading the target data block from the first space of each normal node according to the write node order; Restoring the target data block corresponding to each abnormal node according to the target data block corresponding to each normal node; Generating the data to be read according to the preset mapping relationship and all the target data blocks to respond to the data read request.

2. The method according to claim 1, characterized in that, Before the step of processing the data to be stored to obtain a plurality of data blocks, the method further includes: If the data write request is one, using the current timestamp as the data version number of the data to be stored; If the data write requests are multiple, performing multiple self-increment operations on the current timestamp, and using the result of each self-increment operation as the data version number of the data to be stored in one of the data write requests according to the sequence of receiving time of each data write request.

3. The method according to claim 1, wherein The step of processing the data to be stored to obtain a plurality of data blocks includes: Cutting the data to be stored into a plurality of data stripes according to a preset length; Combining every preset number of the data stripes into an original data block to obtain a plurality of original data blocks; Performing erasure coding on the plurality of original data blocks to obtain a plurality of check data blocks, and the plurality of data blocks include the plurality of original data blocks and the plurality of check data blocks.

4. The method according to claim 1, wherein After reading the target data version number from the second space of each storage node according to the write node order, the method further includes: If all the target data version numbers are the same, read the target data blocks from the first space of each storage node according to the writing node order; Generate the data to be read according to the preset mapping relationship and all the target data blocks to respond to the data read request.

5. The method according to claim 1 or 4, characterized in that The step of generating the data to be read according to the preset mapping relationship and all the target data blocks includes: Allocate serial numbers to each target data block according to the preset mapping relationship and the number of each storage node; Sort all the target data blocks according to the serial numbers of each target data block to obtain the data to be read.

6. The method according to claim 1, wherein The method further includes: Determine the target area in the first space of each abnormal node according to the writing node order and the preset size; For each abnormal node, use the target data block corresponding to the abnormal node to overwrite the content of the target area of the abnormal node.

7. A data processing device, characterized in that, Applied to the data access end in a distributed storage system, the distributed storage system further includes multiple storage nodes, each storage node is set with a number, and each storage node is communicatively connected to the data access end. The device includes: A receiving module, configured to receive a data write request, where the data write request includes data to be stored; A processing module, configured to process the data to be stored to obtain multiple data blocks, and each data block is allocated a serial number; A sending module, configured to, for each data block, determine a target node from the multiple storage nodes according to the serial number of the data block, and send the data block and the data version number of the data to be stored to the target node, so that the target node stores the data block and the data version number in the first space and the second space of the target node respectively, where the serial number of the data block and the number of the target node satisfy a preset mapping relationship, the first space and the second space are continuous, and the first space is located before the second space; The receiving module is further configured to receive a data read request, where the data read request includes the writing node order of the data to be read; The processing module is further configured to, according to the writing node order, read the target data version numbers from the second space of each storage node; if there are inconsistent target data version numbers, divide the multiple storage nodes into normal nodes and abnormal nodes according to each target data version number, where the target data version numbers corresponding to all the normal nodes are the same, and the target data version number corresponding to each abnormal node is inconsistent with the target data version numbers corresponding to all the normal nodes; read the target data blocks from the first space of each normal node according to the writing node order; restore the target data blocks corresponding to each abnormal node according to the target data blocks corresponding to each normal node; generate the data to be read according to the preset mapping relationship and all the target data blocks to respond to the data read request.

8. A data access terminal, characterized in that, It includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the data processing method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1-6.

Images