Distributed storage erasure code method supporting near data processing

Abstract

The invention discloses a distributed storage erasure code method supporting near data processing, which comprises the following steps: uploading a preset number of file objects according to an erasure code strategy, and recording the number and attribute of each file object; cutting off each file object according to the coding length, and coding the cut-off part to obtain a coded file; according to the serial number and the attribute of the file object, obtaining the length of data needing to be read and the offset of the initial position, and reading the file object data of the coded file; and returning the file object data to the client. According to the method, when the erasure codes are coded, independent file object data can fall on independent storage equipment after being coded, so that the independence and integrity of the data are ensured, the situation that semantically associated data are dispersed to different storage equipment when an original erasure code mechanism is coded is prevented, and the application of a near data processing technology can be supported.

Description

Technical Field

[0001] The present invention belongs to the field of distributed storage technology, and in particular relates to a distributed storage erasure coding method supporting near data processing. Background Art

[0002] In the era of big data, due to the huge scale of data, data processing and data storage cannot be balanced and fully developed, and a large amount of data migration has led to a sharp increase in system energy consumption. Researchers have begun to try to move the computing unit to a location close to the memory, that is, near data processing technology. The core idea of ​​near data processing is to use the processing power in the memory to perform simple computing on the data and then only transmit the data processing results to the host, which saves a lot of system resources. It has the characteristics of low latency, high reliability, high scalability and low power consumption, and has broad application prospects. For example, edge computing in the cloud edge terminal transfers computing tasks from the central cloud to the edge of the network, that is, closer to the data source or user. Edge nodes can use near data processing technology to process data collected by user devices nearby to reduce the amount of data uploaded to the data center.

[0003] However, the erasure code implementation methods in mainstream distributed storage systems (such as Ceph, DAOS, and MinIO) cannot efficiently support near data processing. Taking Ceph as an example, its underlying erasure code mechanism will segment and encode the written object data, causing the originally semantically related data to be dispersed into different object storage devices (OSDs). When the application layer requires the OSD to analyze and process the data of an object, the OSD still needs to read and transmit the object's shard data from other OSDs, which violates the original intention of near data processing and will result in a large amount of network transmission overhead. Therefore, there is an urgent need for a distributed storage erasure code method that supports near data processing. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a distributed storage erasure coding method that supports near data processing, which can save the metadata of all file objects and increase the friendliness to the application layer.

[0005] To achieve the above object, the present invention provides a distributed storage erasure coding method supporting near data processing, comprising:

[0006] Upload a preset number of file objects according to the erasure code strategy, and record the number and attributes of each file object;

[0007] Truncating each of the file objects according to the encoding length, encoding the truncated part, and obtaining an encoded file;

[0008] According to the number and attributes of the file object, the length of the data to be read and the starting position offset are obtained, and the file object data of the encoded file is read;

[0009] The file object data is returned to the client.

[0010] Optionally, uploading a preset number of file objects according to the erasure coding strategy includes:

[0011] Obtaining a write file, and determining file data according to the write file;

[0012] Using the storage unit size defined by the Ceph underlying layer, read each file data into a temporary cache file, and record the number and attributes of each file object;

[0013] Write the temporary cache file to the bottom layer and upload it.

[0014] Optionally, determining the file data according to the written file includes:

[0015] According to the written file, determining whether the file content is empty;

[0016] If it is not empty, get the file data;

[0017] Otherwise, output an error message.

[0018] Optionally, truncating the file object according to the encoding length includes:

[0019] Get the number of data blocks according to the erasure code sharding strategy;

[0020] Obtaining the encoding length according to the file object and the number of data blocks;

[0021] According to the encoding length, the original data is truncated and the truncated part is written to the bottom layer.

[0022] Optionally, the method for obtaining the encoding length is:

[0023] The encoding length is obtained by dividing the total data length of the file object by the number of data blocks.

[0024] Optionally, encoding the truncated portion to obtain the encoded file includes:

[0025] The number of stripes is calculated, and when the bottom layer receives the data, the data is written into the stripe, and the data in the stripe is encoded to obtain an encoded file.

[0026] Optionally, calculating the number of stripes includes:

[0027] Get the length of a single strip;

[0028] The number of stripes is obtained according to the total data length of the file object divided by the length of the single stripe.

[0029] Optionally, reading the file object data of the encoded file includes:

[0030] Using a read instruction, the file object data number of the encoded file to be read is passed in;

[0031] Obtain the file object data length that needs to be read from the encoded file and its starting position offset in the encoded file;

[0032] Convert the data length and starting position offset into internal data

[0033] The file object data of the encoded file is read according to the starting position offset and the read length.

[0034] Optionally, the file object data number of the encoded file to be read by using the read instruction includes:

[0035] Determining whether the file object data in the encoded file is correctly stored in the system;

[0036] If it is stored in the system, the get_shard instruction is used to pass in the number of the file object data that needs to be read from the encoded file;

[0037] Otherwise, output an error message.

[0038] Compared with the prior art, the present invention has the following advantages and technical effects:

[0039] The present invention implements a new erasure code architecture that supports near data processing at the bottom layer of the distributed system Ceph, ensuring that when erasure code is encoded, an independent file object data can fall on an independent storage device after encoding, ensuring the independent integrity of the data, preventing the original erasure code mechanism from dispersing semantically related data to different storage devices during encoding, and supporting the application of near data processing technology. An interface that supports simultaneous writing of multiple file object data and reading of a single file object is implemented on the Ceph client, which not only matches the use of the underlying erasure code architecture, but also improves the engineering practicality of the present invention, and saves the metadata of all file objects, increasing the friendliness to the application layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1It is a schematic diagram of the comparison principle between the Ceph original erasure code architecture of the embodiment of the present invention and the Ceph-based erasure code architecture supporting near data processing designed by the present invention;

[0042] Figure 2 A flow chart of a distributed storage erasure coding method supporting near data processing according to an embodiment of the present invention;

[0043] Figure 3 A performance test comparison diagram of the architecture of an embodiment of the present invention and the existing architecture in a near data processing scenario;

[0044] Figure 4 The figure is a performance test comparison chart of the architecture of the embodiment of the present invention and the existing architecture in a conventional scenario. DETAILED DESCRIPTION

[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] The core idea of ​​erasure coding is to shard the data written into the storage system. Assuming there are n storage devices in total, the original data is divided into k fragments as needed, which are data blocks, and the remaining nk=m are check blocks, also called coding blocks. Erasure coding can restore the original data through any k copies of the n copies of data. That is, if there are data fragments less than or equal to m that fail, the original data can still be restored through the remaining data. In distributed storage systems, erasure coding technology has higher storage reliability and storage efficiency than another storage technology with multiple copies, especially in large file storage scenarios, and the cost is lower. This makes erasure coding technology one of the most popular data protection storage strategies. In the era of data information explosion, erasure coding technology is being increasingly widely used in various distributed storage systems.

[0048] The present invention proposes a distributed storage erasure coding method that supports near data processing, such as Figure 2 As shown, the specific steps include:

[0049] Upload a preset number of file objects according to the erasure code strategy, and record the number and attributes of each file object;

[0050] Each file object is truncated according to the encoding length, the truncated part is encoded, and the encoded file is obtained;

[0051] According to the number and attributes of the file object, obtain the data length and starting position offset to be read, and read the file object data of the encoded file;

[0052] Returns the file object data to the client.

[0053] Further, uploading a preset number of file objects according to the erasure coding strategy includes:

[0054] Get the written file, and determine the file data according to the written file;

[0055] Use the storage unit size defined by Ceph underlying layer to read file data into temporary cache file and record the number and attributes of file object;

[0056] Write the temporary cache file to the underlying layer and upload it.

[0057] Specifically, Ceph's RADOS client provides two interfaces that support simultaneous writing of multiple file objects and reading of a single file object, including:

[0058] For interfaces that support writing multiple file objects simultaneously: first obtain the number of files to be written;

[0059] Loop to determine whether the written file is empty, and output an error message if it is empty;

[0060] If all files are readable, the file data is read in batches and merged into a large data cache, and the Ceph underlying data write interface is called to complete the writing.

[0061] Furthermore, truncating the file object according to the encoding length includes:

[0062] Get the number of data blocks according to the erasure code sharding strategy;

[0063] Get the encoding length based on the file object and the number of data blocks;

[0064] According to the encoding length, the original data is truncated and the truncated part is written to the bottom layer.

[0065] Furthermore, the method for obtaining the encoding length is:

[0066] Obtain the encoding length by dividing the total data length of the file object by the number of data blocks.

[0067] Specifically, before writing the object data, the number of data blocks required by the erasure code sharding strategy is obtained;

[0068] Divide the length of the written data by the number of data blocks to get the length of each encoding part;

[0069] The original data length is truncated in sequence according to the length of the encoded part;

[0070] Use the data structure storing key-value pairs to save the truncated parts in sequence and write them to the bottom layer;

[0071] After receiving the data, the bottom layer calculates the total length of the written data;

[0072] Calculate the offset between each data block;

[0073] Divide the written data into multiple stripes, perform data partitioning on each stripe, truncate a portion of each independent data according to the size of the stripe encoding block and write it to the temporary cache until a stripe is full;

[0074] When a stripe is full of data, the underlying calling interface encodes the data in the stripe.

[0075] Furthermore, the method for obtaining the encoding length is:

[0076] Obtain the encoding length by dividing the total data length of the file object by the number of data blocks.

[0077] Furthermore, calculating the number of stripes includes:

[0078] Get the length of a single strip;

[0079] Get the number of stripes by dividing the total data length of the file object by the length of a single stripe.

[0080] Furthermore, using the get_shard instruction, the file object data of the encoded file is read including:

[0081] Use the get_shard command to pass in the file object data number that needs to be read from the encoded file;

[0082] Get the file object data length that needs to be read from the encoded file and its starting position offset in the encoded file;

[0083] Convert the data length and starting position offset into internal data

[0084] Read the file object data of the encoded file according to the starting position offset and the reading length.

[0085] Specifically, for an interface that supports reading a single file object, the file object number that the user writes through the interface and needs to read is first processed at the Ceph bottom layer and processed into a data reading unit supported by Ceph internally;

[0086] After processing, the length of the data to be read and the offset of the starting position obtained from the file object attributes are written to the next layer interface;

[0087] After writing, the interface of this layer judges the written file and outputs an error message if the file is not stored in the system;

[0088] Otherwise, the underlying data read interface of Ceph is called to read the data according to the length and starting position offset of the data to be read.

[0089] This embodiment proposes an erasure code architecture that supports near data processing technology, which is implemented based on the distributed storage system Ceph. Figure 1 When writing object codes, the erasure code of the original version of Ceph distributes the encoded data to multiple different storage devices. However, this embodiment adopts a method in which multiple data objects are encoded simultaneously and a single complete data object is stored on a separate storage device to ensure the integrity and independence of the data. The Ceph client provides the function of supporting the writing of multiple file objects and the reading of single file objects, and saves the metadata of all file objects. The specific implementation method is divided into two aspects:

[0090] On the one hand, an erasure code architecture that supports near data processing is designed in the underlying erasure code module of Ceph. It should be noted that there are some related terms in the erasure code module in Ceph: (1) Chunk: When Ceph encodes an object based on an erasure code, each encoding will generate blocks of several sizes (which must be ordered). (2) Stripe: If the object to be encoded is too large and the encoding cannot be completed in one go, the part that is encoded each time can be called a stripe and encoded multiple times. The stripes within the same object are ordered. (3) Others: k generally refers to the number of data blocks in the stripe; m generally refers to the number of check blocks in the stripe; n is the number of blocks in the stripe. The architecture design method proposed in this embodiment specifically includes the following steps:

[0091] Step 1: Before processing the written object data, call the interface to obtain the number of data blocks k required by the erasure code sharding strategy;

[0092] Step 2: Calculate the length m of each part after writing the data shard, where m = total data length / k. Calculate the length length that needs to be padded, and fill it into the original data in the form of zero padding;

[0093] Step 3: Define a to_encode to store the encoded data using the data structure map that stores key-value pairs. Cut the original data into k parts in units of m, write them into to_encode in order, and write to_encode into the bottom layer;

[0094] Step 4: After the data to_encode is written, the total length of the stored data is calculated first, and then the interface is called to obtain the length of a single stripe. The total length / strip length is used to obtain the total number of stripes required for encoding;

[0095] Step 5: Define a buf for storing encoded data using Ceph's custom memory management data structure bufferlist, and call the interface to obtain the chunk size chunk_size in a single stripe. Traverse the entire to_encode value, cut each of its k parts according to the chunk_size chunk size, and write them to buf until a stripe is full. Repeat the above process until all stripes are full of data. See Appendix Figure 1 , here we assume that the erasure code sharding strategy is k=3, m=2, and the client writes three files. At this time, the three files are integrated and written into the architecture designed in this embodiment, and the three files are divided into multiple small data blocks according to the stripe block size chunk_size. Each time, one block of the three files is written to three different storage devices respectively, and the data blocks of the same file are only written to the same storage device until all stripes are filled;

[0096] Step 6: Call the encoding interface of the Ceph erasure code module to encode the data in units of one stripe until all stripes are encoded.

[0097] Secondly, two file read and write interfaces, put_more and get_shard, are designed on the Ceph client. Put_more supports uploading multiple file objects, and get_shard supports reading a single file object. A metadata preservation mechanism is provided on the Ceph client to save the number and data size of the uploaded file object. The two interfaces are used in conjunction with the underlying architecture to improve the engineering practicality of this embodiment.

[0098] It should be noted that the underlying RADOS storage system of Ceph defines an abstract concept: storage pool. Pool specifies the type of data redundancy and the corresponding replica distribution strategy. Ceph currently implements two types of pools: replica type and erasure code type. The erasure code pool can customize the storage strategy and define the number of data blocks k and check blocks m as needed.

[0099] The object storage core of Ceph is RADOS, which completes the core functions of a storage system, including: the monitor module provides global configuration and system information for the entire storage cluster; implements the object addressing process through the crush algorithm; completes the reading and writing of objects and other data functions; provides data balancing function; completes the process of achieving data consistency in a pg memory through the peering process; provides data automatic recovery function; provides cloning and snapshot functions; realizes the object tiered storage function, etc.

[0100] The metadata preservation mechanism uses the extended attribute xattr of the object, which is equivalent to maintaining an index table inside the Ceph storage cluster. The corresponding file size can be found by file number, thus avoiding the application directly passing in the file length and starting position offset to be read from the client.

[0101] This example mainly studies RADOS type objects. RADOS is a command in Ceph, which mainly performs some operations on objects. Common commands include: (1) Outputting objects to a specified file: rados get{obj-name}{outfile};

[0102] (2) Write an object, you can specify the offset: rados put{obj-name}{infile}[--offsetoffset]-p{pool-name}; (3) Create an object: rados -p{pool-name}create{obj-name}; (4) Delete an object: rados -p{pool-name}rm{obj-name}...[--force-full].

[0103] Since the RADOS client does not support uploading multiple files at the same time, this embodiment provides a new interface put_more, which can upload multiple files to the storage pool at the same time and save the metadata of the files at the same time, including the following steps:

[0104] Step 1: Ceph detects that the client inputs the put_more command, determines whether the input parameters are correct, and calculates the number of input file objects;

[0105] Step 2: For the written data file, determine whether the file content is empty in turn. If it is empty, output an error message;

[0106] Step 3: For each uploaded file, use the variable file_count and the Linux file attribute structure structstat to record the file number and attributes respectively, and use the Linux system function fstat to obtain the size of each file.

[0107] Step 4: Convert the file number file_count and the file size file_stat.st_size into a format that matches the object attribute setting interface setxattr. Call the object attribute setting interface setxattr to save the file number and file size in the xattr attribute of the object.

[0108] Step 5: Call the Linux read function to read the file data into a temporary cache file each time according to the minimum storage unit size defined by the Ceph underlying layer until all the data is read;

[0109] Step 6: Write the temporary cache file to the underlying put_more interface;

[0110] Step 7: In the put_more interface, first obtain the object ID through the object name.

[0111] Step 8: Based on the object ID and the object data set to be written, call the write_full interface of the Ceph underlying layer to complete the writing of the entire object. If an error occurs during writing, an error message is output.

[0112] In this embodiment, the put_more command provided by the Ceph client aggregates multiple file data into a RADOS object and writes it to the storage. The get_shard command matching it needs to read out a single data file therein, thereby ensuring the correctness of the read file data. Specifically, the following steps are included:

[0113] Step 1: Ceph detects the get_shard command passed in by the client and determines whether the length of the file data required to be read by the command meets the internal specifications of Ceph. If not, an error message is output;

[0114] Step 2: Call the object attribute acquisition interface getxattr to obtain the size of the current file and the previous file as the required read length written to get_shard and the starting position offset of the required read file.

[0115] Step 3: Convert the required read length and the starting position offset of the file to be read into the data type supported by the Ceph system. If the conversion is wrong, an error message is output;

[0116] Step 4: After the conversion is successful, call the get_shard interface to pass in the starting position offset and length of the file to be read;

[0117] Step 5: Enter the get_shard interface. First, you need to determine whether the file you want to read is stored in the system. If it does not exist, an error message will be output.

[0118] Step 6: If the data can be read correctly, call the read interface of the underlying Ceph layer to pass in the starting position offset and length of the file to be read. The Ceph system reads the data of the specified file in the specified object based on the passed-in parameters.

[0119] The overall usage process of this embodiment is as follows: on the client, use the put_more command to upload k file objects according to the number of data blocks k defined by the erasure code strategy. The k file objects are stored in a RADOS object, and the metadata of the k file objects is saved. After the object is uploaded, it is encoded to ensure that a single data file is stored completely and independently on a single storage device. Finally, the get_shard interface is called to read the required file object data.

[0120] like Figure 3 A curve is given showing the throughput of reading object data as a function of the number of data blocks k defined by the erasure code strategy in a near data processing scenario, when the total size of written data is fixed at 90MB (the maximum amount of written data limited by the RADOS client), and the amount of data read each time is 90 / kMB. In a near data processing scenario, the OSD receives a processing command for an object from an application, then reads the object for processing and returns the result. In this case, the object does not need to be transmitted to the client. In this embodiment, the Ceph client is installed on the primary OSD and the above interface is called to simulate the object reading operation in a near data processing scenario. As can be seen from the figure, the erasure code architecture supporting near data processing proposed in this embodiment has a significant performance gain when reading object data in a near data processing scenario compared to the original version, which means that this embodiment can have lower latency when reading object data.

[0121] like Figure 4 The following graph shows the throughput of reading object data as a function of the number of data blocks k defined by the erasure coding strategy in a conventional scenario, i.e., when the client and the primary OSD are located on different nodes and the total amount of written data is fixed at 90MB, and the amount of data read each time is 90 / kMB. As can be seen from the figure, the erasure coding architecture supporting near data processing proposed in this embodiment has certain performance gains when reading object data in conventional scenarios compared to the original version.

[0122] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A distributed storage erasure coding method supporting near data processing, characterized in that: include: Upload a preset number of file objects according to the erasure code strategy, and record the number and attributes of each file object; Truncating each of the file objects according to the encoding length, encoding the truncated part, and obtaining an encoded file; According to the number and attributes of the file object, the length of the data to be read and the starting position offset are obtained, and the file object data of the encoded file is read; The file object data is returned to the client.

2. A distributed storage erasure coding method supporting near data processing according to claim 1, characterized in that: Uploading a preset number of file objects according to the erasure coding strategy includes: Obtaining a write file, and determining file data according to the write file; Using the storage unit size defined at the bottom layer of the distributed storage system Ceph, read each of the file data into a temporary cache file, and record the number and attributes of each of the file objects; Write the temporary cache file to the bottom layer and upload it.

3. A distributed storage erasure coding method supporting near data processing according to claim 2, characterized in that: According to each of the written files, determining the file data includes: According to each of the written files, determining whether the file content is empty; If it is not empty, get the file data; Otherwise, output an error message.

4. The distributed storage erasure coding method supporting near data processing according to claim 1, characterized in that: Truncating the file object according to the encoding length includes: Get the number of data blocks according to the erasure code sharding strategy; Obtaining the encoding length according to the file object and the number of data blocks; According to the encoding length, the original data is truncated and the truncated part is written to the bottom layer.

5. A distributed storage erasure coding method supporting near data processing according to claim 4, characterized in that: The method for obtaining the encoding length is: The encoding length is obtained by dividing the total data length of the file object by the number of data blocks.

6. A distributed storage erasure coding method supporting near data processing according to claim 4, characterized in that: Encoding the truncated part to obtain the encoded file includes: The number of stripes is calculated, and when the bottom layer receives the data, the data is written into the stripe, and the data in the stripe is encoded to obtain an encoded file.

7. A distributed storage erasure coding method supporting near data processing according to claim 6, characterized in that: Calculating the number of stripes includes: Get the length of a single strip; The number of stripes is obtained according to the total data length of the file object divided by the length of the single stripe.

8. The distributed storage erasure coding method supporting near data processing according to claim 1, characterized in that: Reading the file object data of the encoded file includes: Using a read instruction, the file object data number of the encoded file to be read is passed in; Obtain the file object data length that needs to be read from the encoded file and its starting position offset in the encoded file; Convert the data length and the starting position offset into internal data; The file object data of the encoded file is read according to the starting position offset and the read length.

9. A distributed storage erasure coding method supporting near data processing according to claim 8, characterized in that: The file object data number to be read from the encoded file passed in by the read instruction includes: Determining whether the file object data in the encoded file is correctly stored in the system; If it is stored in the system, the get_shard instruction is used to pass in the number of the file object data that needs to be read from the encoded file; Otherwise, output an error message.

Images