Parallel data access method and device applied to NFS cloud storage gateway

Abstract

The invention relates to the technical field of computers, and discloses a parallel data access method and device applied to an NFS cloud storage gateway, and the method comprises the steps: obtaining a file read-write request sent by a client, and carrying out the striping processing of the file read-write request; based on the striped file read-write request, using a lock-free queue to carry out multi-thread concurrent access to object storage; and performing file reading and writing on the object cloud storage to obtain a file reading and writing result, and feeding back the file reading and writing result to the client. According to the invention, concurrent lock-free reading and writing of large files are realized, and the file reading and writing rate of the NFS cloud storage gateway is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a parallel data access method and device applied to an NFS cloud storage gateway. Background Art

[0002] With the rise of cloud computing technology, object storage has been widely used for its advantages of high bandwidth, large scale, and strong scalability. In order to enable application services that rely on the POSIX (Portable Operating System Interface) protocol to enjoy the convenience brought by object storage, ZOS (Zettabyte ObjectStorge) provides an NFS (Network File Sysyem) cloud storage gateway based on S3FS (a file system) for users to use. The NFS cloud storage gateway can convert standard file POSIX semantics into S3 (Simple Storage Service) object storage semantics, ensuring that users can quickly use object storage without changing their own software, achieving perfect adaptation.

[0003] However, file access to the NFS cloud storage gateway requires multiple kernel-user interactions. When reading or writing the same file, only extremely inefficient serial access can be used, affecting the normal operation of user services. Summary of the invention

[0004] In view of this, the present invention provides a parallel data access method and device applied to an NFS cloud storage gateway to solve the problem of low access efficiency of the file access method of the NFS cloud storage gateway.

[0005] In a first aspect, the present invention provides a parallel data access method applied to an NFS cloud storage gateway, the method comprising:

[0006] Obtain the file read and write requests sent by the client, and process the file read and write requests in stripes;

[0007] Based on the striped file read and write requests, the lock-free queue multi-threaded concurrent access to the object storage is used;

[0008] Read and write files to the object cloud storage, obtain the file reading and writing results, and feed back the file reading and writing results to the client.

[0009] The present embodiment provides a parallel data access method applied to an NFS cloud storage gateway, which solves the problems of overlay read amplification and write amplification by striping the file read and write requests and dividing the file read and write requests from one object into multiple objects. In addition, based on the striped file read and write requests, the object storage is accessed concurrently by multi-threaded concurrent access using a lock-free queue, and the parallel data access to the file is optimized, thereby realizing concurrent lock-free reading and writing of large files and improving the file read and write rate of the NFS cloud storage gateway. Finally, the object cloud storage is used to read and write files to obtain the file read and write results, and the file read and write results are fed back to the client, thereby improving the access efficiency.

[0010] In an optional implementation, striping the file read and write requests includes:

[0011] Divide the file read and write requests into multiple Pages according to the preset division units;

[0012] Get a slice, and use the slice to divide the Page into multiple chunks; where slices correspond to Pages one to one.

[0013] This embodiment provides a parallel data access method for an NFS cloud storage gateway. By dividing a file read and write request into multiple Pages, and using slices to divide the Pages into multiple chunks, the file read and write request is divided from one into multiple ones. When there is an overwrite write in the file read and write request, it is not necessary to read all large files from the object storage and then write them, but only the corresponding stripe module needs to be changed, which greatly reduces large amplification and write amplification and improves the read and write rates.

[0014] In an optional implementation, based on the striped file read and write requests, using lock-free queues and multi-threaded concurrent access to object storage includes:

[0015] Write multiple chunks into the subtask queue in the lock-free queue respectively;

[0016] The file read and write requests in the subtask queue are asynchronously sent to the object storage through the subtask thread; wherein the subtask thread corresponds to the subtask queue one by one.

[0017] This embodiment provides a parallel data access method applied to an NFS cloud storage gateway. The NFS cloud storage gateway splits a file read and write request into multiple chunks, and writes the multiple chunks into a subtask queue in a lock-free queue. The object storage is accessed concurrently by multiple subtask threads, further improving the read and write efficiency.

[0018] In an optional implementation, writing multiple chunks into the subtask queues in the lock-free queue respectively includes:

[0019] Use the hash algorithm to write multiple chunks into the subtask queue respectively.

[0020] In an optional implementation, reading and writing files in the object cloud storage, obtaining file reading and writing results, and feeding back the file reading and writing results to the client include:

[0021] Determine the length of the file name based on the file read and write request;

[0022] Based on the length of the file name, the radix tree is used to read and write the file to obtain the file reading and writing results;

[0023] The file read and write processing results are written into the lock-free queue, and the file thread is used to feed back the file read and write processing results in the lock-free queue to the client.

[0024] This embodiment provides a parallel data access method applied to an NFS cloud storage gateway. Based on the file name length, a radix tree is used to read and write files to obtain file reading and writing results. The file information is completely saved locally on the NFS cloud storage gateway through a radix tree structure, which solves the problem of poor object storage list performance, ensures that client services are quickly uploaded to the cloud, and greatly improves file query efficiency.

[0025] In an optional implementation, based on the file name length, a radix tree is used to read and write files to obtain file reading and writing results, including:

[0026] Based on the length of the file name, the storage location of the file is queried using the radix tree, and the file is read and written based on the storage location of the file to obtain the file reading and writing result.

[0027] In a second aspect, the present invention provides a parallel data access device applied to an NFS cloud storage gateway, the device comprising:

[0028] The striping processing module is used to obtain the file read and write requests sent by the client and perform striping processing on the file read and write requests;

[0029] Parallel access module, which is used to concurrently access object storage using lock-free queues and multiple threads based on striped file read and write requests;

[0030] The read-write module is used to read and write files in the object cloud storage, obtain the file read-write results, and feed back the file read-write results to the client.

[0031] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute a parallel data access method applied to an NFS cloud storage gateway according to the first aspect or any corresponding embodiment thereof.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute a parallel data access method applied to an NFS cloud storage gateway according to the above-mentioned first aspect or any corresponding embodiment thereof.

[0033] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute a parallel data access method applied to an NFS cloud storage gateway according to the above-mentioned first aspect or any corresponding embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 This is a schematic diagram of an IO access path for reading and writing files by an S3FS-based NFS cloud storage gateway according to an embodiment of the present invention;

[0036] Figure 2 It is a flow chart of a parallel data access method applied to an NFS cloud storage gateway according to an embodiment of the present invention;

[0037] Figure 3 It is a flowchart of another parallel data access method applied to an NFS cloud storage gateway according to an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of NFS cloud storage gateway file striping according to an embodiment of the present invention;

[0039] Figure 5 It is a schematic diagram of a lock-free multi-threaded asynchronous concurrent architecture of an NFS cloud storage gateway according to an embodiment of the present invention;

[0040] Figure 6 It is a flowchart of another parallel data access method applied to an NFS cloud storage gateway according to an embodiment of the present invention;

[0041] Figure 7 It is a schematic diagram of the topological result of the NFS cloud storage gateway persisting file information to the local according to an embodiment of the present invention;

[0042] Figure 8 It is a structural block diagram of a parallel data access device applied to an NFS cloud storage gateway according to an embodiment of the present invention;

[0043] Fig. 9 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0045] NFS is an application based on the UDP / IP protocol (User Datagram Protocol / Internet Protocol). Its implementation mainly adopts the Remote Procedure Call (RPC) mechanism. RPC provides a set of operations for accessing remote files that are independent of the machine, operating system, and underlying transmission protocol. RPC uses the support of XDR, which is a machine-independent data description encoding protocol. It encodes and decodes data transmitted on the Internet in a format independent of any machine architecture and supports data transmission between heterogeneous systems. ZOS provides customers with a massive, elastic, highly reliable, and cost-effective storage product. It is a massive storage form specifically for cloud computing, big data, and unstructured data. It provides unlimited storage services for unstructured data (pictures, audio and video, text, and other format files) through the S3 protocol and standard service interfaces.

[0046] like Figure 1 As shown in the figure, the IO (input and output) path for reading and writing files by the s3fs-based NFS cloud storage gateway is:

[0047] The IO process of NFS cloud storage gateway accessing files: First, the user accesses the file from the user state. When the request is made to the kernel-state VFS, the corresponding file system will be selected for processing according to the user file system mount type, such as: xfs (a high-performance log file system), ext3 (third extended filesystem), fuse (filesystem in userspace), etc. The NFS cloud storage gateway relies on fuse implementation, so the file request will be processed by the kernel-state fuse-module (for module fusion); the fuse-module calls the corresponding file system interface according to the file request and hands it over to the user-state fuse-daemon (a user-space file system daemon, mainly responsible for responding to the kernel's system calls to the user-space file system) for processing; s3fs is a fuse-daemon implemented by calling the libfuse (an open source library) interface; in s3fs, other requests such as reading and writing the file system are handled through the S3 object storage SDK (Software Development Kit, software development kit), which further realizes the protocol conversion between NFS file system and S3 object storage; file read (reading) requires serial calls to ZOS get (a request method), file write (writing) requires serial calls to ZOS put (used to describe the operation of storing or placing data in an electronic device), and file ls (a command used to list files and subdirectories in a directory) requires calling ZOS list (directory).

[0048] like Figure 1 As shown in the figure, file access of NFS cloud storage gateway requires multiple kernel-user interactions. When reading or writing the same file, only extremely inefficient serial access can be used. And when modifying large files, there will be a lot of read amplification and write amplification because object storage does not support overwrite writes.

[0049] In summary, NFS cloud storage gateway can enable more applications to quickly use object storage without additional development, but NFS cloud storage gateway still has the following problems:

[0050] (1) Since object storage can only access a single object serially, when reading or writing files that are too large, the performance is often much lower than when accessing the local file system. Services that can run normally cannot run normally after using the NFS cloud storage gateway.

[0051] (2) Since object storage does not support overwrite, when a user application overwrites a file, the object storage needs to first load the entire object (referring to the completion of the resource loading and execution process) to the local computer, then modify the corresponding data content and upload the object. The entire writing process is very inefficient. This is especially true when the overwrite content is small but the file is large, which causes great read amplification and write amplification.

[0052] (3) Users search for or access files on the NFS cloud storage gateway through the object storage list function. Due to poor list performance, when the number of files is large, the response time is often long, resulting in abnormal user services.

[0053] To solve the above technical problems, an embodiment of the present invention provides a parallel data access method applied to an NFS cloud storage gateway, which is applied to the NFS cloud storage gateway. The NFS cloud storage gateway is connected to a client and improves the file reading and writing rate by means of file data striping and lock-free concurrent access to object storage. File striping divides a file from one object into multiple objects, solving the problems of overwrite read amplification and write amplification. The poor performance of the object storage list is solved by completely saving the file name information locally in the NFS cloud storage gateway through a radix tree structure, thereby ensuring that user services can be quickly uploaded to the cloud.

[0054] According to an embodiment of the present invention, an embodiment of a parallel data access method applied to an NFS cloud storage gateway is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0055] In this embodiment, a parallel data access method applied to an NFS cloud storage gateway is provided, which can be used for the above-mentioned NFS cloud storage gateway. Figure 2 is a flow chart of a parallel data access method applied to an NFS cloud storage gateway according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0056] Step S201, obtaining a file read / write request sent by a client, and performing striping processing on the file read / write request.

[0057] Specifically, the client and the NFS cloud storage gateway communicate using the NFS protocol. The client issues a file read and write request according to the NFS protocol format and sends the file read and write request to the NFS cloud storage gateway. The NFS cloud storage gateway receives the file read and write request from the client.

[0058] Step S202: Based on the striped file read and write requests, use lock-free queues and multiple threads to concurrently access the object storage.

[0059] Step S203, read and write files in the object cloud storage, obtain the file read and write results, and feed back the file read and write results to the client.

[0060] The present embodiment provides a parallel data access method applied to an NFS cloud storage gateway, which solves the problems of overwrite read and write amplification and write amplification by striping the file read and write requests, and utilizes lock-free queue multi-threaded concurrent access to object storage based on the striped file read and write requests, optimizes file parallel data access, implements concurrent lock-free reading and writing of large files, improves the file read and write rate of the NFS cloud storage gateway, and finally performs file reading and writing on the object cloud storage to obtain file read and write results, which are fed back to the client, thereby improving access efficiency.

[0061] In this embodiment, a parallel data access method applied to an NFS cloud storage gateway is provided, which can be used for the above-mentioned NFS cloud storage gateway. Figure 3 is a flow chart of a parallel data access method applied to an NFS cloud storage gateway according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0062] Step S301, obtaining a file read / write request sent by a client, and performing striping processing on the file read / write request.

[0063] Specifically, the above step S301 includes:

[0064] Step S3011: divide the file read and write request into multiple Pages according to a preset division unit.

[0065] Specifically, Figure 4 As shown, the preset division unit is set to 16MB (megabytes), and the file read and write requests are divided into different Pages. The part of the file read and write request that is less than 16MB exists as an independent Page; wherein the data structure of the Page is as follows:

[0066] Struct Page

[0067] {

[0068] list <slice>slices

[0069] };

[0070] Step S2012, obtain a slice, and use the slice to divide the Page into multiple chunks; wherein the slice corresponds to the Page one to one.

[0071] Specifically, each Page corresponding to a file read or write request is managed by a slice, which records the offset of the Page to the file and the range segments (the length of a segment of data) to which the Page corresponds. The data structure of a slice is as follows:

[0072] Struct Slice

[0073] {

[0074] Page*page;

[0075] list <chunk>chunks;

[0076] uint64 offset,

[0077] uint64 length

[0078] };

[0079] Furthermore, when the writing mode of the file read / write request is sequential writing, the Page corresponds to only one slice; when the writing mode of the file read / write request is random writing, the Page corresponds to multiple slices; when the slice corresponding to the newly written file read / write request intersects with the existing slice, the slices are merged; therefore, if the Page is fully written with 16MB, the Page corresponds to only one slice.

[0080] Furthermore, if Figure 4 As shown in the figure, slice acts as the metadata management layer of Page and manages file read and write requests. Slice divides the file read and write requests corresponding to the page into different chunks. Each chunk is an object used to store data. The maximum size of each chunk is 4MB. When slices overlap, objects (chunks) are merged, which can minimize the metadata that the slice needs to record and read and write amplification.

[0081] For example, if file test writes 10MB of data starting from 0, Page 1 is generated for the file and corresponds to a slice, and the slice manages three objects: chunk1, chunk2, and chunk3. When 2-4MB of data is modified in the file, if the file is not striped, the entire 10MB file needs to be read, and then the corresponding 2-4MB data segment is modified and 10MB is written to overwrite the previous object. After the file is striped, only the first object chunk1 of the first slice corresponding to Page 1 of the file needs to be read, and the corresponding data needs to be modified and then written to chunk1. After the file is striped, 6MB of data is read or written less, and the read and write efficiency is improved by 60%.

[0082] Step S302: Based on the striped file read and write requests, use lock-free queues and multiple threads to concurrently access the object storage.

[0083] Specifically, the above step S302 includes:

[0084] Step S3021, write multiple chunks into the subtask queues in the lock-free queue respectively.

[0085] Specifically, a hash algorithm is used to write multiple chunks into the subtask queue respectively.

[0086] Step S3022: asynchronously sending the file read and write requests in the subtask queue to the object storage through the subtask thread; wherein the subtask thread corresponds to the subtask queue one by one.

[0087] Specifically, Figure 5 As shown in the figure, the lock-free multi-threaded asynchronous concurrent architecture of the NFS cloud storage gateway adopts a multi-producer and multi-consumer mode. The IO path includes two lock-free queues, ChunkRequestWQ and ChunkResponseWQ. The NFS cloud storage gateway opens 10 subtask threads by default.

[0088] Furthermore, since the inherent architecture of s3fs is a single-threaded blocking IO model, each file corresponds to only one object. When multiple ranges of a file are accessed concurrently, the access request will be blocked, resulting in very low file read and write performance. Therefore, Figure 5 As shown in the figure, the file read and write requests are first striped into different Pages, and then divided into different chunks according to the range segment; the chunk is written (put) into the subqueue of ChunkRequestWQ through the hash algorithm, and then the client thread can process other file read and write requests; the subtask thread uses a polling method to pop (call) the corresponding file read and write request from the subtask queue of ChunkRequestWQ and send it asynchronously to the object storage (ZOS).

[0089] Step S303: Read and write files in the object cloud storage, obtain the file reading and writing results, and feed back the file reading and writing results to the client. Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.

[0090] This embodiment provides a parallel data access method for an NFS cloud storage gateway. By dividing a file read and write request into multiple Pages, and using slices to divide the Pages into multiple chunks, the file read and write request is divided from one into multiple ones. When there is an overwrite in the file read and write request, it is not necessary to read all large files from the object storage and then write them, but only to change the corresponding stripe module, which greatly reduces large amplification and write amplification and improves the read and write rate. Secondly, the NFS cloud storage gateway writes multiple chunks into the subtask queue in the lock-free queue, and accesses the object storage concurrently through multiple subtask threads, further improving the read and write efficiency.

[0091] In this embodiment, a parallel data access method applied to an NFS cloud storage gateway is provided, which can be used for the above-mentioned NFS cloud storage gateway. Figure 6 is a flow chart of a parallel data access method applied to an NFS cloud storage gateway according to an embodiment of the present invention. Figure 6 As shown, the process includes the following steps:

[0092] Step S601: Obtain the file read / write request sent by the client, and perform striping on the file read / write request. Figure 3 Step S301 of the illustrated embodiment will not be described in detail here.

[0093] Step S602: Based on the striped file read and write requests, use lock-free queues and multiple threads to concurrently access the object storage. Figure 3 Step S302 of the illustrated embodiment will not be described in detail here.

[0094] Step S603, read and write files in the object cloud storage, obtain the file read and write results, and feed back the file read and write results to the client.

[0095] Specifically, due to the poor list performance of ZOS, when the number of objects reaches tens of millions, it may take hundreds of seconds to list all object information. Therefore, the NFS cloud storage gateway uses a radix tree to save file information locally on the gateway.

[0096] Specifically, the above step S603 includes:

[0097] Step S6031, determining the file name length based on the file read and write request.

[0098] Step S6032, based on the file name length, use the radix tree to read and write the file to obtain the file reading and writing result.

[0099] Specifically, based on the length of the file name, the storage location of the file is queried using the radix tree, and the file is read and written based on the storage location of the file to obtain the file reading and writing result.

[0100] Furthermore, the radix tree structure is based on the dictionary tree. The indexing efficiency of the radix tree has nothing to do with the depth of the tree and the number of objects, but is only related to the length of the file name, that is, the indexing efficiency is O(k), where k is the length of the file name; among them, the topology of the radix tree storing file information is consistent with the actual topology of the file, so when files are created and deleted, the impact on the query efficiency of the entire radix tree is almost zero.

[0101] Furthermore, if Figure 7 As shown in the figure, there are three subdirectories in the file system / mnt (i.e., / test1, / test2, and / test3), and different files are distributed under each subdirectory (i.e., / a, / b, / a1, / a2, and / a3). If the file name is / mnt / test1 / a, the file name length is 9, and the index efficiency is O(9), that is, the server CPU calculation speed will not exceed 9us. If the query is performed through the object storage list, it will take more than 20ms, and the overall file retrieval efficiency has been improved by nearly a million times.

[0102] Furthermore, when processing a file read request, the file storage location is queried using the radix tree based on the file name length of the file to be read in the file read request, and then the file corresponding to the file storage location is read; when processing a file write request, the file to be written is written to the corresponding position of the radix tree for storage based on the file name length of the file to be written in the file read request.

[0103] Step S6033, write the file read and write processing results into the lock-free queue, and use the file thread to feed back the file read and write processing results in the lock-free queue to the client.

[0104] Specifically, the file reading and writing results are put into ChunkResponseWQ, and then the file thread uses polling to feed back the file reading and writing results to the client.

[0105] This embodiment provides a parallel data access method applied to an NFS cloud storage gateway. Based on the file name length, a radix tree is used to read and write files to obtain file reading and writing results. The file information is completely saved locally on the NFS cloud storage gateway through a radix tree structure, which solves the problem of poor object storage list performance, ensures that client services are quickly uploaded to the cloud, and greatly improves file query efficiency.

[0106] The following is a specific example to illustrate the specific steps of parallel data access applied to NFS cloud storage gateway.

[0107] Embodiment 1:

[0108] A specific step of parallel data access applied to an NFS cloud storage gateway includes:

[0109] First, the NFS cloud storage gateway file introduces the logical concept of Page, which divides a file into different pages according to the size of 16MB. The part less than 16MB exists independently as a Page, which can be quickly retrieved when accessing different range segments of a file. Each Page of the file is managed by a slice, which records the offset of the Page to the file and which ranges the data under the Page is written to; when the file is written in sequential mode, the Page corresponds to only one slice; when the file is written in random mode, the Page corresponds to multiple slices; when the new write and the existing slices intersect, the slices are merged, and finally if the Page is filled with 16MB, there will only be one slice; slices, as the metadata management layer of the Page, manage file data. Slices divide files into different chunks, each chunk is an object for storing data, and each chunk is up to 4MB. When slices intersect, objects are merged, which can minimize the metadata that the slice needs to record and minimize read-write amplification.

[0110] The inherent architecture of s3fs is a single-threaded blocking IO model, where each file corresponds to only one object. When multiple different ranges of a file are accessed concurrently, the access request will be blocked, resulting in very low file read and write performance. Therefore, the file read and write rate is improved through the IO model of lock-free multi-threaded concurrent access to object storage. The IO architecture of NFS cloud storage gateway adopts a multi-producer multi-consumer mode, and the IO path includes two lock-free queues, ChunkRequestWQ and ChunkResponseWQ.

[0111] The user's read and write requests for files will first be striped into different Pages, and then divided into different chunks according to the range; the chunks are put into the subqueue of ChunkRequestWQ through the hash algorithm, where the NFS cloud storage gateway opens 10 subtask threads by default. The subtask threads use polling to pop the corresponding file read and write requests from the subtask queue of ChunkRequestWQ and send them asynchronously to the object storage ZOS; the processed file read and write requests are put into ChunkResponseWQ, and the file thread will poll and feedback the file read and write processing results to the user. Relying on file striping to divide files into multiple Pages and chunks, using lock-free queues and multi-threaded concurrent processing methods, when users read different ranges of files concurrently, the NFS cloud storage gateway can split the file read and write requests into multiple chunks and write them losslessly into ChunkRequestWQ, and access the object storage concurrently through multiple subtask threads, thereby further improving the read and write efficiency.

[0112] Since the list performance of the object storage ZOS is relatively poor, when the number of objects reaches tens of millions, it may take hundreds of seconds to list all object information. Therefore, the NFS cloud storage gateway uses a radix tree to save file information locally on the gateway. The radix tree structure is based on a dictionary tree, and its indexing efficiency has nothing to do with the depth of the tree and the number of objects, but only with the length of the file name. Since the topology of the radix tree to store file information is consistent with the actual topology of the file, the impact of file creation and deletion on the query efficiency of the entire radix tree is almost zero.

[0113] To optimize the serious read and write amplification problems of object storage when large files are overwritten on the NFS cloud storage gateway, file striping is used to reduce read and write amplification by an average of 48%; large file read and write amplification is reduced by more than 78%; to address the low read and write performance of NFS cloud storage gateway files, the lock-free queue multi-threaded IO architecture is optimized, and file striping can be used to concurrently access object storage, improving read and write performance by more than 21.2%; to address the low efficiency of NFS cloud storage gateway file queries, a radix tree is used to persist file information locally. When the number of files reaches tens of millions, the average delay in querying files can be reduced by more than 56%, and the average query delay is 161ms.

[0114] In the above-mentioned embodiment 1, through the algorithm of striping file data and concurrent access to object storage, the concurrent lock-free reading and writing of large files is realized by relying on lock-free queue multithreading, thereby improving the file reading and writing rate, and solving the problem that the object storage can only access a single object serially, and the performance is low when reading or writing large files, which causes the service to not run normally when using the NFS cloud storage gateway; in addition, the file data striping divides the file object from one into multiple ones. When the file is overwritten, it is not necessary to read the entire large file from the object storage and then write it, but only to change the corresponding stripe module, which greatly reduces the large amplification and write amplification, and further improves the reading and writing rate; in addition, relying on All files in the file system are persisted locally in the form of a tree based on the radix tree data structure. When users search or access files on the NFS cloud storage gateway again, they do not need to use the object storage list, but instead search the local radix tree for files. Therefore, the file read and write rates are improved through file data striping and lock-free concurrent access to object storage. File striping divides a file from one object into multiple objects, solving the problems of overwrite read amplification and write amplification. The poor performance of the object storage list is solved by completely saving the file name information locally on the NFS cloud storage gateway through the radix tree structure, ensuring that client services can be quickly migrated to the cloud.

[0115] In this embodiment, a parallel data access device applied to an NFS cloud storage gateway is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0116] This embodiment provides a parallel data access device applied to an NFS cloud storage gateway, such as Figure 8 As shown, including:

[0117] The striping processing module 801 is used to obtain the file read and write request sent by the client and perform striping processing on the file read and write request;

[0118] A parallel access module 802 is used to concurrently access object storage using lock-free queue multi-thread based on the striped file read and write requests;

[0119] The read / write module 803 is used to read and write files in the object cloud storage, obtain the file read / write results, and feed back the file read / write results to the client.

[0120] In some optional implementations, the striping processing module 801 includes:

[0121] A first division unit, used to divide the file read and write request into multiple Pages according to a preset division unit;

[0122] The second partitioning unit is used to obtain a slice, and use the slice to partition the Page into multiple chunks; wherein the slice corresponds to the Page one by one.

[0123] In some optional implementations, the parallel access module 802 includes:

[0124] The writing unit is used to write multiple chunks into the subtask queues in the lock-free queue respectively;

[0125] The sending unit is used to asynchronously send the file read and write requests in the subtask queue to the object storage through the subtask thread; wherein the subtask thread corresponds to the subtask queue one by one.

[0126] In some optional implementations, the writing unit is specifically configured to write the multiple chunks into the subtask queues respectively by using a hash algorithm.

[0127] In some optional implementations, the read-write module 803 includes:

[0128] A determination unit, used to determine the length of a file name based on a file read and write request;

[0129] The read-write unit is used to read and write files based on the length of the file name using the radix tree to obtain the file reading and writing results;

[0130] The feedback unit is used to write the file reading and writing processing results into the lock-free queue, and use the file thread to feed back the file reading and writing processing results in the lock-free queue to the client.

[0131] In some optional implementations, the read-write unit is specifically used to query the storage location of the file based on the file name length using a radix tree, and read and write the file based on the storage location of the file to obtain the file read-write result.

[0132] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0133] In this embodiment, a parallel data access device applied to an NFS cloud storage gateway is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0134] The embodiment of the present invention also provides a computer device having the above Figure 8 A parallel data access device applied to an NFS cloud storage gateway is shown.

[0135] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig. 9 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.

[0136] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0137] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0138] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0140] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0141] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0142] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0143] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0144] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.< / chunk> < / slice>

Claims

1. A parallel data access method applied to an NFS cloud storage gateway, characterized in that: The method comprises: Obtaining a file read / write request sent by a client, and performing striping processing on the file read / write request; Based on the striped file read and write requests, the lock-free queue multi-threaded concurrent access to the object storage is used; The object cloud storage performs file reading and writing, obtains file reading and writing results, and feeds back the file reading and writing results to the client.

2. The method according to claim 1, characterized in that: The striping process of the file read and write request includes: Divide the file read and write request into multiple Pages according to a preset division unit; A slice is obtained, and the Page is divided into a plurality of chunks using the slice; wherein the slice corresponds to the Page one by one.

3. The method according to claim 2, characterized in that The method of concurrently accessing object storage using multi-threaded lock-free queues based on the striped file read and write requests includes: Writing the multiple chunks into the subtask queues in the lock-free queue respectively; The file read and write requests in the subtask queue are asynchronously sent to the object storage through the subtask thread; wherein the subtask thread corresponds to the subtask queue one by one.

4. The method according to claim 3, characterized in that The step of writing the plurality of chunks into the subtask queues in the lock-free queue respectively includes: The plurality of chunks are written into the subtask queue respectively by using a hash algorithm.

5. The method according to claim 1, characterized in that The performing file reading and writing on the object cloud storage, obtaining the file reading and writing result, and feeding back the file reading and writing result to the client includes: Determine the length of the file name based on the file read and write request; Based on the length of the file name, the file is read and written using a radix tree to obtain the file reading and writing result; The file read and write processing result is written into the lock-free queue, and the file read and write processing result in the lock-free queue is fed back to the client by using a file thread.

6. The method according to claim 5, characterized in that The method of performing file reading and writing based on the file name length by using a radix tree to obtain the file reading and writing result includes: Based on the length of the file name, the storage location of the file is queried using the radix tree, and the file is read and written based on the storage location of the file to obtain the file reading and writing result.

7. A parallel data access device applied to an NFS cloud storage gateway, characterized in that: The device comprises: The striping processing module is used to obtain the file read and write request sent by the client and perform striping processing on the file read and write request; Parallel access module, which is used to concurrently access object storage using lock-free queues and multiple threads based on striped file read and write requests; The read / write module is used to read and write files in the object cloud storage, obtain file read / write results, and feed back the file read / write results to the client.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the parallel data access method applied to the NFS cloud storage gateway according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the parallel data access method applied to an NFS cloud storage gateway according to any one of claims 1 to 6.

10. A computer program product, characterized in that It includes computer instructions, and the computer instructions are used to enable a computer to execute the parallel data access method applied to an NFS cloud storage gateway as described in any one of claims 1 to 6.

Images