A big data storage and access method based on distributed storage system

By directly using the backend storage cluster and asynchronously processing file system tasks within the Hadoop file system, the performance bottleneck of the Hadoop file system is resolved, file system read and write operations are optimized, and the efficiency of file enumeration and renaming is improved.

CN117851369BActive Publication Date: 2024-11-26CHINA TELECOM CLOUD TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202311722314.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-14
Publication Date
2024-11-26
Estimated Expiration
2043-12-14

AI Technical Summary

Technical Problem

Hadoop file system clients cannot directly send read and write requests to the storage cluster, resulting in the existence of an object storage gateway along the entire IO path. This increases the operational cost and leads to poor performance of Hadoop file systems based on object storage in operations such as file enumeration and file renaming.

Method used

A big data storage access method based on a distributed storage system is adopted, which directly uses the data storage capacity of the backend storage cluster through the HDFS client, avoiding the need for an object storage gateway, asynchronously processing file system operation tasks, supporting file IO splitting and chunk management, and optimizing the lease mechanism.

Benefits of technology

It improves file system performance, reduces the risk of object storage gateways becoming performance bottlenecks, simplifies error handling and parallel write operations, and enhances the efficiency of file enumeration and renaming.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117851369B_ABST
    Figure CN117851369B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of cloud storage technology, and in particular to a big data storage access method based on a distributed storage system. Aiming at the problem that an existing Hadoop file system client cannot directly send a read / write request to a storage cluster, and the cost of object storage List Objects and Rename is relatively high, resulting in the problem that the Hadoop file system based on object storage has worse performance than HDFS in operations such as file listing and file renaming, the following scheme is proposed. The big data storage access method comprises the following steps: Σ1, HDFS client instance initialization; Σ2, file system creation; Σ3, file system mounting; Σ4, directory creation; Σ5, file creation, which can shorten the IO path, make full use of the distributed cluster capability, directly seal the chunk if data writing fails, and reselect the chunk to continue writing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of cloud storage technology, and in particular to a big data storage access method based on a distributed storage system. Background Art

[0002] In the existing big data scenario, the big data system based on the big data analysis and computing process includes: the big data computing platform layer, which covers hardware, cloud computing technology and software systems; the big data storage and management layer, which includes data collection and data preprocessing; the big data analysis and computing layer, which covers data mining, Hadoop, etc.; the big data visualization layer, which interactively visualizes the analysis results based on the big data analysis and computing layer.

[0003] Hadoop is a typical open source software for big data processing. The Hadoop cluster runs on cheap machines, forming a complete ecosystem and has become an important platform for big data processing. Hadoop includes the concept of abstract file system and defines the Hadoop abstract file system client interface. The specific implementation of this interface corresponds to the local disk file system, HDFS distributed file system, file system built on other file systems for file archiving, file system supported by FTP server, file system supported by AmazonΣ3, file system supported by Microsoft Azure, file system supported by OpenStack Swift and other file systems.

[0004] HDFS is a distributed file system and one of the core components of the Hadoop ecosystem. Similar to the file system of an operating system, HDFS provides storage and management of massive data and has the advantages of high fault tolerance and high throughput. HDFS is stored across multiple computers and stores very large files in a streaming data access mode. HDFS is a distributed system with a master / slave architecture. The HDFS cluster contains at least one name node (NameNode), one second name node (SecondaryNameNode) and one data node (DataNode). The name node (NameNode) and the data node (DataNode) provide the ability to store and manage data. Users use this ability to interact with the data node through the HDFS client (Client) to access the file system.

[0005] The HDFS client implements the Hadoop abstract file system client interface. Users can initiate requests to read and write HDFS files to the HDFS cluster through the HDFS client, and can also perform file system-like namespace operations through the HDFS client, such as opening, closing, and renaming files or directories.

[0006] The industry has implemented a standard Hadoop file system based on object storage, which can provide support for the integration of object storage for big data computing frameworks such as Hadoop, Spark, and Tez, allowing them to read and write data stored in the object storage by sending Σ3 messages to the object storage in the same way as accessing the HDFS file system.

[0007] In the case of computing and storage separation, in private cloud scenarios, virtualization technology is generally used to create multiple computing clusters to support the computing needs of upper-level big data applications. The most famous s3 client available in the Hadoop ecosystem is S3A, which is applicable to all other S3 backends. Generally speaking, storage uses object storage services to provide a shared storage backend for the data lake, and then provides a connection between the two through the S3A plug-in, allowing Hadoop applications to seamlessly access the Ceph object storage service.

[0008] The HDFS client initiates requests to read, write, create, delete, and rename HDFS files to the HDFS cluster, using the data storage and management capabilities of the HDFS cluster. First, the design of HDFS itself integrates computing resources and storage resources, using one cluster to meet different business needs. Later, as the business volume grows rapidly, different businesses have different computing requirements for storage, and there will be competition for computing resources; when expanding capacity in the later stage, computing and storage cannot be decoupled and expanded, and there is a waste of computing resources or storage resources. Secondly, the HDFS client needs to establish a data flow pipeline based on the location information to read and write files. If an error occurs in a data node of the data block copy, the error handling will be very complicated. It is necessary to apply for a new data node, assign a new timestamp to the data block, and use the new timestamp to establish a new data flow pipeline, etc., which increases the delay of read and write processing. In addition, it does not support multiple HDFS clients to write in parallel, and a lease mechanism is required to ensure mutually exclusive writes to HDFS files. The lease manager needs to maintain a thread to regularly check whether the lease has expired, and the lease check thread also consumes system resources.

[0009] S3A uses object storage solutions and the industry's object storage-based Hadoop file system. The general process is: when uploading data, the Hadoop file system client needs to encapsulate the request into HTTP and then send it to the object storage gateway. The object storage gateway then disassembles it and converts it into an internal request and sends it to the storage cluster. The data is uploaded to the object storage in the way of writing files in the Hadoop file system. For example, Figure 1As shown in the figure, taking Ceph object storage as an example, how data is uploaded. Because both the read and write paths of data need to pass through the object storage gateway, and then the object storage gateway forwards the request to the storage cluster, it is obvious that the object storage gateway becomes a performance bottleneck.

[0010] The Hadoop file system client cannot directly send read and write requests to the storage cluster. The object storage gateway always exists in the entire IO path. Since the cost of object storage List Objects and Rename is relatively high, the Hadoop file system based on object storage will have worse performance than HDFS in operations such as file listing and file renaming. Summary of the invention

[0011] The purpose of the present invention is to solve the problem in the prior art that the Hadoop file system client cannot directly send read and write requests to the storage cluster, and there is always an object storage gateway in the entire IO path. Since the cost of object storage List Objects and Rename is relatively high, the Hadoop file system based on object storage will have worse performance than HDFS in operations such as file listing and file renaming. A big data storage and access method based on a distributed storage system is proposed.

[0012] In order to achieve the above object, the present invention adopts the following technical solutions:

[0013] A big data storage access method based on a distributed storage system, the big data storage access method comprising the following steps:

[0014] Σ1. Initialize the HDFS client instance;

[0015] Σ2. Create a file system;

[0016] Σ3. Mount the file system;

[0017] Σ4. Create a directory;

[0018] Σ5. Create a file;

[0019] Σ6. Open the file with read and write permissions;

[0020] Σ7. Write file data;

[0021] Σ8. Read data from the file;

[0022] Σ9. Close the file;

[0023] Σ10. Delete files;

[0024] Σ11. Delete directory;

[0025] Σ12. Unmount the file system;

[0026] Σ13. Delete the file system.

[0027] Preferably, the step Σ2 includes the following sub-steps:

[0028] Σ21. Generate a task to create a file system in the business thread, put it into the task queue for the polling thread to process, and block and wait for the task processing to end;

[0029] Σ22. The polling thread polls the task, processes the task, and the network sends a request to the file system metadata server using an asynchronous network communication method;

[0030] Σ23. After the request is returned, a task is generated and stored in the class file system guest queue;

[0031] Σ24. The polling thread polls the task and notifies the business thread that the file system has been successfully created. The file system creation process ends.

[0032] Σ25. The business thread returns the result of creating the file system to the upper-layer business.

[0033] Preferably, the step Σ7 further includes the following sub-steps:

[0034] Σ71. Check whether the client lease is invalid and whether the file is unlocked. If the file lock is not lost, go to step Σ72. If it is invalid or the file lock is lost, go to step Σ75.

[0035] Σ72. Get the chunks involved in this write. If there are enough chunks, go to step Σ73. If there are not enough chunks, go to step Σ76.

[0036] Σ73. Send a message to the data storage nodes where all copies of the chunk are located to write data. If the writing is successful, go to step Σ74. If the writing fails, go to step Σ77.

[0037] Σ74. Notify the business thread that data writing is complete and the data writing process ends;

[0038] Σ75. Reopen the file, check the file length, and check whether the writing is successful. If successful, go to step Σ74; otherwise, go to step Σ71.

[0039] Σ76. Apply for chunk. If the application is successful, go to step Σ72. If the returned file is locked by another client, go to step Σ74. If the returned file lock is lost, go to step Σ75. If other errors are returned and the number does not exceed 3 times, go to step Σ76. If other errors are returned and the number exceeds 3 times, go to step Σ74.

[0040] Σ77. Seal the chunk. If the return is successful, go to step Σ71. If the returned file is locked by another client, go to step Σ7. If the returned file lock is lost, go to step Σ75. If other errors are returned, continue to retry and go to step Σ77.

[0041] Preferably, the sub-step Σ22 further comprises the step of:

[0042] Σ221, the client polling thread polls the task, calls the processing function to process the task, calls the previously bound processing function, queries whether the storage pool exists, and if so, creates a file system in the storage pool, uses the asynchronous method of the network communication framework to send a request to the file system metadata server, and registers the callback function before that. After the request returns, the callback function is called to generate a task, the bound processing function, and the task is stored in the queue of the file system client.

[0043] Preferably, the client polling thread polls the task, and in the task processing function, it will eventually activate the conditional variable in the file system creation context, and the file system creation process will end. The file system will be deleted, the file system will be mounted, the file system will be unmounted, the file will be created, and the file storage system access operation process will be deleted. The task is also generated and stored in the task queue, and the polling thread processes the task to execute the task processing function.

[0044] Preferably, the step Σ1 includes the following specific implementation steps:

[0045] Σ101. Apply for client ID and lease time;

[0046] Σ102, cyclically check the task queue and process the tasks;

[0047] Σ103, loop to check whether the lease is invalid, if invalid, go to step Σ104; otherwise, go to step Σ105;

[0048] Σ104. Set the file status to file unlocked;

[0049] Σ105, loop to check whether the lease needs to be renewed, if it needs to be renewed, go to step Σ106, otherwise, go to step Σ102;

[0050] Σ106. Renew the lease and proceed to step Σ102.

[0051] Preferably, the step Σ6 further comprises the following steps:

[0052] Σ61, when opening a file, if it contains write permission, a lease will be applied for from the metadata server. Successful opening means that the write permission of the file within the lease period has been successfully obtained. When writing a file, it will first check whether the client lease has expired. If it has expired, the file status is set to file unlocked;

[0053] During the writing process, the file lease lock may be lost or occupied by other clients. In addition, during the writing process, when applying for or sealing (indicating that data can no longer be written) a chunk, a request will be sent to the file system metadata server. At this time, the file lease lock may be returned as occupied or lost, and the file status will be set to locked or unlocked. If the file is unlocked, the file will be reopened. After opening, the length of the file will be used to check whether the write is successful.

[0054] Preferably, the step Σ6 further comprises the following steps:

[0055] Σ62, when opening a file, the information of the chunks of data that have been written to the file will be obtained from the file system server. The permission to open the file includes write permission, so the chunk will be pre-applied for. When reading the file, the specific chunks to be read will be determined based on the offset and length. Each chunk has multiple strongly consistent copies, and messages are sent to the data storage nodes where the copies are located to read the data. Once the data is read, the next copy will no longer be read.

[0056] Preferably, the further steps of step Σ62 include:

[0057] Σ621, when writing a file, the specific chunks to be written will be determined based on the offset and length. If there are not enough chunks, the backend storage cluster will be requested for chunks. If a failure is returned, the server will retry four times after a certain period of time. After a successful return, the HDFS client will send the requested chunk information to the file system metadata server. If a failure is returned, the server will retry three times.

[0058] Preferably, the further step of step Σ62 includes:

[0059] Σ622, send a message to the data storage nodes where all copies of χηυνκ are located to write data. Failure to write to a copy means failure to write the chunk. Seal the chunk: send a seal request to the backend storage cluster without paying attention to whether it is successful. Send a seal request to the file system metadata server. After returning failure, continue to retry until success. Finally, reselect the chunk to continue writing.

[0060] Beneficial Effects

[0061] 1. When using the object storage solution, the path length is increased by 10, and the object storage gateway will become a performance bottleneck. This patent targets this situation. The HDFS client advocated in the present invention does not need to go through the object storage gateway, but directly uses the data storage capacity of the backend storage cluster.

[0062] 2. When the HDFS client needs to establish a data flow pipeline based on location information when reading and writing files, error handling will be very complicated, increasing the delay of reading and writing processing. In this case, the HDFS client will split the file 10 into chunks 10 and track them. If a copy of the chunk fails to be written, it will be directly sealed and a new chunk will be selected to continue writing.

[0063] 3. When the industry HDFS client (Client) file and directory operations directly access the file system metadata server, synchronously wait for the processing to complete, the request processing throughput is limited, and the Hadoop file system based on object storage has poor performance in file listing, file renaming and other operations. In view of these situations, this patent generates tasks when processing upper-level business requests, and tasks will be generated after the asynchronous request of the network communication framework is returned. These tasks will be stored in the queue and wait for the polling thread to process. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a schematic diagram of data upload to Hadoop file system based on Ceph object storage in the present invention;

[0065] Figure 2 This is a schematic diagram of the present invention showing that the HDFS Client accesses the backend storage cluster;

[0066] Figure 3 A schematic diagram of file system request polling processing according to the present invention;

[0067] Figure 4 It is a schematic diagram of the lease processing of the present invention;

[0068] Figure 5 Schematic diagram of the lease expiration process of the present invention

[0069] Figure 6 Schematic diagram of data writing exception processing of the present invention

[0070] Figure 7 The basic flow chart of the present invention is

[0071] Figure 8 Schematic diagram of the initialization process of the present invention

[0072] Fig. 9 Schematic diagram of creating a file system flow for the present invention

[0073] Fig.10 A schematic diagram of the document writing process of the present invention. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0075] Embodiment 1

[0076] Reference Figure 1-10 , a big data storage access method based on a distributed storage system, the big data storage access method comprising the following steps:

[0077] Σ1. Initialize the HDFS client instance;

[0078] Σ2. Create a file system;

[0079] Σ3. Mount the file system;

[0080] Σ4. Create a directory;

[0081] Σ5. Create a file;

[0082] Σ6. Open the file with read and write permissions;

[0083] Σ7. Write file data;

[0084] Σ8. Read data from the file;

[0085] Σ9. Close the file;

[0086] Σ10. Delete files;

[0087] Σ11. Delete directory;

[0088] Σ12. Unmount the file system;

[0089] Σ13. Delete the file system.

[0090] The step Σ1 includes the following specific implementation steps:

[0091] Σ101. Apply for client ID and lease time;

[0092] Σ102, cyclically check the task queue and process the tasks;

[0093] Σ103, loop to check whether the lease is invalid, if invalid, go to step Σ104; otherwise, go to step Σ105;

[0094] Σ104. Set the file status to file unlocked;

[0095] Σ105, loop to check whether the lease needs to be renewed, if it needs to be renewed, go to step Σ106, otherwise, go to step Σ102;

[0096] Σ106. Renew the lease and proceed to step Σ102.

[0097] The step Σ2 includes the following sub-steps:

[0098] Σ21. Generate a task to create a file system in the business thread, put it into the task queue for the polling thread to process, and block and wait for the task processing to end;

[0099] Σ22. The polling thread polls the task, processes the task, and the network sends a request to the file system metadata server using an asynchronous network communication method;

[0100] Σ23. After the request is returned, a task is generated and stored in the class file system guest queue;

[0101] Σ24. The polling thread polls the task and notifies the business thread that the file system has been successfully created. The file system creation process ends.

[0102] Σ25. The business thread returns the result of creating the file system to the upper-layer business.

[0103] The sub-step Σ22 further comprises the following steps:

[0104] Σ221, the client polling thread polls the task, calls the processing function to process the task, calls the previously bound processing function, queries whether the storage pool exists, and if so, creates a file system in the storage pool, uses the asynchronous method of the network communication framework to send a request to the file system metadata server, and registers the callback function before that. After the request returns, the callback function is called to generate a task, the bound processing function, and the task is stored in the queue of the file system client.

[0105] The step Σ7 also includes the following sub-steps:

[0106] Σ71. Check whether the client lease is invalid and whether the file is unlocked. If the file lock is not lost, go to step Σ72. If it is invalid or the file lock is lost, go to step Σ75.

[0107] Σ72. Get the chunks involved in this write. If there are enough chunks, go to step Σ73. If there are not enough chunks, go to step Σ76.

[0108] Σ73. Send a message to the data storage nodes where all copies of the chunk are located to write the data. If the write is successful, go to step Σ74. If the write fails,

[0109] Go to step Σ77;

[0110] Σ74. Notify the business thread that data writing is complete and the data writing process ends;

[0111] Σ75. Reopen the file, check the file length, and check whether the writing is successful. If successful, go to step Σ74; otherwise, go to step Σ71.

[0112] Σ76. Apply for chunk. If the application is successful, go to step Σ72. If the returned file is locked by another client, go to step Σ74. If the returned file lock is lost, go to step Σ75. If other errors are returned and the number does not exceed 3 times, go to step Σ76. If other errors are returned and the number exceeds 3 times, go to step Σ74.

[0113] Σ77. Seal the chunk. If the return is successful, go to step Σ71. If the returned file is locked by another client, go to step Σ7. If the returned file lock is lost, go to step Σ75. If other errors are returned, continue to retry and go to step Σ77.

[0114] The client polling thread polls the task, and in the task processing function, it will eventually activate the conditional variable in the file system creation context, and the file system creation process ends. The file system is deleted, the file system is mounted, the file system is unmounted, the file is created, and the file big data storage system access operation process also generates a task and stores it in the task queue. The polling thread processes the task and executes the task processing function.

[0115] The step Σ6 further comprises the following steps:

[0116] Σ61, when opening a file, if it contains write permission, a lease will be applied for from the metadata server. Successful opening means that the write permission of the file within the lease period has been successfully obtained. When writing a file, it will first check whether the client lease has expired. If it has expired, the file status is set to file unlocked;

[0117] During the writing process, the file lease lock may be lost or occupied by other clients. In addition, during the writing process, when applying for or sealing (indicating that data can no longer be written) a chunk, a request will be sent to the file system metadata server. At this time, the file lease lock may be returned as occupied or lost, and the file status will be set to locked or unlocked. If the file is unlocked, the file will be reopened. After opening, the length of the file will be used to check whether the write is successful.

[0118] The step Σ6 further comprises the following steps:

[0119] Σ62, when opening a file, the information of the chunks of data that have been written to the file will be obtained from the file system server. The permission to open the file includes write permission, so the chunk will be pre-applied for. When reading the file, the specific chunks to be read will be determined based on the offset and length. Each chunk has multiple strongly consistent copies, and messages are sent to the data storage nodes where the copies are located to read the data. Once the data is read, the next copy will no longer be read.

[0120] The further steps of step Σ62 include:

[0121] Σ621, when writing a file, the specific chunks to be written will be determined based on the offset and length. If there are not enough chunks, the backend storage cluster will be requested for chunks. If a failure is returned, the server will retry four times after a certain period of time. After a successful return, the HDFS client will send the requested chunk information to the file system metadata server. If a failure is returned, the server will retry three times.

[0122] Another step of step Σ62 includes:

[0123] Σ622, send a message to the data storage nodes where all copies of the chunk are located to write data. Failure to write to a copy means failure to write the chunk. Seal the chunk: send a seal request to the backend storage cluster without paying attention to whether it is successful. Send a seal request to the file system metadata server. After returning failure, continue to retry until success. Finally, reselect the chunk to continue writing.

[0124] Embodiment 2

[0125] Improvement based on Example 1: In this patent, the following situations are addressed: computing resources and storage resources are integrated in big data storage, and computing and storage cannot be decoupled to expand services, which will result in competition for computing resources and waste of computing resources or storage resources. When using an object storage solution, the data reading and writing paths of the Hadoop file system client need to pass through the object storage gateway, which increases the path length by 10, and the object storage gateway will become a performance bottleneck.

[0126] The following optimization method is adopted: the HDFS client advocated in the present invention supports reading and appending large files without going through the object storage gateway or using the data storage capacity of the object storage. The backend storage cluster capacity is directly used. The HDFS client splits the file 10 into Chunk 10 requests of the data storage node, sends Chunk 10 to different data storage nodes, and tracks them.

[0127] In this patent, the following situations are addressed: the HDFS client (Client) needs to establish a data flow pipeline based on location information when reading and writing files. If an error occurs in a data node of a data block copy, it is necessary to apply for a new data node, assign a new timestamp to the data block, and use the new timestamp to establish a new data flow pipeline, etc. Error handling will be very complicated, increasing the delay in reading and writing processing.

[0128] The following optimization method is adopted: the HDFS client advocated by this patent will cache file metadata, and will obtain information about the contained chunks when opening a file. After reading data from one of the copies, the next copy will no longer be read. When writing a file, the file will be split and tracked. If a copy of a chunk fails to be written, it will be directly sealed and a new chunk will be selected to continue writing.

[0129] This patent addresses the following situations: HDFS client (Client) file and directory operations require direct access to the file system metadata server, and synchronously wait for the file system metadata server to complete processing. The number of requests that can be processed within a certain period of time is limited. Since the cost of object storage List Objects and Rename is relatively high, the Hadoop file system based on object storage has poor performance in operations such as file listing and file renaming.

[0130] The following optimization method is adopted: the HDFS client claimed in this patent processes the upper-layer business request, generates a file system operation task, stores the task in the queue, waits for the polling thread to process it, the polling thread polls the task, calls the processing function to process the task, uses the asynchronous method of the network communication framework to send a request to the file system metadata server, generates a task after the request returns, stores the task in the queue, and waits for the polling thread to process it.

[0131] In this patent, the following situations are addressed: supporting multiple HDFS clients to write in parallel, requiring a lease mechanism to ensure mutually exclusive writing of HDFS files, and the lease manager needs to maintain a thread to regularly check whether the lease has expired. The lease check thread also consumes system resources.

[0132] The following optimization methods are adopted: the polling thread will cyclically check whether the lease is expired, and will process the lease update: determine whether the lease needs to be updated. If the lease needs to be updated, send a message to the metadata server to update the lease. Opening the file with write permission will obtain the write permission within the lease period. When writing the file, the lease is expired, the file status is set to file unlocked, and the file will be reopened to continue writing.

[0133] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A big data storage access method based on a distributed storage system, characterized in that: The big data storage access method comprises the following steps: Σ1. Initialize the HDFS client instance; Σ2. Create a file system; Σ3. Mount the file system; Σ4. Create a directory; Σ5. Create a file; Σ6. Open the file with read and write permissions; Σ7. Write file data; Σ8. Read data from the file; Σ9. Close the file; Σ10. Delete files; Σ11. Delete directory; Σ12. Unmount the file system; Σ13. Delete the file system; The step Σ2 includes the following sub-steps: Σ21. Generate a task to create a file system in the business thread, put it into the task queue for the polling thread to process, and block and wait for the task processing to end; Σ22. The polling thread polls the task, processes the task, and the network sends a request to the file system metadata server using an asynchronous network communication method; Σ23. After the request is returned, a task is generated and stored in the class file system guest queue; Σ24. The polling thread polls the task and notifies the business thread that the file system has been successfully created. The file system creation process ends. Σ25. The business thread returns the result of creating a file system to the upper-layer business; The client polling thread polls the task, and in the task processing function, the conditional variable in the file system creation context is finally activated, the file system creation process ends, the file system is deleted, the file system is mounted, the file system is unmounted, the file is created, and the file is deleted. The big data storage system access operation process also generates a task and stores it in the task queue, and the polling thread processes the task to execute the task processing function; The sub-step Σ22 further comprises the following steps: Σ221, the client polling thread polls the task, calls the processing function to process the task, calls the previously bound processing function, queries whether the storage pool exists, and if so, creates a file system in the storage pool, uses the asynchronous method of the network communication framework to send a request to the file system metadata server, and registers a callback function before that. After the request returns, the callback function is called to generate a task, the bound processing function, and the task is stored in the queue of the file system client; The step Σ1 includes the following specific implementation steps: Σ101. Apply for client ID and lease time; Σ102, cyclically check the task queue and process the tasks; Σ103, loop to check whether the lease is invalid, if invalid, go to step Σ104; otherwise, go to step Σ105; Σ104. Set the file status to file unlocked; Σ105, loop to check whether the lease needs to be renewed, if it needs to be renewed, go to step Σ106, otherwise, go to step Σ102; Σ106, renew the lease and go to step Σ102; The step Σ6 further comprises the following steps: Σ61, when opening a file, if it contains write permission, a lease will be applied for from the metadata server. Successful opening means that the write permission of the file within the lease period has been successfully obtained. When writing a file, it will first check whether the client lease has expired. If it has expired, the file status is set to file unlocked; During the writing process, the file lease lock may be lost or occupied by another client. In addition, during the writing process, when applying for or sealing a chunk, a request will be sent to the file system metadata server. At this time, the file lease lock may be returned as occupied or lost, and the file status may be set to locked or unlocked. If the file is unlocked, the file will be reopened. After opening, the length of the file will be used to check whether the writing is successful. The step Σ7 also includes the following sub-steps: Σ71. Check whether the client lease is invalid and whether the file is unlocked. If the file lock is not lost, go to step Σ72. If it is invalid or the file lock is lost, go to step Σ75. Σ72. Get the chunks involved in this write. If there are enough chunks, go to step Σ73. If there are not enough chunks, go to step Σ76. Σ73. Send a message to the data storage nodes where all copies of the chunk are located to write data. If the writing is successful, go to step Σ74. If the writing fails, go to step Σ77. Σ74. Notify the business thread that data writing is complete and the data writing process ends; Σ75. Reopen the file, check the file length, and check whether the writing is successful. If successful, go to step Σ74; otherwise, go to step Σ71. Σ76. Apply for chunk. If the application is successful, go to step Σ72. If the returned file is locked by another client, go to step Σ74. If the returned file lock is lost, go to step Σ75. If other errors are returned and the number does not exceed 3 times, go to step Σ76. If other errors are returned and the number exceeds 3 times, go to step Σ74. Σ77. Seal the chunk. If the return is successful, go to step Σ71. If the returned file is locked by another client, go to step Σ7. If the returned file lock is lost, go to step Σ75. If other errors are returned, continue to retry and go to step Σ77.

2. According to the big data storage access method based on the distributed storage system of claim 1, it is characterized in that: The step Σ6 further comprises the following steps: Σ62, when opening a file, the information of the chunks of data that have been written to the file will be obtained from the file system server. The permission to open the file includes write permission, so the chunk will be pre-applied for. When reading the file, the specific chunks to be read will be determined based on the offset and length. Each chunk has multiple strongly consistent copies, and messages are sent to the data storage nodes where the copies are located to read the data. Once the data is read, the next copy will no longer be read.

3. A big data storage and access method based on a distributed storage system according to claim 2, characterized in that: The further steps of step Σ62 include: Σ621, when writing a file, the specific chunks to be written will be determined based on the offset and length. If there are not enough chunks, the backend storage cluster will be requested for chunks. If a failure is returned, the server will retry four times after a certain period of time. After a successful return, the HDFS client will send the requested chunk information to the file system metadata server. If a failure is returned, the server will retry three times.

4. The big data storage access method based on a distributed storage system according to claim 2, characterized in that: Another step of step Σ62 includes: Σ622, send a message to the data storage nodes where all copies of the chunk are located to write data. Failure to write to a copy means failure to write the chunk. Seal the chunk: send a seal request to the backend storage cluster without paying attention to whether it is successful. Send a seal request to the file system metadata server, and continue to retry after returning failure until success. Finally, reselect the chunk to continue writing.

Citation Information

Patent Citations

  • Distributed file system architectural method supporting mass data access

    CN104008152A

  • A distributed metadata management method and system based on file pre-creation strategy

    CN109344122A

  • Distributed file system deployment on a data storage system

    US10740005B1

  • Modifying storage capacity of a set of storage units

    US20160294949A1