High-concurrency read-write optimization system for distributed file system, and medium and device

Abstract

Provided in the present invention are a high-concurrency read-write optimization system for a distributed file system, and a medium and a device. The high-concurrency read-write optimization system comprises: a data read-write concurrency control module, which uses a fine-grained read-write lock to ensure data consistency and high concurrency; a file data cache module, which controls a client system to cache data of a server-side file system; a read-write request delay prediction module, which predicts a running delay of a file read-write request from a client, so as to provide a path optimization scheme; and a read-write task execution module, which ensures read-write performance under a high-concurrency condition by means of simultaneously executing local and remote data read-write operations. The present invention mainly provides a new data read-write optimization system for solving the problem of the performance of a distributed file system constructed on the basis of a non-volatile memory and RDMA technology having a bottleneck regarding a high-concurrency read-write task. The present invention improves the bandwidth utilization rate of a distributed file system for a storage device and a network device, thereby improving the overall performance of a system read-write file.

Description

Highly concurrent read and write optimization systems, media, and devices for distributed file systems Technical Field

[0001] The present invention relates to the technical field of computer system architecture, and in particular to a high-concurrency read-write optimization system, medium, and device for a distributed file system. Background Art

[0002] Distributed storage systems are generally used to address the problem of insufficient storage capacity on a single computer. With the advent of the internet and big data era, many scientific research fields and commercial applications can no longer be satisfied by single-node computer systems. Generally speaking, distributed storage systems operate on a distributed cluster consisting of multiple nodes, some of which store data while others perform computations. These computations can include data I / O requests and application logic functions. The quality of a distributed storage system is primarily determined by whether it meets the basic requirements of distributed systems, such as consistency, availability, and partition tolerance, and whether it provides high-performance data services to the compute nodes in the cluster.

[0003] In recent years, the emergence of emerging hardware technologies has posed new challenges to the design of distributed storage systems. For example, the emergence of non-volatile memory and Remote Direct Memory Access (RDMA) technologies has enabled both storage and distributed system design to break through existing limitations and develop new optimization solutions for new hardware.

[0004] PM (Persistent Memory) is a new type of memory hardware technology. It has three key hardware characteristics: first, it is byte-addressable, supporting direct load / store instructions from the processor; second, it has latency and bandwidth close to DRAM; and third, it is non-volatile, meaning that stored data is not lost even after a power outage. These excellent hardware characteristics enable non-volatile memory to be used as a new storage layer between memory and disk, or to directly replace both to form a single-tier storage system.

[0005] RDMA: Remote Direct Memory Access, a remote direct memory access technology, allows different computers in a cluster to directly access each other's memory. "Directly" here means reading and modifying the memory without notifying the remote CPU. From the perspective of memory computing, RDMA technology has a similar starting point to DMA (direct memory access) in single-node computers. Both reduce the load of memory-related requests on the CPU, thereby improving overall system performance. From the perspective of network connectivity, RDMA technology is also a new generation of network communication technology, with lower communication latency and higher cross-node data throughput than server clusters previously connected by Ethernet.

[0006] Therefore, it is necessary to propose a new technical solution to improve the above technical problems.

[0007] Summary of the Invention

[0008] In view of the defects in the prior art, the purpose of the present invention is to provide a high-concurrency read-write optimization system, medium and device for a distributed file system.

[0009] According to the present invention, a high-concurrency read-write optimization system for a distributed file system is provided, the system comprising the following modules:

[0010] Module M1: The data read and write concurrency control module uses fine-grained read and write locks to obtain consistent and highly concurrent data.

[0011] Module M2: The file data cache module controls the client system's data cache of the server-side file system;

[0012] Module M3: Read and write request delay prediction module predicts the runtime delay of file read and write requests in the file system client;

[0013] Module M4: The read and write task execution module performs local and remote data read and write operations simultaneously to achieve concurrent read and write performance.

[0014] The data read and write concurrency control module processes multi-threaded parallel file data read and write requests from multiple clients to obtain consistent file data and concurrent read and write requests;

[0015] The file data cache module builds a file data cache in the client system, and the application's recent read and write requests for files are stored locally in the data cache;

[0016] The read and write delay prediction module records the delay of data read and write tasks in the system and uses a prediction model to predict the read and write request delay during runtime;

[0017] The read-write task execution module maintains two thread pools, which respectively execute data reading and writing tasks on the remote server and data reading and writing tasks on the local cache, thereby obtaining read-write performance under concurrent conditions.

[0018] Preferably, the module M1 includes:

[0019] Module M1.1: Fine-grained file data read-write lock module based on binary tree;

[0020] Module M1.2: Client data permission lease module based on hash table;

[0021] The fine-grained file data read-write lock module includes a binary tree list, each binary tree in the list corresponds to an open file data read-write lock management unit; the binary tree list uses the unique identification number of the file in the distributed file system as an index;

[0022] Each binary tree contains the data interval occupied by the current file's read and write requests. This interval is represented by a pair consisting of the start offset and length of the read and write request. The root node of the binary tree corresponds to the interval of all the file's data, and each child node corresponds to the data interval after the parent node interval is divided into two. The data segment corresponding to each node contains a read-write lock, and the requester obtains the permission of multiple reads and writes at the same time.

[0023] When a new read / write request is submitted to the file server, the data read / write lock binary tree corresponding to the file number is indexed, and the read / write permission record on the corresponding node is searched, added, or modified; when the read / write permission is revoked, the read / write permission on the corresponding node is modified or deleted;

[0024] The client data permission lease module includes a hash table consisting of lease records, which maintains all read and write lease permissions currently retained by the client system;

[0025] When the client application initiates a file read or write request, it submits the read or write request to the file system server and simultaneously submits a data cache request to the module M2;

[0026] When the read / write permission request and the cache request are processed, a read / write description item is added to the hash table, and the read / write request to be executed is transferred to the module M3.

[0027] Preferably, the module M2 includes:

[0028] A data cache management module running in a distributed file system client system;

[0029] The data cache management module includes a metadata cache unit, a data cache unit and a metadata management unit for local data cache;

[0030] When the client system initiates a metadata operation related request, the metadata cache unit requests metadata authority from the file system server and caches the relevant metadata area locally on the client;

[0031] When the client system initiates a data read / write related request, the data cache unit reads or writes data back to the file system server; at the same time, the metadata management unit of the local data cache modifies the metadata information of the local cache area.

[0032] Preferably, the module M3 includes:

[0033] Module M3.1: Data reading and writing task delay recording module;

[0034] Module M3.2: Read and write delay prediction model training module;

[0035] Module M3.3: Real-time read and write task delay prediction module;

[0036] When the module M4 executes a read / write task, the read / write task delay recording module is awakened and tracks the description parameters and start and end times of the read / write task; at the same time, the module sequentially records the data structure consisting of the description parameters and start and end times of each read / write task into a persistent file storage;

[0037] When the size change of the read and write task delay record file output by the module M3.1 exceeds a certain threshold, the delay prediction model training module reads the newly added task delay record recorded in the file and converts it into empirical knowledge of the prediction model through the model training method;

[0038] When receiving the read / write task transmitted by the module M1, the real-time read / write task delay prediction module passes the received real-time read / write task into the trained delay prediction model, and predicts the delay of the read / write task executed through the remote server node and the delay of the read / write task executed through the local cache bypassing the remote end; the two delays are compared to obtain the options of the read / write path solution.

[0039] Preferably, the module M3.1 includes:

[0040] Module M3.1.1: File system read and write operation tracking module;

[0041] Module M3.1.2: Read and write delay record persistence and aggregation module;

[0042] When the module M4 executes a remote or local read or write task, the file system read or write operation tracking module records the task parameters of the read or write task, which include the read or write file identification number, read or write flag, remote / local flag, read or write offset, and read or write length; at the same time, the file system read or write operation tracking module records the start time of the read or write task;

[0043] When the module M4 completes a remote or local read / write task, the file system read / write operation tracking module records the end time of the read / write task;

[0044] When the file system read / write operation tracking module completes a task record, the read / write delay record persistence and aggregation module writes the record into a persistent local file; the file is opened in append mode, and new content is written to the file sequentially each time it is appended;

[0045] When the length of the persistent local file record exceeds a threshold, the newly appended content in the file is submitted to the read / write delay prediction model training module;

[0046] When the data in the persistent local file record has been processed by the read / write delay prediction model training module, the read / write delay record persistence and aggregation module truncates old data in the local file record, and the record file size does not exceed a threshold.

[0047] Preferably, the module M3.2 includes:

[0048] Module M3.2.1: Read and write latency prediction model construction module;

[0049] Module M3.2.2: Delay prediction model training and update module;

[0050] When the high-concurrency read-write optimization system for a distributed file system is constructed, the read-write delay prediction model construction module constructs a read-write delay prediction model based on a neural network;

[0051] The neural network-based read / write delay prediction model includes a two-layer fully connected neural network; the neural network input data is the read / write task description parameters, and the output data is the real-time delay prediction result of the read / write task, in real-world standard time units;

[0052] Before the read / write latency prediction model is built, there are two initialization schemes for the neural network model parameters included in the prediction model: In scheme 1, the model parameters are initialized to all 0s; in scheme 2, the model parameters are initialized to the parameters of the latency prediction model trained in other systems;

[0053] After the read / write delay prediction model is built, the model is persistently stored as a file. The file is readable and writable to the read / write delay prediction model training module and is read-only to other modules in the system.

[0054] When the delay prediction model training and updating module receives the read and write task records from the read and write delay record persistence and aggregation module, it performs delay prediction model training and updating operations;

[0055] The delay prediction model training and updating module uses the input read and write task records and the corresponding real-time delay results as input and output training data of the neural network model, and uses the gradient descent method to train the delay prediction model until the error between the model's delay prediction result and the actual delay data converges;

[0056] When the error obtained by model convergence is less than the average error at the beginning of training, the delay prediction model training and update module performs model update, that is, persists the model obtained after training into a new storage file and overwrites the original model storage file.

[0057] Preferably, the module M3.3 includes:

[0058] Module M3.3.1: Real-time device read and write pressure collection module;

[0059] Module M3.3.2: Real-time read and write task delay prediction module;

[0060] When the real-time read / write task delay prediction module receives a real-time read / write delay prediction request, the real-time device read / write pressure collection module collects summary information of read / write tasks being executed in the read / write task execution module in the current system, and passes the information as part of the parameters to the neural network-based read / write delay prediction model;

[0061] When the real-time device read / write pressure collection module is completed, the real-time read / write task delay prediction module calls the neural network prediction model to perform real-time read / write delay prediction;

[0062] After the real-time read / write delay prediction is completed, the read / write task delay prediction module submits the prediction result to the read / write task execution module.

[0063] Preferably, the module M4 includes:

[0064] Module M4.1: Remote data reading and writing thread pool module based on RDMA;

[0065] Module M4.2: cache data reading and writing thread pool module based on non-volatile memory;

[0066] When the high-concurrency read-write optimization system for a distributed file system is constructed, the RDMA-based remote data read-write thread pool module initializes a certain number of RDMA connections and completes an information exchange test from the client system to the server file system;

[0067] When the RDMA connection is established, the RDMA-based remote data read and write thread pool module creates a remote data read and write thread pool based on RDMA primitives and sets all threads to idle state;

[0068] When a new remote read / write task is submitted to the read / write task execution module, the RDMA-based remote data read / write thread pool module schedules an idle thread to execute the task;

[0069] When the high-concurrency read-write optimization system for a distributed file system is constructed, the cache data read-write thread pool module based on the non-volatile memory completes the read-write operation test of the client system on the local non-volatile memory;

[0070] When the non-volatile memory read and write operation test is completed, the cache data read and write thread pool module based on the non-volatile memory creates a non-volatile memory data read and write thread pool based on the load / store primitive and sets all threads to idle state;

[0071] When a new local read / write task is submitted to the read / write task execution module, the cache data read / write thread pool module based on non-volatile memory schedules an idle thread to execute the task.

[0072] The present invention also provides a computer-readable storage medium storing a computer program, which implements the functions of the above modules when executed by a processor.

[0073] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the functions of the above modules when executed by the processor.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. This paper addresses the performance bottleneck problem faced by distributed file systems when facing high-concurrency file read and write requests, and proposes optimization methods in three directions: concurrency control, latency optimization, and parallel execution.

[0076] 2. In terms of concurrency control, the fine-grained concurrency control mechanism used in this invention enables concurrent read and write requests on the same file to be executed simultaneously, breaking through the performance bottleneck of the original file read-write lock and bringing significant performance improvement;

[0077] 3. In terms of delay optimization, the present invention adopts a multi-path delay prediction and read / write path selection solution based on a neural network. Its essence is to explore more hardware performance potential and make real-time optimization decisions during system operation;

[0078] 4. The present invention improves the overall performance of the distributed file system and is compatible with mainstream operating systems and file system interfaces, thus having good market prospects and application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0080] FIG1 is a schematic diagram of the overall module of the device according to an embodiment of the present invention;

[0081] FIG2 is a schematic diagram of a device data reading and writing concurrency control module according to an embodiment of the present invention;

[0082] FIG3 is a schematic diagram of a device file data cache module according to an embodiment of the present invention;

[0083] FIG4 is a schematic diagram of a read / write request delay prediction module of a device according to an embodiment of the present invention;

[0084] FIG5 is a schematic diagram of a read-write task execution module of a device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0085] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0086] Example 1:

[0087] According to the present invention, a high-concurrency read-write optimization system for a distributed file system is provided, the system comprising the following modules:

[0088] Module M1: The data read and write concurrency control module uses fine-grained read and write locks to obtain consistent and highly concurrent data.

[0089] Module M1.1: Fine-grained file data read-write lock module based on binary tree;

[0090] Module M1.2: Client data permission lease module based on hash table;

[0091] The fine-grained file data read-write lock module includes a binary tree list, each binary tree in the list corresponds to an open file data read-write lock management unit; the binary tree list uses the unique identification number of the file in the distributed file system as an index;

[0092] Each binary tree contains the data interval occupied by the current file's read and write requests. This interval is represented by a pair consisting of the start offset and length of the read and write request. The root node of the binary tree corresponds to the interval of all the file's data, and each child node corresponds to the data interval after the parent node interval is divided into two. The data segment corresponding to each node contains a read-write lock, and the requester obtains the permission of multiple reads and writes at the same time.

[0093] When a new read / write request is submitted to the file server, the data read / write lock binary tree corresponding to the file number is indexed, and the read / write permission record on the corresponding node is searched, added, or modified; when the read / write permission is revoked, the read / write permission on the corresponding node is modified or deleted;

[0094] The client data permission lease module includes a hash table consisting of lease records, which maintains all read and write lease permissions currently retained by the client system;

[0095] When the client application initiates a file read or write request, it submits the read or write request to the file system server and simultaneously submits a data cache request to the module M2;

[0096] When the read / write permission request and the cache request are processed, a read / write description item is added to the hash table, and the read / write request to be executed is transferred to the module M3.

[0097] Module M2: The file data cache module controls the client system's data cache of the server-side file system;

[0098] The module M2 includes: a data cache management module running in the distributed file system client system;

[0099] The data cache management module includes a metadata cache unit, a data cache unit and a metadata management unit for local data cache;

[0100] When the client system initiates a metadata operation related request, the metadata cache unit requests metadata authority from the file system server and caches the relevant metadata area locally on the client;

[0101] When the client system initiates a data read / write related request, the data cache unit reads or writes data back to the file system server; at the same time, the metadata management unit of the local data cache modifies the metadata information of the local cache area.

[0102] Module M3: Read and write request delay prediction module predicts the runtime delay of file read and write requests in the file system client;

[0103] Module M3.1: Data reading and writing task delay recording module;

[0104] Module M3.1.1: File system read and write operation tracking module;

[0105] Module M3.1.2: Read and write delay record persistence and aggregation module;

[0106] When the module M4 executes a remote or local read or write task, the file system read or write operation tracking module records the task parameters of the read or write task, which include the read or write file identification number, read or write flag, remote / local flag, read or write offset, and read or write length; at the same time, the file system read or write operation tracking module records the start time of the read or write task;

[0107] When the module M4 completes a remote or local read / write task, the file system read / write operation tracking module records the end time of the read / write task;

[0108] When the file system read / write operation tracking module completes a task record, the read / write delay record persistence and aggregation module writes the record into a persistent local file; the file is opened in append mode, and new content is written to the file sequentially each time it is appended;

[0109] When the length of the persistent local file record exceeds a threshold, the newly appended content in the file is submitted to the read / write delay prediction model training module;

[0110] When the data in the persistent local file record has been processed by the read / write delay prediction model training module, the read / write delay record persistence and aggregation module truncates old data in the local file record, and the record file size does not exceed a threshold.

[0111] Module M3.2: Read and write delay prediction model training module;

[0112] Module M3.2.1: Read and write latency prediction model construction module;

[0113] Module M3.2.2: Delay prediction model training and update module;

[0114] When the high-concurrency read-write optimization system for a distributed file system is constructed, the read-write delay prediction model construction module constructs a read-write delay prediction model based on a neural network;

[0115] The neural network-based read / write delay prediction model includes a two-layer fully connected neural network; the neural network input data is the read / write task description parameters, and the output data is the real-time delay prediction result of the read / write task, in real-world standard time units;

[0116] Before the read / write latency prediction model is built, there are two initialization schemes for the neural network model parameters included in the prediction model: In scheme 1, the model parameters are initialized to all 0s; in scheme 2, the model parameters are initialized to the parameters of the latency prediction model trained in other systems;

[0117] After the read / write delay prediction model is built, the model is persistently stored as a file. The file is readable and writable to the read / write delay prediction model training module and is read-only to other modules in the system.

[0118] When the delay prediction model training and updating module receives the read and write task records from the read and write delay record persistence and aggregation module, it performs delay prediction model training and updating operations;

[0119] The delay prediction model training and updating module uses the input read and write task records and the corresponding real-time delay results as input and output training data of the neural network model, and uses the gradient descent method to train the delay prediction model until the error between the model's delay prediction result and the actual delay data converges;

[0120] When the error obtained by model convergence is less than the average error at the beginning of training, the delay prediction model training and update module performs model update, that is, persists the model obtained after training into a new storage file and overwrites the original model storage file.

[0121] Module M3.3: Real-time read and write task delay prediction module;

[0122] Module M3.3.1: Real-time device read and write pressure collection module;

[0123] Module M3.3.2: Real-time read and write task delay prediction module;

[0124] When the real-time read / write task delay prediction module receives a real-time read / write delay prediction request, the real-time device read / write pressure collection module collects summary information of read / write tasks being executed in the read / write task execution module in the current system, and passes the information as part of the parameters to the neural network-based read / write delay prediction model;

[0125] When the real-time device read / write pressure collection module is completed, the real-time read / write task delay prediction module calls the neural network prediction model to perform real-time read / write delay prediction;

[0126] After the real-time read / write delay prediction is completed, the read / write task delay prediction module submits the prediction result to the read / write task execution module.

[0127] When the module M4 executes a read / write task, the read / write task delay recording module is awakened and tracks the description parameters and start and end times of the read / write task; at the same time, the module sequentially records the data structure consisting of the description parameters and start and end times of each read / write task into a persistent file storage;

[0128] When the size change of the read and write task delay record file output by the module M3.1 exceeds a certain threshold, the delay prediction model training module reads the newly added task delay record recorded in the file and converts it into empirical knowledge of the prediction model through the model training method;

[0129] When receiving the read / write task transmitted by the module M1, the real-time read / write task delay prediction module passes the received real-time read / write task into the trained delay prediction model, and predicts the delay of the read / write task executed through the remote server node and the delay of the read / write task executed through the local cache bypassing the remote end; the two delays are compared to obtain the options of the read / write path solution.

[0130] Module M4: The read and write task execution module performs local and remote data read and write operations simultaneously to achieve concurrent read and write performance.

[0131] Module M4.1: Remote data reading and writing thread pool module based on RDMA;

[0132] Module M4.2: cache data reading and writing thread pool module based on non-volatile memory;

[0133] When the high-concurrency read-write optimization system for a distributed file system is constructed, the RDMA-based remote data read-write thread pool module initializes a certain number of RDMA connections and completes an information exchange test from the client system to the server file system;

[0134] When the RDMA connection is established, the RDMA-based remote data read and write thread pool module creates a remote data read and write thread pool based on RDMA primitives and sets all threads to idle state;

[0135] When a new remote read / write task is submitted to the read / write task execution module, the RDMA-based remote data read / write thread pool module schedules an idle thread to execute the task;

[0136] When the high-concurrency read-write optimization system for a distributed file system is constructed, the cache data read-write thread pool module based on the non-volatile memory completes the read-write operation test of the client system on the local non-volatile memory;

[0137] When the non-volatile memory read and write operation test is completed, the cache data read and write thread pool module based on the non-volatile memory creates a non-volatile memory data read and write thread pool based on the load / store primitive and sets all threads to idle state;

[0138] When a new local read / write task is submitted to the read / write task execution module, the cache data read / write thread pool module based on non-volatile memory schedules an idle thread to execute the task.

[0139] The data read and write concurrency control module processes multi-threaded parallel file data read and write requests from multiple clients to obtain consistent file data and concurrent read and write requests;

[0140] The file data cache module builds a file data cache in the client system, and the application's recent read and write requests for files are stored locally in the data cache;

[0141] The read and write delay prediction module records the delay of data read and write tasks in the system and uses a prediction model to predict the read and write request delay during runtime;

[0142] The read-write task execution module maintains two thread pools, which respectively execute data reading and writing tasks on the remote server and data reading and writing tasks on the local cache, thereby obtaining read-write performance under concurrent conditions.

[0143] The present invention also provides a computer-readable storage medium storing a computer program, which implements the functions of the above modules when executed by a processor.

[0144] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the functions of the above modules when executed by the processor.

[0145] Example 2:

[0146] In view of the defects in the prior art, the purpose of the present invention is to provide a high-concurrency read and write optimization method and system for a distributed file system.

[0147] The high-concurrency read-write optimization method and system for a distributed file system provided by the present invention include:

[0148] Module M1: Data read and write concurrency control module, which uses fine-grained read and write locks to ensure data consistency and high concurrency;

[0149] Module M2: File data cache module, which controls the client system to cache data on the server file system;

[0150] Module M3: Read and write request delay prediction module, which predicts the runtime delay of file read and write requests in the file system client;

[0151] Module M4: Read and write task execution module, which ensures optimal read and write performance under concurrent conditions by executing local and remote data read and write operations simultaneously;

[0152] The data reading and writing concurrency control module processes multi-threaded parallel file data reading and writing requests from multiple clients to ensure file data consistency and concurrency of reading and writing requests.

[0153] The file data cache module is used to build a file data cache in the client system to ensure that the latest read and write requests of the application to the file are stored in the local data cache.

[0154] The read / write delay prediction module records the delay of data read / write tasks in the system and uses a prediction model to predict the delay of read / write requests during runtime.

[0155] The read and write task execution module maintains two thread pools to respectively execute data reading and writing tasks on the remote server and data reading and writing tasks on the local cache, thereby ensuring optimal read and write performance under concurrent conditions.

[0156] Preferably, the module M1 includes:

[0157] Module M1.1: Fine-grained file data read-write lock module based on binary tree;

[0158] The fine-grained file data read-write lock module includes a binary tree list, each binary tree in the list corresponds to an open file data read-write lock management unit. The binary tree list uses the unique identification number of the file in the distributed file system as an index.

[0159] Each binary tree contains the data interval occupied by read and write requests for the current file. This interval is represented by a pair consisting of the start offset and length of the read or write request. The root node of the binary tree corresponds to the interval of the entire file data, and each child node corresponds to the data interval obtained by dividing the parent node interval into two. The data segment corresponding to each node contains a read-write lock, allowing the requester to obtain simultaneous write-multiple read permissions.

[0160] When a new read / write request is submitted to the file server, the data read / write lock binary tree corresponding to the file number is indexed, and the read / write permission record on the corresponding node is searched, added, or modified. When the read / write permission is revoked, the read / write permission on the corresponding node is modified or deleted.

[0161] Module M1.2: Client data permission lease module based on hash table;

[0162] The client data permission lease module includes a hash table consisting of lease records, which maintains all read and write lease permissions currently retained by the client system.

[0163] When the client application initiates a file read or write request, it submits the read or write request to the file system server and simultaneously submits a data cache request to the module M2 described in claim 1.

[0164] When the read / write permission request and the cache request are processed, a read / write description item is added to the hash table, and the read / write request to be executed is transferred to the module M3 in claim 1.

[0165] Preferably, the module M2 includes:

[0166] A data cache management module running in a distributed file system client system.

[0167] The data cache management module includes: a metadata cache unit, a data cache unit, and a metadata management unit for local data cache.

[0168] When the client system initiates a metadata operation related request, the metadata cache unit requests metadata authority from the file system server and caches the relevant metadata area locally on the client;

[0169] When the client system initiates a data read or write related request, the data cache unit reads or writes back data to the file system server; at the same time, the metadata management unit of the local data cache modifies the metadata information of the local cache area to ensure the consistency, correctness, and integrity of the data cache area.

[0170] Preferably, the module M3 includes:

[0171] Module M3.1: Data reading and writing task delay recording module;

[0172] When the module M4 in claim 1 executes a read / write task, the read / write task delay recording module is awakened and tracks the description parameters and start and end times of the read / write tasks; at the same time, the module sequentially records a data structure consisting of the description parameters and start and end times of each read / write task into a persistent file storage;

[0173] Module M3.2: Read and write delay prediction model training module;

[0174] When the change in the size of the read and write task delay record file output by module M3.1 in claim 4 exceeds a certain threshold, the delay prediction model training module reads the newly added task delay record recorded in the file and converts it into empirical knowledge of the prediction model through a model training method.

[0175] Module M3.3: Real-time read and write task delay prediction module;

[0176] When receiving the read / write task transmitted by the module M1 described in claim 1, the real-time read / write task delay prediction module passes the received real-time read / write task into the trained delay prediction model, and predicts the delay of the read / write task executed via the remote server node and the delay of the read / write task executed via the local cache bypassing the remote end; the comparison of the two delays provides an option for the read / write path optimization solution.

[0177] Preferably, the module M3.1 includes:

[0178] Module M3.1.1: File system read and write operation tracking module;

[0179] When the module M4 in claim 4 executes a remote or local read or write task, the file system read or write operation tracking module records the task parameters of the read or write task, which parameters include the read or write file identification number, the read or write flag, the remote / local flag, the read or write offset, and the read or write length; and the file system read or write operation tracking module records the start time of the read or write task;

[0180] When the module M4 in claim 4 completes a remote or local read or write task, the file system read or write operation tracking module records the end time of the read or write task;

[0181] Module M3.1.2: Read and write delay record persistence and aggregation module;

[0182] When the file system read / write operation tracking module of claim 5 completes a task record, the read / write delay record persistence and aggregation module writes the record into a persistent local file; the file is opened in append mode, and new content is written to the file sequentially each time it is appended;

[0183] When the length of the persistent local file record exceeds a threshold, the newly appended content in the file is submitted to the read / write delay prediction model training module of claim 4;

[0184] When the data in the persistent local file record has been processed by the read / write delay prediction model training module in claim 4, the read / write delay record persistence and aggregation module truncates the old data in the local file record to ensure that the record file size does not exceed a threshold.

[0185] Preferably, the module M3.2 includes:

[0186] Module M3.2.1: Read and write latency prediction model construction module;

[0187] When the high-concurrency read-write optimization system for a distributed file system as described in claim 1 is constructed, the read-write delay prediction model construction module constructs a read-write delay prediction model based on a neural network;

[0188] The neural network-based read / write delay prediction model includes: a two-layer fully connected neural network; the neural network input data is the read / write task description parameters, and the output data is the real-time delay prediction result of the read / write task, in real-world standard time units;

[0189] Before the read / write latency prediction model is built, there are two initialization schemes for the neural network model parameters included in the prediction model: In scheme 1, the model parameters are initialized to all 0s; in scheme 2, the model parameters are initialized to the parameters of the latency prediction model trained in other systems;

[0190] When the read / write delay prediction model is built, the model is persistently stored as a file. The file is readable and writable to the read / write delay prediction model training module described in claim 4, and is a read-only file for other modules in the system.

[0191] Module M3.2.2: Delay prediction model training and update module;

[0192] When the delay prediction model training and updating module receives the read and write task records from the read and write delay record persistence and aggregation module in claim 5, the delay prediction model training and updating operation is performed;

[0193] The delay prediction model training and updating module uses the input read and write task records and the corresponding real-time delay results as input and output training data of the neural network model, and uses the gradient descent method to train the delay prediction model until the error between the model's delay prediction result and the actual delay data converges;

[0194] When the error obtained by model convergence is less than the average error at the beginning of training, the delay prediction model training and update module performs model update, that is, persists the model obtained after training into a new storage file and overwrites the original model storage file.

[0195] Preferably, the module M3.3 includes:

[0196] Module M3.3.1: Real-time device read and write pressure collection module;

[0197] When the real-time read / write task delay prediction module of claim 4 receives a real-time read / write delay prediction request, the real-time device read / write pressure collection module collects summary information of the read / write tasks being executed in the read / write task execution module of the current system, and passes the information as part of the parameters to the neural network-based read / write delay prediction model of claim 6;

[0198] Module M3.3.2: Real-time read and write task delay prediction module;

[0199] When the real-time device read / write pressure collection module is completed, the real-time read / write task delay prediction module calls the neural network prediction model to perform real-time read / write delay prediction;

[0200] After the real-time read / write delay prediction is completed, the read / write task delay prediction module submits the prediction result to the read / write task execution module according to claim 1.

[0201] Preferably, the module M4 includes:

[0202] Module M4.1: Remote data reading and writing thread pool module based on RDMA;

[0203] When the high-concurrency read-write optimization system for a distributed file system as described in claim 1 is constructed, the RDMA-based remote data read-write thread pool module initializes a certain number of RDMA connections and completes the information exchange test from the client system to the server file system;

[0204] When the RDMA connection is established, the RDMA-based remote data read and write thread pool module creates a remote data read and write thread pool based on RDMA primitives and sets all threads to idle state;

[0205] When a new remote read / write task is submitted to the read / write task execution module according to claim 1, the RDMA-based remote data read / write thread pool module schedules an idle thread to execute the task;

[0206] Module M4.2: cache data reading and writing thread pool module based on non-volatile memory;

[0207] When the high-concurrency read-write optimization system for a distributed file system as described in claim 1 is constructed, the cache data read-write thread pool module based on non-volatile memory completes the read-write operation test of the client system on the local non-volatile memory;

[0208] When the non-volatile memory read and write operation test is completed, the cache data read and write thread pool module based on the non-volatile memory creates a non-volatile memory data read and write thread pool based on the load / store primitive and sets all threads to idle state;

[0209] When a new local read / write task is submitted to the read / write task execution module according to claim 1, the cache data read / write thread pool module based on non-volatile memory schedules an idle thread to execute the task.

[0210] A computer-readable storage medium storing a computer program, wherein the computer program implements the functions of the above modules when executed by a processor.

[0211] As shown in FIG1 , a high-concurrency read-write optimization system for a distributed file system provided by the present invention includes:

[0212] The data read and write concurrency control module constructs fine-grained read-write locks on the file system server to ensure data consistency and high concurrency. It also constructs a hash-table-based client data permission lease record table on the file system client to reduce the time overhead caused by concurrent lock requests.

[0213] File data cache module, which controls the client system's data cache of the server-side file system;

[0214] The read and write request latency prediction module builds a neural network-based read and write task latency prediction model to predict real-time file read and write request latency in the file system client.

[0215] The read and write task execution module builds two read and write task thread pools, schedules local and remote read and write tasks, and simultaneously executes local and remote data read and write operations to improve read and write performance under concurrent conditions.

[0216] When a file system client application initiates a file read or write request, the request is first submitted to the concurrency control module. This module parses the read or write request and submits the data cache request to the data cache module. At the same time, it passes the read or write task description to the delay prediction module to obtain the read or write delay prediction result.

[0217] When the data cache operation and read / write delay prediction are completed, the read / write tasks are submitted to the task execution module by the delay prediction module and executed concurrently by the read / write thread pool therein.

[0218] As shown in FIG2 , the read-write concurrency control module provided according to the present invention includes:

[0219] The binary tree-based fine-grained file data read-write lock module contains a binary tree list, where each binary tree corresponds to an open file data read-write lock management unit. The binary tree list uses the unique identification number of the file in the distributed file system as an index.

[0220] Each binary tree contains the data interval occupied by read and write requests for the current file. This interval is represented by a pair consisting of the start offset and length of the read or write request. The root node of the binary tree corresponds to the interval of the entire file data, and each child node corresponds to the data interval obtained by dividing the parent node interval into two. The data segment corresponding to each node contains a read-write lock, allowing the requester to obtain simultaneous write-multiple read permissions.

[0221] When a new read / write request is submitted to the file server, the data read / write lock binary tree corresponding to the file number is indexed, and the read / write permission record on the corresponding node is searched, added, or modified. When the read / write permission is revoked, the read / write permission on the corresponding node is modified or deleted.

[0222] The client data permission lease module based on hash table includes a hash table consisting of lease records, which maintains all read and write lease permissions currently retained by the client system.

[0223] When a client application initiates a file read or write request, it submits the read or write request to the file system server and simultaneously submits a data cache request to the file data cache module of the present invention.

[0224] When the read / write permission request and the cache request are processed, a read / write description item is added to the hash table, and at the same time, the read / write request to be executed is transferred to the read / write delay prediction module of the present invention.

[0225] In the distributed file system, the file data read-write lock module and the data permission lease module transmit information through the RDMA communication unit between the client and the server.

[0226] As shown in FIG3 , the file data cache module provided according to the present invention includes:

[0227] A metadata cache unit, a file data cache unit, and a metadata management unit for local data cache.

[0228] The metadata cache unit caches metadata information in the file system inode and provides cache services for metadata-related operations of the client to reduce time overhead;

[0229] The file data cache unit only caches the actual data in the file data blocks, providing cache services for client data read and write operations to reduce time overhead. The unit also uses the MESI cache consistency protocol;

[0230] The metadata management unit of the local data cache is responsible for allocating space in the local cache area, allocating non-volatile memory when new cache space is needed, collating and merging cache contents when the cache area is full, or replacing outdated cache data blocks and metadata data blocks;

[0231] When the client system initiates a metadata operation related request, the metadata cache unit requests metadata authority from the file system server and caches the relevant metadata area locally on the client;

[0232] When the client system initiates a data read or write request, the data cache unit reads or writes data back to the file system server; at the same time, the metadata management unit of the local data cache modifies the metadata information of the local cache area to ensure the consistency and correctness of the data cache area.

[0233] As shown in FIG4 , the read / write delay prediction module provided by the present invention includes three submodules: a data read / write task delay recording module, a read / write delay prediction model training module, and a real-time read / write task delay prediction module;

[0234] The data read and write task delay recording module further includes two submodules: a file system read and write operation tracking module and a read and write delay record persistence and aggregation module;

[0235] When the system is running, the file system read and write operation tracking module performs the following operations:

[0236] When the read / write task execution module of the present invention executes a remote or local read / write task, the file system read / write operation tracking module records the task parameters of the read / write task, which parameters include the read / write file identification number, the read / write flag, the remote / local flag, the read / write offset, and the read / write length; at the same time, the file system read / write operation tracking module records the start time of the read / write task;

[0237] When the read / write task execution module of the present invention completes a remote or local read / write task, the file system read / write operation tracking module records the end time of the read / write task;

[0238] When the system is running, the read and write delay record persistence and aggregation module performs the following operations:

[0239] When the file system read / write operation tracking module completes a task record, the read / write delay record persistence and aggregation module writes the record to a persistent local file; the file is opened in append mode, and new content is written to the file sequentially each time it is appended;

[0240] When the length of the persistent local file record exceeds a threshold, the newly appended content in the file is submitted to the read / write delay prediction model training module;

[0241] When the data in the persistent local file record has been processed by the read-write delay prediction model training module, the read-write delay record persistence and aggregation module truncates the old data in the local file record to ensure that the record file size does not exceed a threshold. The setting of this threshold can ensure that the file system storage space occupied by the read-write delay prediction model is controllable, while not affecting the storage of the latest data to support model training.

[0242] In addition, the read and write delay prediction model training module includes two submodules: a read and write delay prediction model construction module and a delay prediction model training and update module;

[0243] When the distributed file system is initialized, the read / write delay prediction model building module completes the task of building a read / write delay prediction model based on a neural network;

[0244] The neural network-based read / write delay prediction model includes: a two-layer fully connected neural network; the neural network input data is the read / write task description parameters, and the output data is the real-time delay prediction result of the read / write task, in real-world standard time units, such as microseconds;

[0245] Before the read / write latency prediction model is built, there are two initialization schemes for the neural network model parameters included in the prediction model: In scheme 1, the model parameters are initialized to all 0s; in scheme 2, the model parameters are initialized to the parameters of the latency prediction model trained in other systems;

[0246] After the read / write delay prediction model is built, the model is persistently stored as a file. The file is readable and writable for the read / write delay prediction model training module, and is read-only for other modules in the system of the present invention.

[0247] When the system is running, when the delay prediction model training and updating module receives the read and write task records from the read and write delay record persistence and aggregation module, it performs the delay prediction model training and updating operation;

[0248] The training data is sent to the latency prediction model training and update module in batches. We set the size of a single training batch to an integer power of 2, for example, 2048 task records, to optimize the memory overhead of the neural network.

[0249] At the beginning of training, the delay prediction model training and update module uses the input read and write task records and the corresponding real-time delay results as the input and output training data of the neural network model, and uses the gradient descent method to train the delay prediction model until the error between the model's delay prediction results and the actual delay data converges, that is, the training error no longer decreases by more than 1% within a training cycle;

[0250] When the error obtained by model convergence is less than the average error at the beginning of training, the delay prediction model training and update module performs model update, that is, persists the model obtained after training into a new storage file and overwrites the original model storage file.

[0251] In addition, the real-time read / write task delay prediction module includes two submodules: a real-time device read / write pressure collection module and a real-time read / write task delay prediction module;

[0252] When the system is running, when the task delay prediction module receives a real-time read / write delay prediction request, the first submodule: the real-time device read / write pressure collection module collects summary information of the read / write tasks being executed in the read / write task execution module in the current system, and passes the information as part of the parameters to the neural network-based read / write delay prediction model to assist in real-time delay prediction;

[0253] When the system is running, when the real-time device read / write pressure collection module is completed, the real-time read / write task delay prediction module calls the neural network prediction model to perform real-time read / write delay prediction;

[0254] When the real-time read and write delay prediction is completed, the read and write task delay prediction module submits the prediction result to the read and write task execution module to assist the module in selecting an appropriate read and write task thread pool and completing efficient data reading and writing.

[0255] As shown in FIG5 , the read-write task execution module provided by the present invention includes two submodules: a remote data read-write thread pool module based on RDMA and a cache data read-write thread pool module based on non-volatile memory;

[0256] The RDMA-based remote data read and write thread pool module is responsible for scheduling and executing remote read and write tasks assigned to the distributed file system server;

[0257] When the high-concurrency read-write optimization system for distributed file systems of the present invention is constructed, the RDMA-based remote data read-write thread pool module initializes a certain number of RDMA connections and completes information exchange tests from the client system to the server file system, including send / recv operation tests and read / write tests;

[0258] When the RDMA connection is established, the RDMA-based remote data read and write thread pool module creates a remote data read and write thread pool based on RDMA primitives and sets all threads to idle state;

[0259] When a new remote read / write task is submitted to the read / write task execution module, the RDMA-based remote data read / write thread pool module schedules an idle thread to execute the task;

[0260] The cache data read and write thread pool module based on non-volatile memory is responsible for scheduling and executing the read and write tasks assigned to the local non-volatile memory cache area;

[0261] When the high-concurrency read-write optimization system for distributed file systems of the present invention is constructed, the cache data read-write thread pool module based on non-volatile memory completes the client system's read and write operation test on the local non-volatile memory, including the non-volatile memory read and write test using load / store;

[0262] When the non-volatile memory read and write operation test is completed, the cache data read and write thread pool module based on non-volatile memory creates a non-volatile memory data read and write thread pool based on memory read and write primitives, and sets all threads to idle state; when a new local read and write task is submitted to the read and write task execution module, the cache data read and write thread pool module based on non-volatile memory schedules an idle thread to execute the task.

[0263] Those skilled in the art may understand this embodiment as a more specific description of Embodiment 1.

[0264] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functionality of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered structures within the hardware component. Alternatively, the devices, modules, and units for implementing various functions can be considered both software modules implementing the method and structures within the hardware component.

[0265] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A high-concurrency read-write optimization system for distributed file systems, characterized in that: The system includes the following modules: Module M1: The data read-write concurrency control module uses fine-grained read-write locks to obtain consistent and highly concurrent data; Module M2: The file data cache module controls the client system to cache the data of the server file system; Module M3: The read and write request delay prediction module predicts the runtime delay of file read and write requests in the file system client; Module M4: The read and write task execution module obtains the read and write performance under concurrent conditions by executing local and remote data read and write operations simultaneously; The data read and write concurrency control module processes multi-threaded parallel file data read and write requests from multiple clients to obtain consistent file data and concurrent read and write requests; The file data cache module constructs a file data cache in the client system, and the application's recent read and write requests for files are locally stored with a data cache; The read / write delay prediction module records the delay of data read / write tasks in the system and uses a prediction model to predict the delay of read / write requests during runtime; The read-write task execution module maintains two thread pools, which respectively execute the data read-write tasks on the remote server and the data read-write tasks on the local cache, so as to obtain the read-write performance under concurrent conditions.

2. The high-concurrency read-write optimization system for distributed file systems according to claim 1, characterized in that: The module M1 comprises: Module M1.1: Fine-grained file data read-write lock module based on binary tree; Module M1.2: Client data permission lease module based on hash table; The fine-grained file data read-write lock module includes a binary tree list, each binary tree in the list corresponds to an open file data read-write lock management unit; the binary tree list uses the unique identification number of the file in the distributed file system as an index; Each binary tree contains the data interval occupied by the read and write requests of the current file, which is represented by a pair consisting of the starting offset and length of the read and write requests. The root node of the binary tree corresponds to the interval of all the data of the file, and each child node corresponds to the data interval after the parent node interval is divided into two. The data segment corresponding to each node contains a read-write lock, and the requester obtains the permission of one write and multiple reads at the same time. When a new read / write request is submitted to the file server, the binary tree of data read / write locks corresponding to the file number is indexed. And search, add, or modify the read and write permission records on the corresponding nodes; when the read and write permissions are revoked, the read and write permissions on the corresponding nodes are modified or deleted; The client data permission lease module includes a hash table consisting of lease records, and the hash table maintains all read and write lease permissions currently reserved by the client system; When the client application initiates a file read or write request, it submits the read or write request to the file system server and simultaneously submits a data cache request to the module M2; When the read / write permission request and the cache request are processed, a read / write description item is added to the hash table, and the read / write request to be executed is transmitted to the module M3.

3. The high-concurrency read-write optimization system for distributed file systems according to claim 1, characterized in that: The module M2 comprises: A data cache management module running in a distributed file system client system; The data cache management module includes a metadata cache unit, a data cache unit and a metadata management unit for local data cache; When the client system initiates a metadata operation related request, the metadata cache unit requests metadata authority from the file system server and caches the related metadata area locally on the client; When the client system initiates a data read / write related request, the data cache unit reads or writes data back to the file system server; at the same time, the metadata management unit of the local data cache modifies the metadata information of the local cache area.

4. The high-concurrency read-write optimization system for distributed file systems according to claim 1, characterized in that: The module M3 comprises: Module M3.1: Data reading and writing task delay recording module; Module M3.2: Read and write delay prediction model training module; Module M3.3: real-time read and write task delay prediction module; When the module M4 executes the read / write task, the read / write task delay recording module is awakened and tracks the description parameters and start / end time of the read / write task; at the same time, the module sequentially records the data structure composed of the description parameters and start / end time of each read / write task into a persistent file storage; When the size change of the read / write task delay record file output by the module M3.1 exceeds a certain threshold, the delay prediction model training module reads the newly added task delay record recorded in the file and converts it into empirical knowledge of the prediction model through the model training method; When receiving the read / write task transmitted by the module M1, the real-time read / write task delay prediction module passes the received real-time read / write task into the trained delay prediction model, and predicts the read / write task via the remote server. The latency of executing at the remote node and the latency of executing via the local cache bypassing the remote node are compared to obtain the options of the read and write path solution.

5. The high-concurrency read-write optimization system for distributed file systems according to claim 4, characterized in that: The module M3.1 includes: Module M3.1.1: File system read and write operation tracking module; Module M3.1.2: Read and write delay record persistence and aggregation module; When the module M4 executes a remote or local read / write task, the file system read / write operation tracking module records the task parameters of the read / write task, which parameters include the read / write file identification number, the read / write identification bit, the remote / local identification bit, the read / write offset, and the read / write length; at the same time, the file system read / write operation tracking module records the start time of the read / write task; When the module M4 completes a remote or local read / write task, the file system read / write operation tracking module records the end time of the read / write task; When the file system read / write operation tracking module completes a task record, the read / write delay record persistence and aggregation module writes the record into a persistent local file; the file is opened in append mode, and new content is sequentially written into the file each time it is appended; When the length of the persistent local file record exceeds a threshold, the newly appended content in the file is submitted to the read / write delay prediction model training module; When the data in the persistent local file record has been processed by the read / write delay prediction model training module, the read / write delay record persistence and aggregation module truncates the old data in the local file record, and the record file size does not exceed a threshold.

6. The high-concurrency read-write optimization system for distributed file systems according to claim 4, characterized in that: The module M3.2 includes: Module M3.2.1: Read and write delay prediction model building module; Module M3.2.2: Delay prediction model training and update module; When the high-concurrency read-write optimization system for a distributed file system is constructed, the read-write delay prediction model construction module constructs a read-write delay prediction model based on a neural network; The neural network-based read / write delay prediction model includes a two-layer fully connected neural network; the neural network input data is the read / write task description parameters, and the output data is the real-time delay prediction result of the read / write task, in real-world standard time units; Before the read / write latency prediction model is built, the neural network model parameters included in the prediction model are initialized in two ways: Solution: In Solution 1, the model parameters are initialized to all 0; in Solution 2, the model parameters are initialized to the delay prediction model parameters trained in other systems; When the read-write delay prediction model is built, the model is persistently stored as a file, which is readable and writable to the read-write delay prediction model training module and is a read-only file for other modules in the system; When the delay prediction model training and updating module receives the read and write task records from the read and write delay record persistence and aggregation module, the delay prediction model training and updating operation is performed; The delay prediction model training and updating module uses the input read and write task records and the corresponding real-time delay results as input and output training data of the neural network model, and uses the gradient descent method to train the delay prediction model until the error between the model's prediction result of the delay and the actual delay data converges; When the error obtained by model convergence is less than the average error at the beginning of training, the delay prediction model training and update module performs model update, that is, persists the model obtained after training as a new storage file and overwrites the original model storage file.

7. The high-concurrency read-write optimization system for distributed file systems according to claim 4, characterized in that: The module M3.3 includes: Module M3.3.1: Real-time device read and write pressure collection module; Module M3.3.2: Real-time read and write task delay prediction module; When the real-time read / write task delay prediction module receives a real-time read / write delay prediction request, the real-time device read / write pressure collection module collects summary information of read / write tasks being executed in the read / write task execution module in the current system, and passes the information as part of the parameters to the neural network-based read / write delay prediction model; When the real-time device read / write pressure collection module is completed, the real-time read / write task delay prediction module calls the neural network prediction model to perform real-time read / write delay prediction; When the real-time read / write delay prediction is completed, the read / write task delay prediction module submits the prediction result to the read / write task execution module.

8. The high-concurrency read-write optimization system for distributed file systems according to claim 1, characterized in that: The module M4 comprises: Module M4.1: Remote data reading and writing thread pool module based on RDMA; Module M4.2: cache data reading and writing thread pool module based on non-volatile memory; When the high-concurrency read-write optimization system for distributed file systems is constructed, the RDMA-based remote data read-write thread pool module initializes a certain number of RDMA connections and completes the information exchange test from the client system to the server file system; When the RDMA connection is established, the RDMA-based remote data read-write thread pool module creates a remote data read-write thread pool based on the RDMA primitive and sets all threads to an idle state; When a new remote read / write task is submitted to the read / write task execution module, the RDMA-based remote data read / write thread pool module schedules an idle thread to execute the task; When the high-concurrency read-write optimization system for distributed file systems is constructed, the cache data read-write thread pool module based on non-volatile memory completes the client system's read-write operation test on the local non-volatile memory; When the non-volatile memory read and write operation test is completed, the cache data read and write thread pool module based on the non-volatile memory creates a non-volatile memory data read and write thread pool based on the load / store primitive, and sets all threads to an idle state; When a new local read / write task is submitted to the read / write task execution module, the cache data read / write thread pool module based on non-volatile memory schedules an idle thread to execute the task.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the function of the module described in any one of claims 1 to 8 is realized.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the function of the module according to any one of claims 1 to 8 is realized.

Images