Method and system for realizing storage-computing separation through distributed file cache system

Abstract

The application provides a method and system for realizing storage-computing separation through a distributed file cache system, comprising the following steps: S1, deploying a service end of a network file system in a bottom-layer storage system, and sharing a source data directory to an active area where a computing cluster is located; S2, deploying a distributed file cache system in the active area where the computing cluster is located, and mounting the shared source data directory in the distributed file cache system; and S3, selecting a submitted computing task node in the active area where the computing cluster is located, reading data, and writing the data into a working node. According to the application, the working node of the cluster can pull data from a remote storage and then locally cache the data to realize one-time reading and multiple-time use, thereby reducing the number of cross-city network access times.

Description

Technical Field

[0001] This invention relates to the field of data analysis, specifically to a method and system for achieving storage-computation separation through a distributed file caching system, and more specifically to a method and system for achieving storage-computation separation and improving remote storage read / write performance through a distributed file caching system. Background Technology

[0002] When a user requests file processing (such as uploading a file) to the file service cluster, the file will be uploaded to a node in the entire cluster in a random or round-robin manner. If the selected node is not in the same data center as the current application node, cross-data center file processing will occur, resulting in high latency and low performance in file processing.

[0003] Patent document CN113961529A (application number: CN202111257974.4) discloses a distributed file system and file processing method. The system includes at least two data centers, each housing an Nginx server, an application server, and a FastDFS server. The FastDFS server includes at least one tracking server and a storage server. A master data center and slave data centers are configured, with the Nginx server, application server, tracking server, and storage server in the master data center sequentially configured as the master Nginx server, master application server, master tracking server, and master storage server, respectively. When a file processing request is received, the file can be processed through the cooperation of the master Nginx server, master application server, master tracking server, and master storage server. However, this invention fails to decouple the computing framework from the persistent storage system, shield the underlying storage, and provide a unified persistent data access interface. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for achieving storage-computation separation through a distributed file caching system.

[0005] A method for implementing storage-computation separation through a distributed file caching system, provided by the present invention, includes:

[0006] Step S1: Deploy the network file system server on the underlying storage system and share the source data directory with the active area where the computing cluster is located;

[0007] Step S2: Deploy a distributed file caching system in the active area where the computing cluster is located, and mount the shared source data directory within the distributed file caching system;

[0008] Step S3: Select the node in the active area where the computing cluster is located to submit the computing task, read the data and write it to the worker node.

[0009] Preferably, when multiple computing tasks depend on the same source data, the cached data will be copied between worker nodes to achieve short-circuit read and write.

[0010] Preferably, in step S1:

[0011] Step S1.1: Select a node in the storage cluster to configure the network file system server, and write the source data directory and return data directory of the underlying storage system into the configuration file;

[0012] Step S1.2: Start the RPC service and network file system service on the server side;

[0013] Step S1.3: Configure the network file system client on selected nodes in the computing cluster;

[0014] Step S1.4: Mount the source data directory and return the data directory on the client.

[0015] Preferably, in step S2:

[0016] Step S2.1: Configure a distributed file caching system on each compute node, define the underlying storage system, and define the worker nodes;

[0017] Step S2.2: Establish mutual trust between nodes, copy the configuration to all worker nodes, and format it;

[0018] Step S2.3: Run the distributed file caching system;

[0019] Step S2.4: Mount the source data directory on the worker nodes within the distributed file caching system.

[0020] Preferably, in step S3:

[0021] Step S3.1: Create a file cache directory within the distributed file caching system;

[0022] Step S3.2: Select some nodes in the distributed file caching system as computing task submission nodes, deploy the application implemented by distributed lock, write the data of the underlying storage system to the file cache directory through IO stream, and use it as the input source for subsequent processing tasks.

[0023] According to the present invention, a system for achieving storage-computation separation through a distributed file caching system includes:

[0024] Module M1: Deploys the network file system server in the underlying storage system and shares the source data directory with the active area where the computing cluster is located;

[0025] Module M2: Deploys a distributed file caching system in the active area where the computing cluster is located, and mounts the shared source data directory within the distributed file caching system;

[0026] Module M3: Selects the node in the active area where the computing cluster is located to submit computing tasks, reads data, and writes it to the worker node.

[0027] Preferably, when multiple computing tasks depend on the same source data, the cached data will be copied between worker nodes to achieve short-circuit read and write.

[0028] Preferably, in module M1:

[0029] Module M1.1: Select a node in the storage cluster to configure a network file system server, and write the source data directory and return data directory of the underlying storage system into the configuration file;

[0030] Module M1.2: Starts RPC service and network file system service on the server side;

[0031] Module M1.3: Configures a network file system client on selected nodes in the computing cluster;

[0032] Module M1.4: Mounts the source data directory and return data directory on the client side.

[0033] Preferably, in module M2:

[0034] Module M2.1: Configures a distributed file caching system on each compute node, defines the underlying storage system, and defines the worker nodes;

[0035] Module M2.2: Establishes mutual trust between nodes, copies the configuration to all worker nodes, and formats it;

[0036] Module M2.3: Runs a distributed file caching system;

[0037] Module M2.4: Mounts the source data directory on worker nodes within a distributed file caching system.

[0038] Preferably, in module M3:

[0039] Module M3.1: Creates a file cache directory within the distributed file caching system;

[0040] Module M3.2: Selects some nodes within the distributed file caching system as task submission nodes, deploys an application implemented with distributed locks, writes data from the underlying storage system to the file cache directory via IO streams, and uses it as the input source for subsequent processing tasks.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. This invention deploys a distributed file caching cluster, where the cluster's worker nodes can retrieve data from remote storage and cache it locally, enabling multiple uses from a single read and reducing the number of cross-city network accesses.

[0043] 2. In this invention, the cluster can decouple the computing framework and the persistent storage system, shield the underlying storage, provide a unified persistent data access interface, and realize storage-computation separation and high-performance read and write. Attached Figure Description

[0044] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0045] Figure 1 This is a technical architecture diagram of the present invention. Detailed Implementation

[0046] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0047] Example 1:

[0048] High-concurrency cross-data center data reading will result in high network bandwidth consumption. By deploying a distributed file caching cluster, the cluster's worker nodes can pull data from remote storage and cache it locally to achieve read once and use it multiple times, reducing the number of cross-city network accesses.

[0049] Meanwhile, the cluster can decouple the computing framework and persistent storage system, shield the underlying storage, provide a unified persistent data access interface, and achieve storage-computation separation and high-performance read and write.

[0050] like Figure 1 As shown, (1) UFS is GPFS, which is mounted to the Alluxio Master node and all Worker nodes via NFS; (2) Worker and computing framework are co-located, that is, the Worker node also serves as the NodeManager node; (3) AlluxioClient (application) reads UFS data and caches it to alluxio: / / host:port / app / file / input, and at the same time caches the decompressed data to alluxio: / / host:port / unzipfile. The input source for the computing task is alluxio: / / host:port / unzipfile.

[0051] A method for achieving storage-computation separation and improving remote storage read / write performance through a distributed file caching system includes:

[0052] Step 1: By default, the compute cluster and storage cluster are not in the same active area. First, deploy the network file system server on the underlying storage system and share the source data directory and return data directory with the active area where the compute cluster is located.

[0053] Step 2: Deploy a distributed file caching system in the active area where the computing cluster is located, and remount the shared source data directory within the distributed file caching system;

[0054] Step 3: Select the task submission node in the active area where the computing cluster is located, read data from the underlying storage system through the distributed file caching system, and write the data to the worker node;

[0055] Step 4: When multiple computing tasks depend on the same source data, the cached data will be copied between worker nodes, ultimately achieving short-circuit read and write.

[0056] Step 1 includes the following steps:

[0057] Step 1.1: Select one node in the storage cluster to configure the network file system server, and write the source data directory and return data directory of the underlying storage system into the configuration file;

[0058] Step 1.2: Start the RPC service and network file system service on the server side;

[0059] Step 1.3: Configure the network file system client on selected nodes in the computing cluster;

[0060] Step 1.4: Mount the source data directory and return the data directory on the client.

[0061] Step 2 includes the following steps:

[0062] Step 2.1: Configure a distributed file caching system on each compute node, define the underlying storage system, and define the worker nodes;

[0063] Step 2.2: Establish mutual trust between nodes, copy the configuration to all worker nodes, and format it;

[0064] Step 2.3: Run the distributed file caching system;

[0065] Step 2.4: Mount the source data directory on the worker nodes within the distributed file caching system.

[0066] Step 3 includes the following steps:

[0067] Step 3.1: Create a file cache directory within the distributed file caching system;

[0068] Step 3.2: Select some nodes within the distributed file caching system as computation task submission nodes, deploy an application implemented with distributed locks, write data from the underlying storage system to the file cache directory via IO streams, and use it as the input source for subsequent processing tasks.

[0069] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0070] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for implementing storage-computation separation through a distributed file caching system, characterized in that, Comprising: Step S1: deploying a network file system server in the underlying storage system, and sharing the source data directory to the active area where the computing cluster is located; Step S2: deploying a distributed file cache system in the active area where the computing cluster is located, and mounting the shared source data directory in the distributed file cache system; Step S3: selecting a computing task submission node in the active area where the computing cluster is located, reading data and writing to the worker node; In the step S2: Step S2.1: configuring the distributed file cache system on each computing node, defining the underlying storage system, and defining the worker node; Step S2.2: establishing mutual trust between nodes, copying the configuration to all worker nodes, and formatting; Step S2.3: running the distributed file cache system; Step S2.4: mounting the source data directory in the worker node in the distributed file cache system; In the step S3: Step S3.1: creating a file cache directory in the distributed file cache system; Step S3.2: selecting some nodes in the distributed file cache system as computing task submission nodes, deploying an application implemented through a distributed lock, writing data from the underlying storage system to the file cache directory through an IO stream, and using it as the input source for subsequent processing tasks.

2. The method for realizing storage-computing separation through a distributed file cache system according to claim 1, characterized in that: When there are multiple computing tasks dependent on the same source data, the cached data will be replicated between the worker nodes to realize short-circuit read and write. 3.The method for realizing storage computing separation through the distributed file cache system according to claim 1, characterized in that, In the step S1: Step S1.1: selecting a node in the storage cluster to configure a network file system server, and writing the source data directory and the return data directory of the underlying storage system into the configuration file; Step S1.2: starting the RPC service and the network file system service in the server; Step S1.3: selecting some nodes in the computing cluster to configure a network file system client; Step S1.4: mounting the source data directory and the return data directory in the client.

4. A system for implementing compute storage separation through a distributed file cache system, the system comprising: Comprising: Module M1: deploying a network file system server in the underlying storage system, and sharing the source data directory to the active area where the computing cluster is located; Module M2: deploying a distributed file cache system in the active area where the computing cluster is located, and mounting the shared source data directory in the distributed file cache system; Module M3: selecting a computing task submission node in the active area where the computing cluster is located, reading data and writing to the worker node; In the module M2: Module M2.1: configuring the distributed file cache system on each computing node, defining the underlying storage system, and defining the worker node; Module M2.2: establishing mutual trust between nodes, copying the configuration to all worker nodes, and formatting; Module M2.3: running the distributed file cache system; Module M2.4: mounting the source data directory in the worker node in the distributed file cache system; In the module M3: Module M3.1: creating a file cache directory in the distributed file cache system; Module M3.2: selecting some nodes in the distributed file cache system as computing task submission nodes, deploying an application implemented through a distributed lock, writing data from the underlying storage system to the file cache directory through an IO stream, and using it as the input source for subsequent processing tasks. Module M3.2: Select part of nodes in the distributed file cache system as computing task submission nodes, deploy the application implemented through the distributed lock, write the data of the underlying storage system into the file cache directory through the IO stream mode, and use the data as the input source for subsequent processing tasks.

5. The system for realizing storage-computing separation through the distributed file cache system according to claim 4, characterized in that: When there are multiple computing tasks depending on the same source data, the cache data will be replicated among the working nodes to realize short-circuit read-write.

6. The system for realizing storage computing separation through a distributed file cache system according to claim 4, wherein, In the module M1: Module M1.1: Select a node in the storage cluster to configure a network file system server, and write the source data directory and the returned data directory of the underlying storage system into a configuration file; Module M1.2: Start the RPC service and the network file system service on the server; Module M1.3: Select part of nodes in the computing cluster to configure a network file system client; Module M1.4: Mount the source data directory and the returned data directory on the client.

Images