A metadata accelerated access method and distributed file system

Abstract

The present application discloses a metadata accelerated access method and a distributed file system, the method comprising: a Samba server receiving a metadata access request sent by a Samba client; the Samba server sending the metadata access request to a metadata server, so that the metadata server responds to the metadata access request. The present application enables the Samba server to directly interact with the metadata server by installing a VFS interface on the metadata server, and the metadata server can directly share the metadata of files / directories in the cluster to the Samba server without passing through a kernel client or a fuse client, thereby simplifying the data transmission path, reducing the data processing and conversion process in the intermediate links, and improving the access speed of metadata, thereby improving the access speed of files / directories.

Description

Technical Field

[0001] The present application relates to the technical field of file systems, and in particular to a metadata accelerated access method and a distributed file system. Background Art

[0002] Currently, when a Ceph cluster shares its file interface through samba (Server Messages Block), the kernel client or fuse client needs to first mount the Ceph file interface on the kernel client or fuse client, and then the VFS module (VFS, i.e. Virtual File System) in the kernel client or fuse client interacts with the samba module to share the file directory mounted on the kernel client or fuse client with the samba module. This will delay the metadata reading and writing process and affect the access speed of files / directories, especially for directories containing a large number of files.

[0003] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0004] The technical problem to be solved by the present application is to provide a metadata accelerated access method and a distributed file system in view of the deficiencies of the prior art.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a metadata accelerated access method, wherein the metadata accelerated access method specifically includes:

[0006] The Samba server receives the metadata access request sent by the Samba client;

[0007] The Samba server sends the metadata access request to the metadata server, so that the metadata server responds to the metadata access request.

[0008] The metadata accelerated access method, wherein the metadata server is configured with a VFS interface, and the Samba server sends the metadata access request to the metadata server so that the metadata server responds to the metadata access request specifically includes:

[0009] The Samba server accesses the VFS interface in the metadata server through the VFS component;

[0010] The metadata server is controlled to receive a metadata access request through the VFS interface, and respond to the metadata access request to access metadata content in the metadata server.

[0011] The metadata accelerated access method, wherein the method further comprises:

[0012] The metadata server responds to the metadata access request and obtains metadata corresponding to the metadata access request;

[0013] The metadata server sends the metadata to the Samba server, and the Samba server feeds back the metadata to the Samba client.

[0014] The metadata accelerated access method, wherein, after the Samba server receives the metadata access request sent by the Samba client, the method further comprises:

[0015] The number of requests received by the Samba server to read metadata access requests;

[0016] When the number is greater than 1, the received metadata access requests are merged into one metadata access request.

[0017] In the metadata accelerated access method, when the metadata access request is obtained by merging multiple metadata access requests, the step of feeding back the metadata to the Samba client through the Samba server specifically includes:

[0018] Reading multiple metadata access requests corresponding to the metadata access request;

[0019] The target metadata corresponding to each metadata access request is selected from the metadata, and each target metadata is fed back to the Samba client corresponding to each metadata access request.

[0020] The metadata accelerated access method, wherein, before the Samba server receives the metadata access request sent by the Samba client, the method further comprises:

[0021] Install the samba server into the distributed file system.

[0022] The metadata accelerated access method, wherein the step of installing the samba server into the distributed file system is as follows:

[0023] The Samba server is installed in the distributed file system by installing the SMB protocol package on the distributed file system.

[0024] A second aspect of the present application provides a distributed file system, wherein the distributed file system includes a Samba server and a metadata server, the Samba server is used to receive a metadata access request sent by a Samba client, and send the metadata access request to the metadata server, so that the metadata server responds to the metadata access request.

[0025] The distributed file system, wherein the metadata server is deployed with a VFS interface.

[0026] The distributed file system, wherein the metadata server is used to respond to the metadata access request, obtain the metadata corresponding to the metadata access request; send the metadata to the Samba server, and feed back the metadata to the Samba client through the Samba server.

[0027] Beneficial effect: Compared with the prior art, the present application provides a metadata accelerated access method and a distributed file system, the method comprising: a Samba server receiving a metadata access request sent by a Samba client; the Samba server sending the metadata access request to a metadata server, so that the metadata server responds to the metadata access request. The present application enables the Samba server to interact directly with the metadata server by installing a VFS interface on the metadata server, and the metadata server can directly share the metadata of the files / directories in the cluster to the Samba server without going through the kernel client or the fuse client, thereby simplifying the data transmission path, reducing the data processing and conversion process in the intermediate links, and improving the access speed of the metadata, thereby improving the access speed of the files / directories. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 This is a block diagram of the distributed file system provided in an embodiment of the present application.

[0030] Figure 2 A flowchart of a metadata accelerated access method provided in an embodiment of the present application.

[0031] Figure 3 A flowchart of a merging process for metadata access requests is shown in FIG. DETAILED DESCRIPTION

[0032] The embodiments of the present application provide a metadata accelerated access method and a distributed file system. To make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0033] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0034] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0035] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0036] After research, it is found that when the Ceph cluster shares its file interface through samba (Server Messages Block), the kernel client or fuse client needs to first mount the Ceph file interface on the kernel client or fuse client, and then the VFS module (VFS, i.e. Virtual File System) in the kernel client or fuse client interacts with the samba module to share the file directory mounted on the kernel client or fuse client with the samba module. This will delay the metadata reading and writing process and affect the access speed of files / directories, especially for directories containing a large number of files.

[0037] In order to solve the above problem, in an embodiment of the present application, the Samba server receives a metadata access request sent by the Samba client; the Samba server sends the metadata access request to the metadata server, so that the metadata server responds to the metadata access request. The present application enables the Samba server to interact directly with the metadata server by installing a VFS interface on the metadata server. The metadata server can directly share the metadata of the files / directories in the cluster to the Samba server without going through the kernel client or the fuse client, thereby improving the access speed of the metadata, and further improving the access speed of the files / directories.

[0038] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0039] This embodiment provides a metadata accelerated access method, which is applied to a distributed file system, such as a Ceph cluster. Figure 1 As shown, the distributed file system includes a Samba server 100, a kernel client 200 and a metadata server 300. A bidirectional connection is established between the metadata server 300 and the Samba server 100, and the metadata server 300 is connected to the kernel client 200. When the Samba client needs to access files / directories of the metadata server in the distributed file system, the Samba server can be directly used to interact with the metadata server to obtain metadata corresponding to the metadata access request. Specifically, Figure 2 As shown, the metadata accelerated access method provided in the embodiment of the present application specifically includes:

[0040] S10. The Samba server receives the metadata access request sent by the Samba client.

[0041] Specifically, Samba is a software module that implements the SMB file sharing protocol on Linux and Windows systems. It consists of a server and a client. Samba allows Linux systems and Windows systems to share resources with each other. In Linux systems, smb servers are used to provide shared resources, and smb clients are used to access shared resources provided by other smb servers. In Windows systems, there are SMB / CIFS servers and clients. To this end, the Samba server implements a resource sharing server through the Samba software module, and the Samba client is a client that requests the Samba server to share resources through the Samba software module. For example, the Samba server and the Samba client are servers and clients in Windows systems, or, smbd servers and smb clients are used in Linux systems.

[0042] The Samba server in the embodiment of the present application is located in the Ceph cluster, and the metadata in the metadata server in the Ceph cluster is shared with the smb client through the Samba server. Specifically, the Samba server can be installed on the gateway node, and the Samba client is connected to the Ceph cluster through the Samba server to read and write data. Specifically, the samba server can be installed on the gateway node of the distributed file system, wherein the installation method can be a method of installing it on the gateway node through the SMB protocol package. The SMB (Server Message Block) protocol is a protocol for sharing resources on a local area network, and it can provide a file system for Windows systems, Linux systems or IOS systems in a local area network.

[0043] The metadata access request is used to request metadata content from the metadata server. That is, when the Samba client needs to access metadata, the Samba client will communicate with the Samba server and send a metadata access request to the Samba server, and the Samba server will respond to the metadata access request sent by the Samba client. Among them, the Samba server and the Samba client can establish a connection through the VFS component, that is, the Samba client connects with the VFS component in the Samba server through its own VFS component to perform metadata access requests and metadata transmission. For example, the Samba client can access the Samba server through the VFS component to view the shared directory in the Samba server, and then trigger a metadata access request based on the shared directory to obtain the metadata corresponding to the files under the shared directory.

[0044] Furthermore, the Samba server can interact with multiple Samba clients, and among the multiple Samba clients, there may be several Samba clients that simultaneously send metadata access requests to the Samba server, and several Samba clients simultaneously send metadata access requests to the Samba server. Therefore, when the Samba server receives the metadata access requests sent by the Samba clients, it can directly send the metadata access requests sent by each Samba client to the metadata server, or it can first detect whether other metadata access requests are received at the current time or in a preset time period. If other metadata access requests are received, the metadata access request and other metadata access requests will be integrated into one metadata access request.

[0045] In one implementation, if Figure 3 As shown, after the Samba server receives the metadata access request sent by the Samba client, the method further includes:

[0046] S11, the number of metadata access requests read by the Samba server;

[0047] S12: When the number is greater than 1, merge the received metadata access requests into one metadata access request.

[0048] Specifically, the number of requests can be the number of metadata access requests received at the same time, that is, when the Samba server receives the metadata access request sent by the Samba client, it will obtain all metadata access requests received at the current moment, and count the number of all metadata access requests obtained to obtain the number of requests, so as to ensure the response efficiency of the metadata access request.

[0049] In addition, in order to avoid metadata access request response delay or response failure caused by metadata server congestion, a metadata access request queue can be set in the Samba server. When the number of metadata access requests being responded to by the metadata server reaches a preset number threshold, the number of requests is the number of metadata access requests received within a period of time, that is, when the Samba server receives a metadata access request, it caches the metadata access request to the metadata access request queue, and detects the time interval between the cache time of the most cached metadata access request and the current time. When the time interval is greater than the preset time, the number of metadata access requests in the metadata access request queue is counted to obtain the number of requests, so as to reduce the number of requests from the Samba server to the metadata server. Then, when the number of metadata access requests sent by the Samba server to the metadata server is lower than the preset number threshold, the number of requests is converted to the number of metadata access requests received at the same time. That is to say, the Samba server will convert its own configured reading method of the number of requests according to the number of metadata access requests being responded to by the metadata server. Specifically, when the number of metadata access requests being responded to by the metadata server does not reach the preset number threshold, the reading method of the number of requests is the number of metadata access requests received at the same time; when the number of metadata access requests being responded to by the metadata server reaches the preset number threshold, the reading method of the number of requests is the number of metadata access requests received within a period of time.

[0050] Further, when the number is equal to 1, the metadata access request is directly sent to the metadata server; when the number is greater than 1, each metadata access request is merged into one metadata access request; specifically, the request content included in each metadata access request may be placed in one metadata access request, or the request content of each metadata access request may be merged to remove duplicate request content; wherein, when the request content of each metadata access request is merged to remove duplicate request content, it is necessary to record the request content required for the metadata access request corresponding to each Samba end, so that after obtaining the metadata fed back by the metadata server, metadata is fed back to each Samba end according to the request content required for the metadata access request corresponding to each Samba end.

[0051] In addition, in actual applications, in order to improve the security of metadata in the metadata server, before the Samba client sends a metadata access request to the Samba server, the Samba client needs to log in to the Samba server first, and after passing the Samba server verification, send the metadata access request to the Samba client. For example, first intercept and obtain the login request information of the user terminal accessing the Samba server, based on the management terminal or based on the self-configured management module, verify the user information contained in the login request information, if the verification is passed, establish a connection with the Samba client, if the verification is not passed, reject the login request of the Samba client. Among them, the user information may include a user name and a login password, and the Samba client is verified based on the user name and the login password. When the user name and the login password are verified, the historical login information and the current login information of the Samba client are obtained, and the historical login information and the current login information are input into a preset verification large model, and the secondary verification result of the Samba client is output through the preset verification large model. If several secondary verification results are passed, it is determined that the Samba client verification is passed, otherwise it is determined that the Samba client verification is not passed. The historical login information may include several historical login addresses, several historical login IP addresses, and several historical requested metadata types, and the current login information may include the current login address, the current login IP address, and the metadata type to be requested, etc. The embodiment of the present application can improve the security of the Samba client by performing secondary verification on the Samba client, thereby avoiding the security of the Samba server, and thereby improving the security of the metadata in the metadata server.

[0052] Of course, in actual applications, in order to avoid erroneous verification failure of the Samba client, when the Samba client fails the secondary verification, the user information of the Samba client can be sent to the preset administrator or the Samba client can be told that it needs to obtain the management verification identifier of the preset administrator, and when the preset administrator feeds back the management verification identifier or the Samba client sends the management verification identifier, the verification failure of the Samba client is changed to verification success, and a connection is established with the Samba client to avoid misjudgment of the Samba client.

[0053] S20. The Samba server sends the metadata access request to the metadata server, so that the metadata server responds to the metadata access request.

[0054] Specifically, the metadata server is responsible for managing all metadata information in the distributed file system, and the Samba server communicates with the metadata server, and transmits metadata access requests and metadata corresponding to the metadata access requests through the communication. That is to say, after receiving the metadata access request, the Samba server can directly send the metadata access request to the metadata server through the Samba server communicating with the metadata server without passing through the kernel client, that is, the Samba server does not need to first send the metadata access request to the kernel client, and then send the metadata access request to the metadata server through the kernel client, which can reduce the metadata access request and the intermediate processing process of metadata (such as data processing and conversion process, etc.) transmitted between the Samba server and the metadata server, and improve the access efficiency of metadata.

[0055] In one implementation, the Samba server sends the metadata access request to the metadata server so that the metadata server responds to the metadata access request specifically including:

[0056] The Samba server accesses the VFS interface in the metadata server through the VFS component;

[0057] The metadata server is controlled to receive a metadata access request through the VFS interface, and respond to the metadata access request to access metadata content in the metadata server.

[0058] Specifically, the metadata server is configured with a VFS interface, and the VFS interface is used to connect with the Samba server through the VFS component to receive the metadata access request sent by the Samba server through the VFS interface, and the transfer process of the kernel client is omitted. That is to say, the Samba server accesses the VFS interface in the metadata server through the VFS component to establish a connection with the metadata server, and then sends the metadata access request to the metadata server through the connection, so that the metadata server can bypass the kernel client and directly obtain the metadata access request.

[0059] After receiving the metadata access request, the metadata server will directly feed back the metadata corresponding to the metadata access request to the Samba server. To this end, the method also includes: the metadata server responds to the metadata access request and obtains the metadata corresponding to the metadata access request; the metadata server sends the metadata to the Samba server, and feeds back the metadata to the Samba client through the Samba server. The metadata server directly feeds back the metadata to the Samba server through the connection established by the VFS interface configured by it and the VFS component in the Samba server, so that the metadata server can bypass the kernel client and directly feed back the metadata to the Samba server.

[0060] Furthermore, when the Samba server feeds back the metadata to the Samba client, it will determine the metadata access request corresponding to the metadata fed back by the metadata server. When the metadata access request corresponding to the metadata is obtained by merging multiple metadata access requests, the Samba server will read the multiple metadata access requests corresponding to the metadata access request; and select the target metadata corresponding to each metadata access request from the metadata, and feed back each target metadata to the Samba client corresponding to each metadata access request. In this way, the metadata corresponding to each Samba client can be fed back, the number of metadata access requests sent to the metadata server can be reduced, and repeated transmission of metadata between the Samba server and the metadata server can be avoided.

[0061] In summary, this embodiment provides a method for accelerating metadata access, the method comprising: a Samba server receiving a metadata access request sent by a Samba client; the Samba server sending the metadata access request to a metadata server, so that the metadata server responds to the metadata access request. This application enables the Samba server to interact directly with the metadata server by installing a VFS interface on the metadata server, and the metadata server can directly share the metadata of files / directories in the cluster to the Samba server without going through a kernel client or a fuse client, thereby simplifying the data transmission path, reducing the data processing and conversion process in the intermediate links, and improving the access speed of metadata, thereby improving the access speed of files / directories.

[0062] Based on the above metadata accelerated access method, the present application embodiment provides a distributed file system, such as Figure 1As shown, the distributed file system includes a Samba server 100, a kernel client 200 and a metadata server 300, wherein a bidirectional connection is established between the metadata server 300 and the Samba server 100, and a connection is established between the metadata server 300 and the kernel client 200, wherein the bidirectional connection established between the metadata server 300 and the Samba server 100 is used to realize direct transmission of metadata access requests and metadata between the metadata server 300 and the Samba server 100, and the connection established between the metadata server 300 and the kernel client 200 is used to inform the kernel client 200 of the metadata information shared by the metadata server 300 to the Samba server 100. The Samba server 100 is used to receive the metadata access request sent by the Samba client, and send the metadata access request to the metadata server 300, so that the metadata server 300 responds to the metadata access request. The metadata server 300 is used to respond to the metadata access request, obtain the metadata corresponding to the metadata access request; send the metadata to the Samba server 100, and feed the metadata back to the Samba client through the Samba server 100.

[0063] Furthermore, the metadata server 300 is deployed with a VFS interface, and the VFS interface deployed by the metadata server 300 and the VFS component in the Samba server 100 are set with a private transmission protocol. The Samba server 100 accesses the VFS interface deployed by the metadata server through the VFS component, so that a two-way connection is established between the metadata server 300 and the Samba server 100.

[0064] In addition, the specific working processes of the metadata server and the Samba server have been described in detail in the above method and will not be described here one by one.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A metadata accelerated access method, characterized in that: The metadata accelerated access method specifically includes: The Samba server receives the metadata access request sent by the Samba client, wherein the Samba server is installed on the gateway node of the distributed file system, the Samba client is connected to the Ceph cluster through the Samba server, and the Samba server interacts with multiple Samba clients; The Samba server reads the number of requests for metadata access received, and when the number is greater than 1, merges the received metadata access requests into one metadata access request, and records the request content required for the metadata access request corresponding to each Samba terminal, so that after obtaining the metadata fed back by the metadata server, metadata is fed back to each Samba terminal according to the request content required for the metadata access request corresponding to each Samba terminal; The Samba server sends the metadata access request to the metadata server, so that the metadata server responds to the metadata access request, wherein the metadata server is configured with a VFS interface, and the Samba server accesses the VFS interface in the metadata server through the VFS component; The metadata server responds to the metadata access request and obtains metadata corresponding to the metadata access request; The metadata server sends the metadata to the Samba server, and the Samba server feeds back the metadata to the Samba client; Among them, the Samba server will convert its own configured request quantity reading method according to the number of metadata access requests being responded to by the metadata server. Specifically, when the number of metadata access requests being responded to by the metadata server does not reach the preset number threshold, the request quantity reading method is the request number of metadata access requests received at the same time; when the number of metadata access requests being responded to by the metadata server reaches the preset number threshold, the request quantity reading method is the request number of metadata access requests received within a period of time.

2. The metadata accelerated access method according to claim 1, characterized in that: The Samba server sends the metadata access request to the metadata server, so that the metadata server responds to the metadata access request specifically including; The metadata server is controlled to receive a metadata access request through a VFS interface, and respond to the metadata access request to access metadata content in the metadata server.

3. The metadata accelerated access method according to claim 1, characterized in that: When the metadata access request is obtained by merging multiple metadata access requests, the step of feeding back the metadata to the Samba client through the Samba server specifically includes: Reading multiple metadata access requests corresponding to the metadata access request; The target metadata corresponding to each metadata access request is selected from the metadata, and each target metadata is fed back to the Samba client corresponding to each metadata access request.

4. The data access acceleration method according to claim 1, characterized in that: Installing the samba server into the distributed file system is as follows: The Samba server is installed in the distributed file system by installing the SMB protocol package on the distributed file system.

5. A distributed file system, characterized in that: The distributed file system includes a Samba server and a metadata server. The Samba server is installed on a gateway node of the distributed file system. The Samba client is connected to the Ceph cluster through the Samba server. The Samba server interacts with multiple Samba clients. The metadata server is configured with a VFS interface. The Samba server accesses the VFS interface in the metadata server through the VFS component. The Samba server is used to receive the metadata access request sent by the Samba client, read the request number of the received metadata access request, and when the number is greater than 1, merge the received metadata access requests into one metadata access request, and record the request content required by the metadata access request corresponding to each Samba end, so as to feedback metadata to each Samba end according to the request content required by the metadata access request corresponding to each Samba end after obtaining the metadata fed back by the metadata server; Sending the metadata access request to a metadata server so that the metadata server responds to the metadata access request; The metadata server is used to respond to the metadata access request and obtain metadata corresponding to the metadata access request; Sending the metadata to a Samba server, and feeding the metadata back to a Samba client through the Samba server; Among them, the Samba server will convert its own configured request quantity reading method according to the number of metadata access requests being responded to by the metadata server. Specifically, when the number of metadata access requests being responded to by the metadata server does not reach the preset number threshold, the request quantity reading method is the request number of metadata access requests received at the same time; when the number of metadata access requests being responded to by the metadata server reaches the preset number threshold, the request quantity reading method is the request number of metadata access requests received within a period of time.

Images